You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oozie.apache.org by an...@apache.org on 2018/09/14 14:38:44 UTC
[01/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Repository: oozie
Updated Branches:
refs/heads/master 4e5b3cb59 -> 6a6f21996
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/release-log.txt
----------------------------------------------------------------------
diff --git a/release-log.txt b/release-log.txt
index bf6c1c7..68f4412 100644
--- a/release-log.txt
+++ b/release-log.txt
@@ -1,5 +1,6 @@
-- Oozie 5.1.0 release (trunk - unreleased)
+OOZIE-2734 amend [docs] Switch from TWiki to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
OOZIE-2734 [docs] Switch from TWiki to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
OOZIE-3298 [MapReduce action] External ID is not filled properly and failing MR job is treated as SUCCEEDED (andras.piros via pbacsko, asasvari, gezapeti)
OOZIE-3317 amend [build] Fix false positive precommit reports (kmarton via andras.piros)
[18/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_ActionAuthentication.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_ActionAuthentication.md b/docs/src/site/markdown/DG_ActionAuthentication.md
new file mode 100644
index 0000000..d280baf
--- /dev/null
+++ b/docs/src/site/markdown/DG_ActionAuthentication.md
@@ -0,0 +1,132 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Action Authentication
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Background
+
+A secure cluster requires that actions have been authenticated (typically via Kerberos). However, due to the way that Oozie runs
+actions, Kerberos credentials are not easily made available to actions launched by Oozie. For many action types, this is not a
+problem because they are self contained (beyond core Hadoop components). For example, a Pig action typically only talks to
+MapReduce and HDFS. However, some actions require talking to external services (e.g. HCatalog, HBase Region Server, Hive Server 2)
+and in these cases, the actions require some extra configuration in Oozie to authenticate. To be clear, this extra configuration
+is only required if an action will be talking to these types of external services; running a typical MapReduce, Pig, Hive, etc
+action will not require any of this.
+
+For these situations, Oozie will have to use its Kerberos credentials to obtain "delegation tokens" (think of it like a cookie) on
+behalf of the user from the service in question. The details of what this means is beyond the scope of this documentation, but
+basically, Oozie needs some extra configuration in the workflow so that it can obtain this delegation token.
+
+## Oozie Server Configuration
+
+The code to obtain delegation tokens is pluggable so that it is easy to add support for different services by simply subclassing
+org.apache.oozie.action.hadoop.Credentials to retrieve a delegation token from the service and add it to the Configuration.
+
+Out of the box, Oozie already comes with support for some credential types
+(see [Built-in Credentials Implementations](DG_ActionAuthentication.html#Built-in_Credentials_Implementations)).
+The credential classes that Oozie should load are specified by the following property in oozie-site.xml. The left hand side of the
+equals sign is the type for the credential type, while the right hand side is the class.
+
+
+```
+ <property>
+ <name>oozie.credentials.credentialclasses</name>
+ <value>
+ hcat=org.apache.oozie.action.hadoop.HCatCredentials,
+ hbase=org.apache.oozie.action.hadoop.HbaseCredentials,
+ hive2=org.apache.oozie.action.hadoop.Hive2Credentials
+ </value>
+ </property>
+```
+
+## Workflow Changes
+
+The user should add a `credentials` section to the top of their workflow that contains 1 or more `credential` sections. Each of
+these `credential` sections contains a name for the credential, the type for the credential, and any configuration properties
+needed by that type of credential for obtaining a delegation token. The `credentials` section is available in workflow schema
+version 0.3 and later.
+
+For example, the following workflow is configured to obtain an HCatalog delegation token, which is given to a Pig action so that the
+Pig action can talk to a secure HCatalog:
+
+
+```
+ <workflow-app xmlns='uri:oozie:workflow:0.4' name='pig-wf'>
+ <credentials>
+ <credential name='my-hcat-creds' type='hcat'>
+ <property>
+ <name>hcat.metastore.uri</name>
+ <value>HCAT_URI</value>
+ </property>
+ <property>
+ <name>hcat.metastore.principal</name>
+ <value>HCAT_PRINCIPAL</value>
+ </property>
+ </credential>
+ </credentials>
+ ...
+ <action name='pig' cred='my-hcat-creds'>
+ <pig>
+ <job-tracker>JT</job-tracker>
+ <name-node>NN</name-node>
+ <configuration>
+ <property>
+ <name>TESTING</name>
+ <value>${start}</value>
+ </property>
+ </configuration>
+ </pig>
+ </action>
+ ...
+ </workflow-app>
+```
+
+The type of the `credential` is "hcat", which is the type name we gave for the HCatCredentials class in oozie-site.xml. We gave
+the `credential` a name, "my-hcat-creds", which can be whatever you want; we then specify cred='my-hcat-creds' in the Pig action,
+so that Oozie will include these credentials with the action. You can include multiple credentials with an action by specifying
+a comma-separated list of `credential` names. And finally, the HCatCredentials required two properties (the metastore URI and
+principal), which we also specified.
+
+Adding the `credentials` section to a workflow and referencing it in an action will make Oozie always try to obtain that delegation
+token. Ordinarily, this would mean that you cannot re-use this workflow in a non-secure cluster without editing it because trying
+to obtain the delegation token will likely fail. However, you can tell Oozie to ignore the `credentials` for a workflow by setting
+the job-level property `oozie.credentials.skip` to `true`; this will allow you to use the same workflow.xml in a secure and
+non-secure cluster by simply changing the job-level property at runtime. If omitted or set to `false`, Oozie will handle
+the `credentials` section normally. In addition, you can also set this property at the action-level or server-level to skip getting
+credentials for just that action or for all workflows, respectively. The order of priority is this:
+
+ 1. `oozie.credentials.skip` in the `configuration` section of an action, if set
+ 1. `oozie.credentials.skip` in the job.properties for a workflow, if set
+ 1. `oozie.credentials.skip` in oozie-site.xml for all workflows, if set
+ 1. (don't skip)
+
+## Built-in Credentials Implementations
+
+Oozie currently comes with the following Credentials implementations:
+
+ 1. HCatalog and Hive Metastore: `org.apache.oozie.action.hadoop.HCatCredentials`
+ 1. HBase: `org.apache.oozie.action.hadoop.HBaseCredentials`
+ 1. Hive Server 2: `org.apache.oozie.action.hadoop.Hive2Credentials`
+
+HCatCredentials requires these two properties:
+
+ 1. `hcat.metastore.principal` or hive.metastore.kerberos.principal
+ 1. `hcat.metastore.uri` or hive.metastore.uris
+
+**Note:** The HCatalog Metastore and Hive Metastore are one and the same and so the "hcat" type credential can also be used to talk
+to a secure Hive Metastore, though the property names would still start with "hcat.".
+
+HBase does not require any additional properties since the hbase-site.xml on the Oozie server provides necessary information
+to obtain a delegation token; though properties can be overwritten here if desired.
+
+Hive2Credentials requires these two properties:
+
+ 1. `hive2.server.principal`
+ 1. `hive2.jdbc.url`
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
[07/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_CommandLineTool.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_CommandLineTool.twiki b/docs/src/site/twiki/DG_CommandLineTool.twiki
deleted file mode 100644
index 51382c7..0000000
--- a/docs/src/site/twiki/DG_CommandLineTool.twiki
+++ /dev/null
@@ -1,2295 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Command Line Interface Utilities
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Introduction
-
-Oozie provides a command line utility, `oozie`, to perform job and admin tasks. All operations are done via
-sub-commands of the `oozie` CLI.
-
-The `oozie` CLI interacts with Oozie via its WS API.
-
-## Oozie Command Line Usage
-
-
-```
-usage:
- the env variable 'OOZIE_URL' is used as default value for the '-oozie' option
- the env variable 'OOZIE_TIMEZONE' is used as default value for the '-timezone' option
- the env variable 'OOZIE_AUTH' is used as default value for the '-auth' option
- custom headers for Oozie web services can be specified using '-Dheader:NAME=VALUE'
-```
-
-### Oozie basic commands
-
-```
-
-oozie help : display usage
-
-oozie version : show client version
-
-```
-
-### Oozie job operation commands
-
-```
-
-oozie job <OPTIONS> : job operations
- -action <arg> coordinator rerun/kill on action ids (requires -rerun/-kill);
- coordinator log retrieval on action ids (requires -log)
- -allruns Get workflow jobs corresponding to a coordinator action
- including all the reruns
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -change <arg> change a coordinator or bundle job
- -config <arg> job configuration file '.xml' or '.properties'
- -configcontent <arg> job configuration
- -coordinator <arg> bundle rerun on coordinator names (requires -rerun)
- -D <property=value> set/override value for given property
- -date <arg> coordinator/bundle rerun on action dates (requires -rerun);
- coordinator log retrieval on action dates (requires -log)
- -debug Use debug mode to see debugging statements on stdout
- -definition <arg> job definition
- -diff <arg> Show diff of the new coord definition and properties with the
- existing one (default true)
- -doas <arg> doAs user, impersonates as the specified user
- -dryrun Dryrun a coordinator (since 2.0), a workflow (since 3.3.2) or a bundle (since 5.1)
- without actually executing it.
- -failed re-runs the failed workflow actions of the coordinator actions (requires -rerun)
- -filter <arg> <key><comparator><value>[;<key><comparator><value>]*
- (All Coordinator actions satisfying the filters will be retrieved).
- key: status or nominaltime
- comparator: =, !=, <, <=, >, >=. = is used as OR and others as AND
- status: values are valid status like SUCCEEDED, KILLED etc. Only = and != apply
- for status.
- nominaltime: time of format yyyy-MM-dd'T'HH:mm'Z'
- -ignore <arg> change status of a coordinator job or action to IGNORED
- (-action required to ignore coord actions)
- -info <arg> info of a job
- -interval <arg> polling interval in minutes (default is 5, requires -poll)
- -kill <arg> kill a job (coordinator can mention -action or -date)
- -len <arg> number of actions (default TOTAL ACTIONS, requires -info)
- -localtime use local time (same as passing your time zone to -timezone).
- Overrides -timezone option
- -log <arg> job log
- -errorlog <arg> job error log
- -auditlog <arg> job audit log
- -logfilter <arg> job log search parameter. Can be specified as -logfilter
- opt1=val1;opt2=val1;opt3=val1. Supported options are recent,
- start, end, loglevel, text, limit and debug
- -nocleanup do not clean up output-events of the coordinator rerun
- actions (requires -rerun)
- -offset <arg> job info offset of actions (default '1', requires -info)
- -oozie <arg> Oozie URL
- -order <arg> order to show coord actions (default ascending order, 'desc'
- for descending order, requires -info)
- -poll <arg> poll Oozie until a job reaches a terminal state or a timeout
- occurs
- -refresh re-materialize the coordinator rerun actions (requires
- -rerun)
- -rerun <arg> rerun a job (coordinator requires -action or -date, bundle
- requires -coordinator or -date)
- -resume <arg> resume a job
- -run run a job
- -start <arg> start a job
- -submit submit a job
- -suspend <arg> suspend a job
- -timeout <arg> timeout in minutes (default is 30, negative values indicate
- no timeout, requires -poll)
- -timezone <arg> use time zone with the specified ID (default GMT).
- See 'oozie info -timezones' for a list
- -update <arg> Update coord definition and properties
- -value <arg> new endtime/concurrency/pausetime value for changing a
- coordinator job
- -verbose verbose mode
- -sladisable disables sla alerts for the job and its children
- -slaenable enables sla alerts for the job and its children
- -slachange Update sla param for jobs, supported param are should-start, should-end and max-duration
- -retries Get information of the retry attempts for a given workflow action.
- -apijarcheck <jar> based on the supplied Fluent Job API jar, a workflow definition XML is generated and checked whether
- it's a valid Oozie workflow. Output is whether the generated workflow is a valid one
- -apijarsubmit <jar> based on the supplied Fluent Job API jar, a workflow definition XML is generated and submitted. When
- the parameter =oozie.wf.application.path= isn't supplied on the command line, an HDFS location with
- the prefix defined in
- =oozie-site.xml#oozie.client.jobs.application.generated.path= is used. Output is the workflow ID
- -apijarrun <jar> based on the supplied Fluent Job API jar, a workflow definition XML is generated and run. When the
- parameter =oozie.wf.application.path= isn't supplied on the command line, an HDFS location with the
- prefix defined in
- =oozie-site.xml#oozie.client.jobs.application.generated.path= is used. Output is the workflow ID
-
-```
-
-### Oozie jobs operation commands
-
-```
-oozie jobs <OPTIONS> : jobs status
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -doas <arg> doAs user, impersonates as the specified user.
- -filter <arg> user=<U>\;name=<N>\;group=<G>\;status=<S>\;frequency=<F>\;unit=<M>\;startcreatedtime=<SC>\;
- endcreatedtime=<EC>\;sortby=<SB>
- -jobtype <arg> job type ('Supported in Oozie-2.0 or later versions ONLY - coordinator' or 'wf' (default))
- -len <arg> number of jobs (default '100')
- -localtime use local time (same as passing your time zone to -timezone). Overrides -timezone option
- -offset <arg> jobs offset (default '1')
- -oozie <arg> Oozie URL
- -timezone <arg> use time zone with the specified ID (default GMT). See 'oozie info -timezones' for a list
- -kill kill all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without
- other options, it will kill all the first 50 workflow jobs. Command will fail if one or more
- of the jobs is in wrong state.
- -suspend suspend all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without
- other options, it will suspend all the first 50 workflow jobs. Command will fail if one or more
- of the jobs is in wrong state.
- -resume resume all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without
- other options, it will resume all the first 50 workflow jobs. Command will fail if one or more
- of the jobs is in wrong state.
- -verbose verbose mode
-```
-
-### Oozie admin operation commands
-
-```
-oozie admin <OPTIONS> : admin operations
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -configuration show Oozie system configuration
- -doas <arg> doAs user, impersonates as the specified user
- -instrumentation show Oozie system instrumentation
- -javasysprops show Oozie Java system properties
- -metrics show Oozie system metrics
- -oozie <arg> Oozie URL
- -osenv show Oozie system OS environment
- -queuedump show Oozie server queue elements
- -servers list available Oozie servers (more than one only if HA is enabled)
- -shareliblist List available sharelib that can be specified in a workflow action
- -sharelibupdate Update server to use a newer version of sharelib
- -status show the current system status
- -systemmode <arg> Supported in Oozie-2.0 or later versions ONLY. Change oozie
- system mode [NORMAL|NOWEBSERVICE|SAFEMODE]
- -version show Oozie server build version
- -purge <arg> purge old oozie workflow, coordinator and bundle records from DB (parameter unit: day)
- wf=<N>\;coord=<N>\;bundle=<N>\;limit=<N>\;oldcoordaction=<true/false>
-```
-
-### Oozie validate command
-
-```
-oozie validate <OPTIONS> <ARGS> : validate a workflow, coordinator, bundle XML file
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -oozie <arg> Oozie URL
-```
-
-### Oozie SLA operation commands
-
-```
-oozie sla <OPTIONS> : sla operations (Deprecated as of Oozie 4.0)
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -len <arg> number of results (default '100', max limited by oozie server setting which defaults to '1000')
- -offset <arg> start offset (default '0')
- -oozie <arg> Oozie URL
- -filter <arg> jobid=<JobID/ActionID>\;appname=<Application Name>
-```
-
-### Oozie Pig submit command
-
-```
-oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass-through parameters to pig, any '-D' arguments
- after '-X' are put in <configuration>
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -doas <arg> doAs user, impersonates as the specified user.
- -config <arg> job configuration file '.properties'
- -D <property=value> set/override value for given property
- -file <arg> Pig script
- -oozie <arg> Oozie URL
- -P <property=value> set parameters for script
-```
-
-### Oozie Hive submit command
-
-```
-oozie hive <OPTIONS> -X<ARGS> : submit a hive job, everything after '-X' are pass-through parameters to hive, any '-D' arguments
- after '-X' are put in <configuration>
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -config <arg> job configuration file '.properties'
- -D <property=value> set/override value for given property
- -doas <arg> doAs user, impersonates as the specified user
- -file <arg> hive script
- -oozie <arg> Oozie URL
- -P <property=value> set parameters for script
-```
-
-### Oozie Sqoop submit command
-
-```
-oozie sqoop <OPTIONS> -X<ARGS> : submit a sqoop job, any '-D' arguments after '-X' are put in <configuration>
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -config <arg> job configuration file '.properties'
- -D <property=value> set/override value for given property
- -doas <arg> doAs user, impersonates as the specified user
- -command <arg> sqoop command
- -oozie <arg> Oozie URL
-```
-
-### Oozie info command
-
-```
-oozie info <OPTIONS> : get more detailed info about specific topics
- -timezones display a list of available time zones
-```
-
-### Oozie MapReduce job command
-
-```
-oozie mapreduce <OPTIONS> : submit a mapreduce job
- -auth <arg> select authentication type [SIMPLE|KERBEROS]
- -config <arg> job configuration file '.properties'
- -D <property=value> set/override value for given property
- -doas <arg> doAs user, impersonates as the specified user
- -oozie <arg> Oozie URL
-```
-
-## Common CLI Options
-
-### Authentication
-
-The `oozie` CLI automatically perform authentication if the Oozie server requests it. By default it supports both
-pseudo/simple authentication and Kerberos HTTP SPNEGO authentication.
-
-To perform a specific authentication, the `auth` option with authentication type requests Oozie client to run the
-specified authentication mechanism only. Oozie client provides two types `simple` and `kerberos` which supports `pseudo/simple` and `Kerberos`.
-
-For pseudo/simple authentication the `oozie` CLI uses the user name of the current OS user.
-
-For Kerberos HTTP SPNEGO authentication the `oozie` CLI uses the default principal for the OS Kerberos cache
-(normally the principal that did `kinit`).
-
-Oozie uses Apache Hadoop-Auth (Java HTTP SPNEGO) library for authentication.
-This library can be extended to support other authentication mechanisms.
-
-Once authentication is performed successfully the received authentication token is cached in the user home directory
-in the `.oozie-auth-token` file with owner-only permissions. Subsequent requests reuse the cached token while valid.
-
-The use of the cache file can be disabled by invoking the `oozie` CLI with the `-Doozie.auth.token.cache`false=
-option.
-
-To use an custom authentication mechanism, a Hadoop-Auth `Authenticator` implementation must be specified with the
- `-Dauthenticator.class=CLASS` option.
-
-### Impersonation, doAs
-
-The `-doas` option allows the current user to impersonate other users when interacting with the Oozie
-system. The current user must be configured as a proxyuser in the Oozie system. The proxyuser configuration may
-restrict from which hosts a user may impersonate users, as well as users of which groups can be impersonated.
-
-### Oozie URL
-
-All `oozie` CLI sub-commands expect the `-oozie OOZIE_URL` option indicating the URL of the Oozie system
-to run the command against. If the OOZIE_URL environment variable has not been set, `oozie` will use the default
-URL specified in oozie-client-env.sh (equivalent to `!http://$(hostname -f):11000/oozie`).
-
-If the `-oozie` option is not specified, the `oozie` CLI will look for the `OOZIE_URL` environment variable
-and uses it if set.
-
-If the option is not provided and the environment variable is not set, the `oozie` CLI will fail.
-
-### Time zone
-
-The `-timezone TIME_ZONE_ID` option in the `job` and `jobs` sub-commands allows you to specify the time zone to use in
-the output of those sub-commands. The `TIME_ZONE_ID` should be one of the standard Java Time Zone IDs. You can get a
-list of the available time zones with the command `oozie info -timezones`.
-
-If the `-localtime` option is used, it will cause Oozie to use whatever the time zone is of the machine. If
-both `-localtime` and `-timezone TIME_ZONE_ID` are used, the `-localtime` option will override
-the `-timezone TIME_ZONE_ID` option. If neither option is given, Oozie will look for the `OOZIE_TIMEZONE` environment
-variable and uses it if set. If neither option is given and the environment variable is not set, or if Oozie is given an invalid
-time zone, it will use GMT.
-
-### Debug Mode
-
-If you export `OOZIE_DEBUG=1` then the Oozie CLI will output the Web Services API details used by any commands you
-execute. This is useful for debugging purposes to or see how the Oozie CLI works with the WS API.
-
-### CLI retry
-Oozie CLI retries connection to Oozie servers for transparent high availability failover when one of the Oozie servers go down.
-`Oozie` CLI command will retry for all commands in case of ConnectException.
-In case of SocketException, all commands except `PUT` and `POST` will have retry logic.
-All job submit are POST call, examples of PUT and POST commands can be find out from [WebServicesAPI](WebServicesAPI.html).
-Retry count can be configured with system property `oozie.connection.retry.count`. Default count is 4.
-
-
-## Job Operations
-
-### Submitting a Workflow, Coordinator or Bundle Job
-
-* Submitting bundle feature is only supported in Oozie 3.0 or later. Similarly, all bundle operation features below are only
-supported in Oozie 3.0 or later.
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -config job.properties -submit
-.
-job: 14-20090525161321-oozie-joe
-```
-
-The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the `-config` option.
-
-The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
-coordinator application path must be specified in the file with the `oozie.coord.application.path` property.The
-bundle application path must be specified in the file with the `oozie.bundle.application.path` property.
-Specified path must be an HDFS path.
-
-The job will be created, but it will not be started, it will be in `PREP` status.
-
-### Starting a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -start 14-20090525161321-oozie-joe
-```
-
-The `start` option starts a previously submitted workflow job or bundle job that is in `PREP` status.
-
-After the command is executed the workflow job will be in `RUNNING` status and bundle job will be in `RUNNING`status.
-
-A coordinator job does not support the `start` action. It will show the following error message when trying to start it
-via the CLI:
-
-
-```
-Error: E0303 : E0303: Invalid parameter value, [action] = [start]
-```
-
-### Running a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -config job.properties -run
-.
-job: 15-20090525161321-oozie-joe
-```
-
-The `run` option creates and starts a workflow job, coordinator job or bundle job.
-
-The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the `-config` option.
-
-The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
-coordinator application path must be specified in the file with the `oozie.coord.application.path` property. The
-bundle application path must be specified in the file with the `oozie.bundle.application.path` property.The
-specified path must be an HDFS path.
-
-The job will be created and it will started, the job will be in `RUNNING` status.
-
-### Suspending a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -suspend 14-20090525161321-oozie-joe
-```
-
-The `suspend` option suspends a workflow job in `RUNNING` status.
-After the command is executed the workflow job will be in `SUSPENDED` status.
-
-The `suspend` option suspends a coordinator/bundle job in `RUNNING`, `RUNNINGWITHERROR` or `PREP` status.
-When the coordinator job is suspended, running coordinator actions will stay in running and the workflows will be suspended. If the coordinator job is in `RUNNING`status, it will transit to `SUSPENDED`status; if it is in `RUNNINGWITHERROR`status, it will transit to `SUSPENDEDWITHERROR`; if it is in `PREP`status, it will transit to `PREPSUSPENDED`status.
-
-When the bundle job is suspended, running coordinators will be suspended. If the bundle job is in `RUNNING`status, it will transit to `SUSPENDED`status; if it is in `RUNNINGWITHERROR`status, it will transit to `SUSPENDEDWITHERROR`; if it is in `PREP`status, it will transit to `PREPSUSPENDED`status.
-
-### Resuming a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -resume 14-20090525161321-oozie-joe
-```
-
-The `resume` option resumes a workflow job in `SUSPENDED` status.
-
-After the command is executed the workflow job will be in `RUNNING` status.
-
-
-The `suspend` option suspends a coordinator/bundle job in `SUSPENDED`, `SUSPENDEDWITHERROR` or `PREPSUSPENDED` status.
-If the coordinator job is in `SUSPENDED`status, it will transit to `RUNNING`status; if it is in `SUSPENDEDWITHERROR`status, it will transit to `RUNNINGWITHERROR`; if it is in `PREPSUSPENDED`status, it will transit to `PREP`status.
-
-When the coordinator job is resumed it will create all the coordinator actions that should have been created during the time
-it was suspended, actions will not be lost, they will delayed.
-
-When the bundle job is resumed, suspended coordinators will resume running. If the bundle job is in `SUSPENDED`status, it will transit to `RUNNING`status; if it is in `SUSPENDEDWITHERROR`status, it will transit to `RUNNINGWITHERROR`; if it is in `PREPSUSPENDED`status, it will transit to `PREP`status.
-
-
-### Killing a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -kill 14-20090525161321-oozie-joe
-```
-
-The `kill` option kills a workflow job in `PREP`, `SUSPENDED` or `RUNNING` status and a coordinator/bundle job in
-`PREP`, `RUNNING`, `PREPSUSPENDED`, `SUSPENDED`, `PREPPAUSED`, or `PAUSED` status.
-
-After the command is executed the job will be in `KILLED` status.
-
-### Killing a Coordinator Action or Multiple Actions
-
-Example:
-
-
-```
-$oozie job -kill <coord_Job_id> [-action 1, 3-4, 7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
-```
-
- * The `kill` option here for a range of coordinator actions kills a non-terminal (`RUNNING`, `WAITING`, `READY`, `SUSPENDED`) coordinator action when coordinator job is not in `FAILED` or `KILLED` state.
- * Either -action or -date should be given.
- * If neither -action nor -date is given, the exception will be thrown. Also if BOTH -action and -date are given, an error will be thrown.
- * Multiple ranges can be used in -action or -date. See the above example.
- * If one of the actions in the given list of -action is already in terminal state, the output of this command will only include the other actions.
- * The dates specified in -date must be UTC.
- * Single date specified in -date must be able to find an action with matched nominal time to be effective.
- * After the command is executed the killed coordinator action will have `KILLED` status.
-
-### Changing endtime/concurrency/pausetime/status of a Coordinator Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value endtime=2011-12-01T05:00Z\;concurrency=100\;2011-10-01T05:00Z
-$ oozie job -oozie http://localhost:11000/oozie -change 0000001-140321155112907-oozie-puru-C -value status=RUNNING
-```
-
-The `endtime/concurrency/pausetime` option changes a coordinator job that is not in `KILLED` status.
-
-Valid value names are:
-
-
- * endtime: the end time of the coordinator job.
- * concurrency: the concurrency of the coordinator job.
- * pausetime: the pause time of the coordinator job.
- * status: new status for coordinator job.
-
-Conditions and usage:
-
- * Repeated value names are not allowed.
- * New end time should not be before job's start time and last action time.
- * If end time is before job start time and if the job has not materialized any actions, then job status is changed to SUCCEEDED.
- * Currently status only takes RUNNING and can be used to change the status of FAILED, KILLED, IGNORED coordinator job to RUNNING and resuming materialization. This status change command does not affect the status of already materialized actions in the coordinator. If there are FAILED, KILLED or IGNORED coordinator actions they have to be rerun separately.
- * New concurrency value has to be a valid integer.
- * All lookahead actions which are in WAITING/READY state will be revoked according to the new pause/end time. If any action after new pause/end time is not in WAITING/READY state, an exception will be thrown.
- * Also empty string "" can be used to reset pause time to none.
- * Endtime/concurrency/pausetime of IGNORED Job cannot be changed.
-
-After the command is executed the job's end time, concurrency or pause time should be changed. If an already-succeeded job changes its end time, its status will become running.
-
-### Changing endtime/pausetime of a Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value pausetime=2011-12-01T05:00Z
-```
-
-The `change` option changes a bundle job that is not in `KILLED` status.
-
-Valid value names are:
-
- * pausetime: the pause time of the bundle job.
- * endtime: the end time of the bundle job.
-
-Repeated value names are not allowed. An empty string "" can be used to reset pause time to none. New end time should not be before job's kickoff time.
-
-Bundle will execute pause/end date change command on each coordinator job. Refer conditions and usage section of coordinator change command for more details [Coordinator job change command](DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job).
-
-### Rerunning a Workflow Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -config job.properties -rerun 14-20090525161321-oozie-joe
-```
-
-The `rerun` option reruns a completed ( `SUCCEEDED`, `FAILED` or `KILLED` ) job skipping the specified nodes.
-
-The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the `-config` option.
-
-The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
-specified path must be an HDFS path.
-
-The list of nodes to skipped must be provided in the `oozie.wf.rerun.skip.nodes` property separated by commas.
-
-After the command is executed the job will be in `RUNNING` status.
-
-Refer to the [Rerunning Workflow Jobs](DG_WorkflowReRun.html) for details on rerun.
-
-### Rerunning a Coordinator Action or Multiple Actions
-
-Example:
-
-
-```
-$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>]
-[-action 1, 3-4, 7-40] (-action or -date is required to rerun.)
-[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
-(if neither -action nor -date is given, the exception will be thrown.)
-```
-
-The `rerun` option reruns a terminated (`TIMEDOUT`, `SUCCEEDED`, `KILLED`, `FAILED`, `IGNORED`) coordinator action when coordinator job
-is not in `FAILED` or `KILLED` state.
-
-After the command is executed the rerun coordinator action will be in `WAITING` status.
-
-Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details on rerun.
-
-### Rerunning a Bundle Job
-
-Example:
-
-
-```
-$oozie job -rerun <bundle_Job_id> [-nocleanup] [-refresh]
-[-coordinator c1, c3, c4] (-coordinator or -date is required to rerun.)
-[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
-(if neither -coordinator nor -date is given, the exception will be thrown.)
-```
-
-The `rerun` option reruns coordinator actions belonging to specified coordinators within the specified date range.
-
-After the command is executed the rerun coordinator action will be in `WAITING` status.
-
-
-### Checking the Information and Status of a Workflow, Coordinator or Bundle Job or a Coordinator Action
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-joe
-.
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-Workflow Name : map-reduce-wf
-App Path : hdfs://localhost:8020/user/joe/workflows/map-reduce
-Status : SUCCEEDED
-Run : 0
-User : joe
-Group : users
-Created : 2009-05-26 05:01 +0000
-Started : 2009-05-26 05:01 +0000
-Ended : 2009-05-26 05:01 +0000
-Actions
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-Action Name Type Status Transition External Id External Status Error Code Start End
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-hadoop1 map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-```
-
-The `info` option can display information about a workflow job or coordinator job or coordinator action.
-The `info` option for a Coordinator job will retrieve the Coordinator actions ordered by nominal time. However, the `info` command may timeout if the number of Coordinator actions are very high. In that case, `info` should be used with `offset` and `len` option.
-
-The `offset` and `len` option should be used for pagination. offset determines the start offset of the action
-returned among all the actions that matched the filter criteria. len determines number of actions to be returned.
-
-The `localtime` option displays times in local time, if not specified times are displayed in GMT.
-
-The `filter` option can be used to filter coordinator actions based on some criteria.
-The filter option syntax is: `<key><comparator><value>[;<key><comparator><value>]*`.
-(Note escape `\` needed before semicolon to specify multiple names for filter in shell)
-key: status or nominalTime
-comparator: `, !`, <, <`, >, >`
-value: valid status like SUCCEEDED, KILLED, RUNNING etc. Only ` and !` apply for status
-value for nominalTime is valid date of the format yyyy-MM-dd'T'HH:mm'Z' (like 2014-06-01T00:00Z)
-
-Multiple values must be specified as different name value pairs. The query is formed by doing AND of all conditions,
-with the exception of = which uses OR if there are multiple values for the same key. For example,
-filter 'status`RUNNING;status`WAITING;nominalTime>`2014-06-01T00:00Z' maps to query (status ` RUNNING OR status =
-WAITING) AND nominalTime >` 2014-06-01T00:00Z which returns all waiting or running actions with nominalTime >`
-2014-06-01T00:00Z.
-
-Currently, the filter option can be used only with an `info` option on Coordinator job.
-
-The `verbose` option gives more detailed information for all the actions, if checking for workflow job or coordinator job.
-An example below shows how the `verbose` option can be used to gather action statistics information for a job:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -info 0000001-111219170928042-oozie-para-W@mr-node -verbose
-ID : 0000001-111219170928042-oozie-para-W@mr-node
-------------------------------------------------------------------------------------------------------------------------------------
-Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201112191708_0006
-Error Code : -
-Error Message : -
-External ID : job_201112191708_0006
-External Status : SUCCEEDED
-Name : mr-node
-Retries : 0
-Tracker URI : localhost:8021
-Type : map-reduce
-Started : 2011-12-20 01:12
-Status : OK
-Ended : 2011-12-20 01:12
-External Stats : {"org.apache.hadoop.mapred.JobInProgress$Counter":{"TOTAL_LAUNCHED_REDUCES":1,"TOTAL_LAUNCHED_MAPS":1,"DATA_LOCAL_MAPS":1},"ACTION_TYPE":"MAP_REDUCE","FileSystemCounters":{"FILE_BYTES_READ":1746,"HDFS_BYTES_READ":1409,"FILE_BYTES_WRITTEN":3524,"HDFS_BYTES_WRITTEN":1547},"org.apache.hadoop.mapred.Task$Counter":{"REDUCE_INPUT_GROUPS":33,"COMBINE_OUTPUT_RECORDS":0,"MAP_INPUT_RECORDS":33,"REDUCE_SHUFFLE_BYTES":0,"REDUCE_OUTPUT_RECORDS":33,"SPILLED_RECORDS":66,"MAP_OUTPUT_BYTES":1674,"MAP_INPUT_BYTES":1409,"MAP_OUTPUT_RECORDS":33,"COMBINE_INPUT_RECORDS":0,"REDUCE_INPUT_RECORDS":33}}
-External ChildIDs : null
-------------------------------------------------------------------------------------------------------------------------------------
-```
-
-The two fields External Stats and External ChildIDs display the action statistics information (that includes counter information in case of MR action and PigStats information in case of a pig action) and child ids of the given job.
-
-Note that the user can turn on/off External Stats by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in workflow.xml. By default, it is set to false (not to collect External Stats). External ChildIDs will always be stored.
-
-### Listing all the Workflows for a Coordinator Action
-
-A coordinator action kicks off different workflows for its original run and all subsequent reruns.
-Getting a list of those workflow ids is a useful tool to keep track of your actions' runs and
-to go debug the workflow job logs if required. Along with ids, it also lists their statuses,
-and start and end times for quick reference.
-
-This is achieved by using the Coordinator Action info command and specifying a flag **`allruns`**
-along with the `info` command.
-
-
-```
-$ oozie job -info 0000001-111219170928042-oozie-joe-C@1 -allruns -oozie http://localhost:11000/oozie
-.
-Job ID Status Started Ended
-.----------------------------------------------------------------------------------------------------
-0000001-140324163709596-oozie-joe-W SUCCEEDED 2014-03-24 23:40 GMT 2014-03-24 23:40 GMT
-.----------------------------------------------------------------------------------------------------
-0000000-140324164318985-oozie-joe-W SUCCEEDED 2014-03-24 23:44 GMT 2014-03-24 23:44 GMT
-.----------------------------------------------------------------------------------------------------
-0000001-140324164318985-oozie-joe-W SUCCEEDED 2014-03-24 23:44 GMT 2014-03-24 23:44 GMT
-.----------------------------------------------------------------------------------------------------
-```
-
-### Listing all retry attempts of a workflow action
-
-When retry-max is specified for an action in the workflow definition, and there is a failure, it will be retried till it succeeds or retry-max attempts are exhausted. To get information on all the retry attempts, `-retries` command can be used.
-
-
-```
-$ oozie job -retries 0000000-161212175234862-oozie-puru-W@pig-node -oozie http://localhost:11000/oozie
-
-ID : 0000000-161212175234862-oozie-puru-W@pig-node
-------------------------------------------------------------------------------------------------------------------------------------
-Attempt : 1
-Start Time : Tue, 13 Dec 2016 01:56:24 GMT
-End Time : Tue, 13 Dec 2016 01:56:27 GMT
-Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
-------------------------------------------------------------------------------------------------------------------------------------
-Attempt : 2
-Start Time : Tue, 13 Dec 2016 01:56:24 GMT
-End Time : Tue, 13 Dec 2016 01:56:27 GMT
-Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
-------------------------------------------------------------------------------------------------------------------------------------
-Attempt : 3
-Start Time : Tue, 13 Dec 2016 01:56:24 GMT
-End Time : Tue, 13 Dec 2016 01:56:27 GMT
-Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
-------------------------------------------------------------------------------------------------------------------------------------
-$
-```
-
-
-### Checking the xml definition of a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -definition 14-20090525161321-oozie-joe
-.
-<workflow-app xmlns="uri:oozie:workflow:0.2" name="sm3-segment-2413">
- <start to="p0"/>
- <action name="p0">
- </action>
- <end name="end"/>
-</workflow-app>
-
-```
-
-### Checking the server logs of a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-
-$ oozie job -oozie http://localhost:11000/oozie -log 14-20090525161321-oozie-joe
-
-```
-
-### Checking the server error logs of a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -errorlog 0000000-150121110331712-oozie-puru-B
-2015-01-21 11:33:29,090 WARN CoordSubmitXCommand:523 - SERVER[-] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150121110331712-oozie-puru-B] ACTION[] SAXException :
-org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-type.2.4.a: Invalid content was found starting with element 'concurrency'. One of '{"uri:oozie:coordinator:0.2":controls, "uri:oozie:coordinator:0.2":datasets, "uri:oozie:coordinator:0.2":input-events, "uri:oozie:coordinator:0.2":output-events, "uri:oozie:coordinator:0.2":action}' is expected.
- at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
- at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
- at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
- at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
- at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
- at org.apache.xerces.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(Unknown Source)
- at org.apache.xerces.impl.xs.XMLSchemaValidator.reportSchemaError(Unknown Source)
- at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source)
- at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source)
- at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
- at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
- at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
- at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(Unknown Source)
- at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(Unknown Source)
-```
-
-
-### Checking the audit logs of a Workflow, Coordinator or Bundle Job
-
-Example:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -auditlog 0000000-150322000230582-oozie-puru-C
-2015-03-22 00:04:35,494 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 00:05:13,823 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 00:06:59,561 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 23:22:20,012 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-```
-
-### Checking the server logs for particular actions of a Coordinator Job
-
-Example:
-
-
-```
-
-$ oozie job -log <coord_job_id> [-action 1, 3-4, 7-40] (-action is optional.)
-
-```
-
-### Filtering the server logs with logfilter options
-
-User can provide multiple option to filter logs using -logfilter opt1=val1;opt2=val1;opt3=val1. This can be used to fetch only just logs of interest faster as fetching Oozie server logs is slow due to the overhead of pattern matching.
-Available options are:
-
- * recent: Specify recent hours/min of logs to scan. The recent offset specified is taken relative to the `end` time specified, job end time or the current system time if the job is still running in that order of precedence. For eg: recent=3h or recent=30m will fetch logs starting 3 hours/30 minutes before the end time and up to the end time. H/h is used to denote hour and M/m is used to denote minutes. If no suffix is specified default is hours.
- * start: Start time for scanning logs. Default is start time of the job. User can provide a valid date or offset similar to `recent` option. Valid date formats are "yyyy-MM-dd'T'HH:mm'Z'" and "yyyy-MM-dd HH:mm:ss,SSS". When an offset is specified, it is calculated relative to the start time of the job. For eg: start=2h will fetch logs starting 2 hours after the job was started.
- * end: End time for scanning logs. Default is end time of the job or current system time if the job is still running. User can provide a valid date or offset similar to `start` option. When an offset is specified, it is calculated relative to start time i.e job start time . For eg: end=2h will fetch logs from start time and start time plus 2 hours.
- * loglevel : Multiple log levels separated by "|" can be specified. Supported log levels are ALL, DEBUG, ERROR, INFO, TRACE, WARN, FATAL.
- * text: String to search in logs.
- * limit : Limit number of line to be searched. Log search will end when when n lines(excluding stack-trace) have been matched.
- * debug : Prints debug information on the log files, time ranges and patterns being searched for. Can be used to debug if expected logs are not shown with the filter options provided.
-
-Examples.
-Searching log with log level ERROR or WARN will only give log with Error and Warning (with stack-trace) only.
-This will be useful if job has failed and user want to find error logs with exception.
-
-```
-
-$ ./oozie job -log 0000006-140319184715726-oozie-puru-W -logfilter loglevel=WARN\;limit=3 -oozie http://localhost:11000/oozie/
-2014-03-20 10:01:52,977 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@:start:] [***0000006-140319184715726-oozie-puru-W@:start:***]Action status=DONE
-2014-03-20 10:01:52,977 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@:start:] [***0000006-140319184715726-oozie-puru-W@:start:***]Action updated in DB!
-2014-03-20 10:01:53,189 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@mr-node-1] Error starting action [mr-node-1]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: java.io.IOException: java.io.IOException: Queue "aaadefault" does not exist
- at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615)
- at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
- at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
- at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
- at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
- at java.lang.reflect.Method.invoke(Method.java:597)
- at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
- at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
- at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
- at java.security.AccessController.doPrivileged(Native Method)
- at javax.security.auth.Subject.doAs(Subject.java:394)
- at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
- at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
-Caused by: java.io.IOException: Queue "aaadefault" does not exist
- at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:433)
- at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613)
- ... 12 more
-$
-```
-
-Search with specific text and recent option.
-
-```
-$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter text=Missing\;limit=4\;recent=1h -oozie http://localhost:11000/oozie/
-2014-03-20 09:59:50,329 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/
-2014-03-20 09:59:50,330 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing.
-2014-03-20 10:02:19,087 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/
-2014-03-20 10:02:19,088 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing.
-$
-```
-
-Search example with specific date range.
-
-```
-$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter "start=2014-03-20 10:00:57,063;end=2014-03-20 10:10:57,063" -oozie http://localhost:11000/oozie/
-2014-03-20 10:00:57,063 INFO CoordActionUpdateXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] Updating Coordinator action id :0000003-140319184715726-oozie-puru-C@1 status to KILLED, pending = 0
-2014-03-20 10:02:18,967 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] materialize actions for tz=Coordinated Universal Time,
- start=Thu Dec 31 18:00:00 PST 2009, end=Thu Dec 31 19:00:00 PST 2009,
- timeUnit 12,
- frequency :60:MINUTE,
- lastActionNumber 1
-2014-03-20 10:02:18,971 WARN CoordELFunctions:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] If the initial instance of the dataset is later than the current-instance specified, such as coord:current(-200) in this case, an empty string is returned. This means that no data is available at the current-instance specified by the user and the user could try modifying his initial-instance to an earlier time.
-2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] [0000003-140319184715726-oozie-puru-C]: all actions have been materialized, set pending to true
-2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] Coord Job status updated to = RUNNING
-```
-
-### Dryrun of Coordinator Job
-
-* This feature is only supported in Oozie 2.0 or later.
-
-Example:
-
-
-```
-
-$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
-***coordJob after parsing: ***
-<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20"
-start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles">
- <output-events>
- <data-out name="Output" dataset="DayLogs">
- <dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
- <uri-template>hdfs://localhost:8020/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template>
- </dataset>
- <instance>${coord:current(0)}</instance>
- </data-out>
- </output-events>
- <action>
- </action>
-</coordinator-app>
-
-***actions for instance***
-***total coord actions is 1 ***
-------------------------------------------------------------------------------------------------------------------------------------
-coordAction instance: 1:
-<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20"
-start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles">
- <output-events>
- <data-out name="Output" dataset="DayLogs">
- <uris>hdfs://localhost:8020/user/angeloh/coord_examples/2009/03/06</uris>
- <dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
- <uri-template>hdfs://localhost:8020/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template>
- </dataset>
- </data-out>
- </output-events>
- <action>
- </action>
-</coordinator-app>
-------------------------------------------------------------------------------------------------------------------------------------
-
-```
-
-The `dryrun` option tests running a coordinator job with given job properties and does not create the job.
-
-The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the `-config` option.
-
-The coordinator application path must be specified in the file with the `oozie.coord.application.path` property. The
-specified path must be an HDFS path.
-
-### Dryrun of Workflow Job
-
-* This feature is only supported in Oozie 3.3.2 or later.
-
-Example:
-
-
-```
-
-$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
-OK
-
-```
-
-The `dryrun` option tests running a workflow job with given job properties and does not create the job.
-
-The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the `-config` option.
-
-The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
-specified path must be an HDFS path.
-
-If the workflow is accepted (i.e. Oozie is able to successfully read and parse it), it will return `"OK"`; otherwise, it will return
-an error message describing why it was rejected.
-
-### Dryrun of Bundle Job
-
-* This feature is only supported in Oozie 5.1 or later.
-
-Example:
-
-
-```
-
-$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
-***Bundle job after parsing: ***
-<bundle-app xmlns="uri:oozie:bundle:0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="My Bundle">
- <parameters>
- <property>
- <name>oozie.use.system.libpath</name>
- <value>true</value>
- </property>
- </parameters>
- <controls>
- <kick-off-time>2017-02-14T00:13Z</kick-off-time>
- </controls>
- <coordinator name="My Coordinator-0">
- <app-path>hdfs://localhost:8020/user/hue/oozie/deployments/_admin_-oozie-24-1487060026.39</app-path>
- <configuration>
- <property>
- <name>wf_application_path</name>
- <value>hdfs://localhost:8020/user/hue/oozie/workspaces/hue-oozie-1486786607.01</value>
- </property>
- <property>
- <name>start_date</name>
- <value>2017-02-14T08:12Z</value>
- </property>
- <property>
- <name>end_date</name>
- <value>2017-02-21T08:12Z</value>
- </property>
- </configuration>
- </coordinator>
-</bundle-app>
-
-```
-
-The `dryrun` option tests running a bundle job with given job properties and does not create the job.
-
-The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the `-config` option.
-
-If the bundle job is accepted (i.e. Oozie is able to successfully read and parse it), it will return the parsed bundle job in xml;
-otherwise, it will return an error message describing why it was rejected.
-
-### Updating coordinator definition and properties
-Existing coordinator definition will be replaced by new definition. The refreshed coordinator would keep the same coordinator ID, state, and coordinator actions.
-All created coord action(including in WAITING) will use old configuration.
-One can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action
-
-Update command also verifies coordinator definition like submit command, if there is any issue with definition, update will fail.
-Update command with -dryrun will show coordinator definition and properties differences.
-Config option is optional, if not specified existing coordinator property is used to find coordinator path.
-
-Update command doesn't allow update of coordinator name, frequency, start time, end time and timezone and will fail on an attempt to change any of them. To change end time of coordinator use the `-change` command.
-
-To change the entire XML for a running coordinator, hdfs path for the new XML can be specified
-as `oozie.coord.application.path` in job.properties. Then, use `-config job.properties` in the update command.
-
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -config job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun
-.
-**********Job definition changes**********
-@@ -3,8 +3,8 @@
- <concurrency>1</concurrency>
- </controls>
- <input-events>
-- <data-in name="input" dataset="raw-logs">
-- <dataset name="raw-logs" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
-+ <data-in name="input" dataset="raw-logs-rename">
-+ <dataset name="raw-logs-rename" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
- <uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
- <done-flag />
- </dataset>
-**********************************
-**********Job conf changes**********
-@@ -8,10 +8,6 @@
- <value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
- </property>
- <property>
-- <name>old</name>
-- <value>test</value>
-- </property>
-- <property>
- <name>user.name</name>
- <value>purushah</value>
- </property>
-@@ -28,6 +24,10 @@
- <value>hdfs://localhost:9000</value>
- </property>
- <property>
-+ <name>adding</name>
-+ <value>new</value>
-+ </property>
-+ <property>
- <name>jobTracker</name>
- <value>localhost:9001</value>
- </property>
-**********************************
-```
-
-
-### Ignore a Coordinator Job
-
-Example:
-
-
-```
-$oozie job -ignore <coord_Job_id>
-```
-
-The `ignore` option changes a coordinator job in `KILLED`, `FAILED` to `IGNORED` state.
-When a coordinator job in a bundle is in `IGNORED` state, the coordinator job doesn't impact the state of the bundle job.
-For example, when a coordinator job in a bundle failed and afterwards ignored, the bundle job becomes `SUCCEEDED` instead of `DONEWITHERROR` as long as other coordinator jobs in the bundle succeeded.
- A ignored coordinator job can be changed to `RUNNING` using -change command.
- Refer to the [Coordinator job change command](DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job) for details.
-
-### Ignore a Coordinator Action or Multiple Coordinator Actions
-
-Example:
-
-
-```
-$oozie job -ignore <coord_Job_id> -action 1,3-4,7-40
-```
-The `ignore` option changes a coordinator action(s) in terminal state (`KILLED`, `FAILED`, `TIMEDOUT`) to `IGNORED` state, while not changing the state of the coordinator job.
-When a coordinator action is in `IGNORED` state, the action doesn't impact the state of a coordinator job.
-For example, when a coordinator action failed and afterwards ignored, a coordinator job becomes `SUCCEEDED` instead of `DONEWITHERROR` as long
- as other coordinator actions succeeded.
-
-A ignored coordinator action can be rerun using -rerun command.
-Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details.
-When a workflow job of a ignored coordinator action is rerun, the coordinator action becomes `RUNNING` state.
-
-### Polling an Oozie job
-
-This command allows polling the Oozie server for an Oozie job until it reaches a completed status (e.g. `SUCCEEDED`, `KILLED`, etc).
-
-Example:
-
-
-```
-$ oozie job -poll <job_id> -interval 10 -timeout 60 -verbose
-.
-RUNNING
-RUNNING
-RUNNING
-SUCCEEDED
-```
-
-The `-poll` argument takes a valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.
-
-All other arguments are optional:
-
- * `verbose` will cause the job status to be printed at each poll; otherwise, there will be no output
- * `interval` allows specifying the polling interval in minutes (default is 5)
- * `timeout` allows specifying the timeout in minutes (default is 30 minutes); negative values indicate no timeout
-
-### Changing job SLA definition and alerting
- * slaenable command can be used to enable job sla alerts.
- * sladisable command can be used to disable job sla alerts.
- * slachange command can be used to change sla job definition.
- * Supported parameters for sla change command are should-start, should-end and max-duration. Please specify the value in single quotes instead of double quotes in command line to avoid bash interpreting braces in EL functions and causing error.
- * All sla commands takes -action or -date parameter. For bundle jobs additional -coordinator (coord_name/id) parameter can be passed. Sla change command need extra parameter -value to specify new sla definition.
- * Sla commands without -action or -date parameter is applied to all non terminated actions and all future actions.
- * Sla commands with -action or -date parameter will be applied to only non terminated actions.
-
- Eg.
-
-```
- $oozie job -slaenable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z]
- $oozie job -sladisable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z]
- $oozie job -slachange <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] -value 'sla-max-duration=${10 ** MINUTES};sla-should-end=${30 ** MINUTES};sla-max-duration=${30 * MINUTES}'
- $oozie job -slaenable <bundle_job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] [-coordinator <List_of_coord_names/ids]
-```
-
-### Getting missing dependencies of coordinator action(s)
- * Coordination action id can be specified directly for getting missing dependencies of a single action.
- * To get information on multiple actions, either -action or -date option can be specified with the coordinator job id.
- * missingdeps command doesn't recompute dependencies. It list missing dependencies which were last computed.
- * Oozie checks missing dependencies sequentially, and it will stop on first missing dependency. `Blocked On` is the first missing dependency for action. So, there could be a chance that Oozie will report some missing dependencies, but it might be present. To resolve the waiting issue, one should fix the blockedOn missing dependency.
- * For input logic, missingdeps command doesn't compute input-logic expression. It will report everything which is missing or not computed.
-
-```
-$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C -action 1
-$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C@1
-.
-CoordAction : 1
-Blocked on : hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS
-.
-Dataset : input-1
-Pending Dependencies :
- hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/01/00/_SUCCESS
-.
-Dataset : input-2
-Pending Dependencies :
- hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/01/00/_SUCCESS
- hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/40/_SUCCESS
- hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/20/_SUCCESS
- hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS
-$
-```
-
-### Checking a workflow definition generated by a Fluent Job API jar file
-
-Since Oozie 5.1.0.
-
-Generate a workflow definition given the Fluent Job API jar file supplied at command line, and check for its correctness.
-
-**Preconditions:**
-
- * the Fluent Job API jar file has to be present and readable by the current user at the local path provided
- * the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
- definition is written there
-
-If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
-
-For more information what an Fluent Job API jar file is, how to build it etc.,
-refer to [Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
-
-**Example:**
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -validatejar /tmp/workflow-api-jar.jar
-Valid workflow-app
-```
-
-**Example (verbose):**
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -validatejar /tmp/workflow-api-jar.jar -verbose
-Checking API jar:/tmp/workflow-api-jar.jar
-Loading API jar /tmp/workflow-api-jar.jar
-Workflow job definition generated from API jar:
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" ... name="shell-example">
-...
-</workflow:workflow-app>
-.
-API jar is written to /tmp/workflow1876390751841950810.xml
-Servlet response is:
-Valid workflow-app
-API jar is valid.
-```
-
-### Submitting a workflow definition generated by a Fluent Job API jar file
-
-Since Oozie 5.1.0.
-
-Generate a workflow definition given the Fluent Job API jar file supplied at command line, write it to a provided or generated HDFS
-location, and submit.
-
-**Preconditions:**
-
- * all the parameters that are present in the workflow definition have to be supplied either as command line parameters or via
- `job.properties` passed along with the `-config` option
- * the Fluent Job API jar file has to be present and readable by the current user at the local path provided
- * the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
- definition is written there
- * the HDFS folder either given by `-Doozie.wf.application.path` command line parameter, or in its absence contained by
- `oozie-site.xml#oozie.client.jobs.application.generated.path` has to be writable by the current user
-
-If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
-
-For more information what an Fluent Job API jar file is, how to build it etc., refer to
-[Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
-
-**Example:**
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -submitjar /tmp/workflow-api-jar.jar -config /tmp/job.properties
-job: 0000009-180107110323219-oozie-oozi-W
-```
-
-**Example (verbose):**
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -submitjar /tmp/workflow-api-jar.jar -config /tmp/job.properties -verbose
-Submitting a job based on API jar: /tmp/workflow-api-jar.jar
-Loading API jar /tmp/workflow-api-jar.jar
-Workflow job definition generated from API jar:
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" ... name="shell-example">
-...
-</workflow:workflow-app>
-.
-job: 0000010-180107110323219-oozie-oozi-W
-Job based on API jar submitted successfully.
-```
-
-### Running a workflow definition generated by a Fluent Job API jar file
-
-Since Oozie 5.1.0.
-
-Generate a workflow definition given the Fluent Job API jar file supplied at command line, write it to a provided or generated HDFS
-location, submit and run.
-
-**Preconditions:**
-
- * all the parameters that are present in the workflow definition have to be supplied either as command line parameters or via
- `job.properties` passed along with the `-config` option
- * the Fluent Job API jar file has to be present and readable by the current user at the local path provided
- * the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
- definition is written there
- * the HDFS folder either given by `-Doozie.wf.application.path` command line parameter, or in its absence contained by
- `oozie-site.xml#oozie.client.jobs.application.generated.path` has to be writable by the current user
-
-If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
-
-For more information what an Fluent Job API jar file is, how to build it etc., refer to
-[Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
-
-**Example:**
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -runjar /tmp/workflow-api-jar.jar -config /tmp/job.properties
-job: 0000011-180107110323219-oozie-oozi-W
-```
-
-**Example (verbose):**
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -runjar /tmp/workflow-api-jar.jar -config /tmp/job.properties -verbose
-Submitting a job based on API jar: /tmp/workflow-api-jar.jar
-Loading API jar /tmp/workflow-api-jar.jar
-Workflow job definition generated from API jar:
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" ... name="shell-example">
-...
-</workflow:workflow-app>
-.
-job: 0000010-180107110323219-oozie-oozi-W
-Job based on API jar run successfully.
-```
-
-## Jobs Operations
-
-### Checking the Status of multiple Workflow Jobs
-
-Example:
-
-
-```
-$ oozie jobs -oozie http://localhost:11000/oozie -localtime -len 2 -filter status=RUNNING
-.
-Job Id Workflow Name Status Run User Group Created Started Ended
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-4-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:34 +0530 2009-05-27 15:34 +0530 -
-0-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:15 +0530 2009-05-27 15:15 +0530 -
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-```
-
-The `jobs` sub-command will display information about multiple jobs.
-
-The `offset` and `len` option specified the offset and number of jobs to display, default values are `1` and `100`
-respectively.
-
-The `localtime` option displays times in local time, if not specified times are displayed in GMT.
-
-The `verbose` option gives more detailed information for each job.
-
-A filter can be specified after all options.
-
-The `filter`option syntax is: `[NAME=VALUE][;NAME=VALUE]*`.
-
-Valid filter names are:
-
- * text: any text that might be a part of application name or a part of user name or a complete job ID
- * name: the workflow application name from the workflow definition.
- * user: the user that submitted the job.
- * group: the group for the job.
- * status: the status of the job.
- * startcreatedtime: the start of time window in specifying createdtime range filter.
- * endcreatedtime: the end of time window in specifying createdtime range filter
- * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
-
-The query will do an AND among all the filter names. The query will do an OR among all the filter values for the same
-name. Multiple values must be specified as different name value pairs.
-
-startCreatedTime and endCreatedTime should be specified either in **ISO8601 (UTC)** format (**yyyy-MM-dd'T'HH:mm'Z'**) or a offset value in days or hours from the current time. For example, -2d means the current time - 2 days. -3h means the current time - 3 hours, -5m means the current time - 5 minutes
-
-### Checking the Status of multiple Coordinator Jobs
-
-* This feature is only supported in Oozie 2.0 or later.
-
-Example:
-
-
-```
-$ oozie jobs -oozie http://localhost:11000/oozie -jobtype coordinator
-.
-Job ID App Name Status Freq Unit Started Next Materialized
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-0004165-100531045722929-oozie-wrkf-C smaggs-xaggsptechno-coordinator SUCCEEDED 1440 MINUTE 2010-05-27 00:00 2010-05-29 00:00
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-0003823-100531045722929-oozie-wrkf-C coordcal2880minutescurrent SUCCEEDED 2880 MINUTE 2010-02-01 16:30 2010-02-05 16:30
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-```
-
-The `jobtype` option specified the job type to display, default value is 'wf'. To see the coordinator jobs, value is 'coordinator'.
-
-Valid filter names are:
-
- * name: the workflow application name from the workflow definition.
- * user: the user that submitted the job.
- * group: the group for the job.
- * status: the status of the job.
- * frequency: the frequency of the Coordinator job.
- * unit: the time unit. It can take one of the following four values: months, days, hours or minutes. Time unit should be added only when frequency is specified.
- * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
-
-### Checking the Status of multiple Bundle Jobs
-
-* This feature is only supported in Oozie 3.0 or later.
-
-Example:
-
-
-```
-$ oozie jobs -oozie http://localhost:11000/oozie -jobtype bundle
-Job ID Bundle Name Status Kickoff Created User Group
-.------------------------------------------------------------------------------------------------------------------------------------
-0000027-110322105610515-oozie-chao-B BUNDLE-TEST RUNNING 2012-01-15 00:24 2011-03-22 18:07 joe users
-.------------------------------------------------------------------------------------------------------------------------------------
-0000001-110322105610515-oozie-chao-B BUNDLE-TEST RUNNING 2012-01-15 00:24 2011-03-22 18:06 joe users
-.------------------------------------------------------------------------------------------------------------------------------------
-0000000-110322105610515-oozie-chao-B BUNDLE-TEST DONEWITHERROR2012-01-15 00:24 2011-03-22 17:58 joe users
-.------------------------------------------------------------------------------------------------------------------------------------
-```
-
-The `jobtype` option specified the job type to display, default value is 'wf'. To see the bundle jobs, value is 'bundle'.
-
-### Bulk kill, suspend or resume multiple jobs
-
-Example:
-
-
-```
-$ oozie jobs -oozie http://localhost:11000/oozie -kill|-suspend|-resume -filter name=cron-coord -jobtype coordinator
-The following jobs have been killed|suspended|resumed
-Job ID App Name Status Freq Unit Started Next Materialized
-.------------------------------------------------------------------------------------------------------------------------------------
-0000005-150224141553231-oozie-bzha-C cron-coord KILLED 10 MINUTE 2015-02-24 22:05 GMT 2015-02-24 23:05 GMT
-.------------------------------------------------------------------------------------------------------------------------------------
-0000001-150224141553231-oozie-bzha-C cron-coord KILLED 10 MINUTE 2015-02-24 22:00 GMT 2015-02-24 23:00 GMT
-.------------------------------------------------------------------------------------------------------------------------------------
-0000000-150224141553231-oozie-bzha-C cron-coord KILLED 10 MINUTE 2015-02-25 22:00 GMT -
-.------------------------------------------------------------------------------------------------------------------------------------
-```
-
-The above command will kill, suspend, or resume all the coordinator jobs with name of "cron-coord" starting with offset 1
-to 50.
-The `jobs` sub-command will bulk modify all the jobs that satisfy the filter, len, offset, and jobtype options when adding
-a -kill|-suspend|-resume option. Another way to think about is the subcommand works to modify all the jobs that will be
-displayed if the write option(kill|suspend|resume) is not there.
-
-The `offset` and `len` option specified the offset and number of jobs to be modified, default values are `1` and `50`
-respectively.
-The `jobtype` option specifies the type of jobs to be modified, be it "wf", "coordinator" or "bundle". default value is "wf".
-
-A filter can be specified after all options.
-
-The `filter`option syntax is: `[NAME=VALUE][;NAME=VALUE]*`.
-
-Valid filter names are:
-
- * name: the workflow application name from the workflow definition.
- * user: the user that submitted the job.
- * group: the group for the job.
- * status: the status of the job.
- * frequency: the frequency of the Coordinator job.
- * unit: the time unit. It can take one of the following four values: months, days, hours or minutes. Time unit should be added only when frequency is specified.
- * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
-
-The query will do an AND among all the filter names. The query will do an OR among all the filter values for the same
-name. Multiple values must be specified as different name value pairs.
-
-The following example shows how to suspend the first 20 bundle jobs whose name is "bundle-app":
-
-
-```
-$ oozie jobs -oozie http://localhost:11000/oozie -suspend -filter name=bundle-app -jobtype bundle -len 20
-```
-
-### Bulk monitoring for jobs launched via Bundles
-
-* This command-line query helps to directly query for a bulk of jobs using a set of rich filters.
-The jobs need to have a parent **Bundle**, and this performs a deep query to provide results about all the workflows that satisfy your filters.
-These results display relevant job-ids, app-names, and error message (if any) and are most helpful when you need to monitor a bulk of jobs and get a gist,
-while avoiding typing multiple queries.
-
-This feature is only supported in Oozie 3.3 or later.
-
-Example 1:
-
-
-```
-$ oozie jobs -oozie http://localhost:11000/oozie -bulk bundle=bundle-app-1
-.
-Bundle Name Bundle ID Coord Name Coord Action ID External ID Status Created Time Error Message
-.-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-bundle-app-1 0000000-130408151805946-oozie-chit-B coord-1 0000001-130408151805946-oozie-chit-C@1 0000002-130408151805946-oozie-chit-W KILLED 2013-04-08 22:20 GMT null
-.-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-```
-
-Example 2: This example illustrates giving multiple arguments and -verbose option.
-
-_NOTE: The filter string after -bulk should be enclosed within quotes_
-
-
-```
-.
-$ oozie jobs -oozie http://localhost:11000/oozie -bulk 'bundle=bundle-app-2;actionstatus=SUCCEEDED' -verbose
-.
-Bundle Name : bundle-app-2
-.------------------------------------------------------------------------------------------------------------------------------------
-Bundle ID : 0000000-130422184245158-oozie-chit-B
-Coordinator Name : coord-1
-Coord Action ID : 0000001-130422184245158-oozie-chit-C@1
-Action Status : SUCCEEDED
-External ID : 0000002-130422184245158-oozie-chit-W
-Created Time : 2013-04-23 01:43 GMT
-User : user_xyz
-Error Message : -
-.------------------------------------------------------------------------------------------------------------------------------------
-```
-
-You can type 'help' to view usage for the CLI options and filters available. Namely:
-
- * bundle: Bundle app-name (mandatory)
- * coordinators: multiple, comma-separated Coordinator app-names. By default, if none specified, all coordinators pertaining to the bundle are included
- * actionstatus: status of jobs (default job-type is coordinator action aka workflow job) you wish to filter on. Default value is (KILLED,FAILED)
- * startcreatedtime/endcreatedtime: specify boundaries for the created-time. Only jobs created within this window are included.
- * startscheduledtime/endscheduledtime: specify boundaries for the nominal-time. Only jobs scheduled to run within this window are included.
-
-
-```
-$ oozie jobs <OPTIONS> : jobs status
- -bulk <arg> key-value pairs to filter bulk jobs response. e.g.
- bundle=<B>\;coordinators=<C>\;actionstatus=<S>\;
- startcreatedtime=<SC>\;endcreatedtime=<EC>\;
- startscheduledtime=<SS>\;endscheduledtime=<ES>\;
- coordinators and actionstatus can be multiple comma
- separated values. "bundle" and "coordinators" are 'names' of those jobs.
- Bundle name is mandatory, other params are optional
-```
-
-Similar to the usual jobs filter, different filter arguments here should be separated by semicolon (;).
-
-## Admin Operations
-
-### Checking the Status of the Oozie System
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -status
-.
-Safemode: OFF
-```
-
-It returns the current status of the Oozie system.
-
-### Changing the Status of the Oozie System
-
-* This feature is only supported in Oozie 2.0 or later.
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -systemmode [NORMAL|NOWEBSERVICE|SAFEMODE]
-.
-Safemode: ON
-```
-
-It returns the current status of the Oozie system.
-
-### Displaying the Build Version of the Oozie System
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -version
-.
-Oozie server build version: 2.0.2.1-0.20.1.3092118008--
-```
-
-It returns the Oozie server build version.
-
-### Displaying the queue dump of the Oozie System
-
-* This feature is for administrator debugging purpose
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -queuedump
-.
-[Server Queue Dump]:
-(coord_action_start,1),(coord_action_start,1),(coord_action_start,1)
-(coord_action_ready,1)
-(action.check,2)
-
-```
-
-It returns the Oozie server current queued commands.
-
-### Displaying the list of available Oozie Servers
-
-Example:
-
-
-```
-$ oozie admin -oozie http://hostA:11000/oozie -servers
-hostA : http://hostA:11000/oozie
-hostB : http://hostB:11000/oozie
-hostC : http://hostC:11000/oozie
-```
-
-It returns a list of available Oozie Servers. This is useful when Oozie is configured for [High Availability](AG_Install.html#HA); if
-not, it will simply return the one Oozie Server.
-
-### Displaying the Oozie server configuration
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -configuration
-local.realm : LOCALHOST
-oozie.JobCommand.job.console.url : http://localhost:11000/oozie?job=
-oozie.action.fs.glob.max : 1000
-oozie.action.jobinfo.enable : false
-oozie.action.launcher.mapreduce.job.ubertask.enable : true
-oozie.action.launcher.yarn.timeline-service.enabled : false
-oozie.action.mapreduce.uber.jar.enable : false
-oozie.action.max.output.data : 2048
-oozie.action.retries.max : 3
-...
-```
-
-It returns a list of the configuration properties and values from oozie-site.xml and oozie-default.xml being used by the Oozie
-server.
-
-### Displaying the Oozie server OS environment
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -osenv
-...
-JETTY_OPTS : -Doozie.home.dir=/Users/asasvari/dev/oozie -Doozie.config.dir=/Users/asasvari/dev/oozie/conf -Doozie.log.dir=/Users/asasvari/dev/oozie/logs -Doozie.data.dir=/Users/asasvari/dev/oozie/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Djava.library.path= -cp /Users/asasvari/dev/oozie/embedded-oozie-server/**:/Users/asasvari/dev/oozie/embedded-oozie-server/dependency/**:/Users/asasvari/dev/oozie/lib/**:/Users/asasvari/dev/oozie/libtools/**:/Users/asasvari/dev/oozie/embedded-oozie-server
-JETTY_OUT : /Users/asasvari/dev/oozie/logs/jetty.out
-JETTY_PID_FILE : /Users/asasvari/dev/oozie/embedded-oozie-server/oozie.pid
-OOZIE_CONFIG : /Users/asasvari/dev/oozie/conf
-OOZIE_CONFIG_FILE : oozie-site.xml
-OOZIE_DATA : /Users/asasvari/dev/oozie/data
-OOZIE_HOME : /Users/asasvari/dev/oozie
-OOZIE_LOG : /Users/asasvari/dev/oozie/logs
-OOZIE_LOG4J_FILE : oozie-log4j.properties
-OOZIE_LOG4J_RELOAD : 10
-...
-```
-
-It returns a list of OS environment variables in the Oozie server.
-
-### Displaying the Oozie server Java system properties
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -javasysprops
-...
-oozie.config.dir : /Users/asasvari/dev/oozie/conf
-oozie.config.file : oozie-site.xml
-oozie.data.dir : /Users/asasvari/dev/oozie/data
-oozie.home.dir : /Users/asasvari/dev/oozie
-oozie.log.dir : /Users/asasvari/dev/oozie/logs
-oozie.log4j.file : oozie-log4j.properties
-oozie.log4j.reload : 10
-...
-```
-
-It returns a list of java system properties in the Oozie server.
-
-### Displaying the Oozie server Instrumentation
-
-Deprecated and by default disabled since 5.0.0.
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -instrumentation
-COUNTERS
---------
-callablequeue.executed : 1
-callablequeue.queued : 1
-commands.purge.executions : 1
-...
-
-VARIABLES
----------
-configuration.action.types : [hive, shell, :START:, :FORK:, switch, spark, ssh, hive2, pig, :END:, email, distcp, :KILL:, sub-workflow, fs, java, :JOIN:, sqoop, map-reduce]
-configuration.config.dir : /Users/rkanter/dev/oozie/conf
-configuration.config.file : /Users/rkanter/dev/oozie/conf/oozie-site.xml
-...
-
-SAMPLERS
----------
-callablequeue.queue.size : 0.0
-callablequeue.threads.active : 0.0
-jdbc.connections.active : 0.0
-...
-
-TIMERS
----------
-callablequeue.time.in.queue
- own time standard deviation : -1.0
- own average time : 1
- own max time : 1
- own min time : 1
- total time standard deviation : -1.0
- total average time : 1
- total max time : 1
- total min time : 1
- ticks : 1
-commands.purge.call
- own time standard deviation : -1.0
- own average time : 222
- own max time : 222
- own min time : 222
- total time standard deviation : -1.0
- total average time : 222
- total max time : 222
- total min time : 222
- ticks : 1
-...
-```
-
-It returns the instrumentation from the Oozie server. Keep in mind that timers and counters that the Oozie server
-hasn't incremented yet will not show up.
-
-**Note:** If Instrumentation is enabled, then Metrics is unavailable.
-
-### Displaying the Oozie server Metrics
-
-By default enabled since 5.0.0.
-
-Example:
-
-
-```
-$ oozie admin -oozie http://localhost:11000/oozie -metrics
-COUNTERS
---------
-callablequeue.executed : 1
-callablequeue.queued : 1
-commands.purge.executions : 1
-...
-
-GAUGES
-------
-configuration.action.types : [hive, shell, :START:, :FORK:, switch, spark, ssh, hive2, pig, :END:, email, distcp, :KILL:, sub-workflow, fs, java, :JOIN:, sqoop, map-reduce]
-configuration.config.dir : /Users/rkanter/dev/oozie/conf
-configuration.config.file : /Users/rkanter/dev/oozie/conf/oozie-site.xml
-...
-
-TIMERS
-------
-callablequeue.time.in.queue.timer
- 999th percentile : 4.0
- 99th percentile : 4.0
- 98th percentile : 4.0
- 95th percentile : 4.0
- 75th percentile : 4.0
- 50th percentile : 4.0
- mean : 4.0
- max : 4.0
- min : 4.0
- count : 1
- standard deviation : 0.0
- 15 minute rate : 0.0
- 5 minute rate : 0.0
- 1 minute rate : 0.0
- mean rate : 0.0
- duration units : milliseconds
- rate units : calls/millisecond
-commands.purge.call.timer
- 999th percentile : 260.0
- 99th percentile : 260.0
- 98th percentile : 260.0
- 95th percentile : 260.0
- 75th percentile : 260.0
- 50th percentile : 260.0
- mean : 260.0
- max : 260.0
- min : 260.0
- count : 1
- standard deviation : 0.0
- 15 minute rate : 0.0
- 5 min
<TRUNCATED>
[03/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/WebServicesAPI.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/WebServicesAPI.twiki b/docs/src/site/twiki/WebServicesAPI.twiki
deleted file mode 100644
index a303802..0000000
--- a/docs/src/site/twiki/WebServicesAPI.twiki
+++ /dev/null
@@ -1,2379 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Oozie Web Services API, V1 (Workflow, Coordinator, And Bundle)
-
-The Oozie Web Services API is a HTTP REST JSON API.
-
-All responses are in `UTF-8`.
-
-Assuming Oozie is running at `OOZIE_URL`, the following web services end points are supported:
-
- * \<OOZIE_URL\>/versions
- * \<OOZIE_URL\>/v1/admin
- * \<OOZIE_URL\>/v1/job
- * \<OOZIE_URL\>/v1/jobs
- * \<OOZIE_URL\>/v2/job
- * \<OOZIE_URL\>/v2/jobs
- * \<OOZIE_URL\>/v2/admin
- * \<OOZIE_URL\>/v2/sla
-
-Documentation on the API is below; in some cases, looking at the corresponding command in the
-[Command Line Documentation](DG_CommandLineTool.html) page will provide additional details and examples. Most of the functionality
-offered by the Oozie CLI is using the WS API. If you export `OOZIE_DEBUG` then the Oozie CLI will output the WS API
-details used by any commands you execute. This is useful for debugging purposes to or see how the Oozie CLI works with the WS API.
-
-### Versions End-Point
-
-_Identical to the corresponding Oozie v0 WS API_
-
-This endpoint is for clients to perform protocol negotiation.
-
-It support only HTTP GET request and not sub-resources.
-
-It returns the supported Oozie protocol versions by the server.
-
-Current returned values are `0, 1, 2`.
-
-**Request:**
-
-
-```
-GET /oozie/versions
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-[0,1]
-```
-
-### Admin End-Point
-
-This endpoint is for obtaining Oozie system status and configuration information.
-
-It supports the following sub-resources: `status, os-env, sys-props, configuration, instrumentation, systems, available-timezones`.
-
-#### System Status
-
-_Identical to the corresponding Oozie v0 WS API_
-
-A HTTP GET request returns the system status.
-
-**Request:**
-
-
-```
-GET /oozie/v1/admin/status
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{"systemMode":NORMAL}
-```
-
-With a HTTP PUT request it is possible to change the system status between `NORMAL`, `NOWEBSERVICE`, and `SAFEMODE`.
-
-**Request:**
-
-
-```
-PUT /oozie/v1/admin/status?systemmode=SAFEMODE
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-```
-
-#### OS Environment
-
-_Identical to the corresponding Oozie v0 WS API_
-
-A HTTP GET request returns the Oozie system OS environment.
-
-**Request:**
-
-
-```
-GET /oozie/v1/admin/os-env
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- TERM: "xterm",
- JAVA_HOME: "/usr/java/latest",
- XCURSOR_SIZE: "",
- SSH_CLIENT: "::ffff:127.0.0.1 49082 22",
- XCURSOR_THEME: "default",
- INPUTRC: "/etc/inputrc",
- HISTSIZE: "1000",
- PATH: "/usr/java/latest/bin"
- KDE_FULL_SESSION: "true",
- LANG: "en_US.UTF-8",
- ...
-}
-```
-
-#### Java System Properties
-
-_Identical to the corresponding Oozie v0 WS API_
-
-A HTTP GET request returns the Oozie Java system properties.
-
-**Request:**
-
-
-```
-GET /oozie/v1/admin/java-sys-properties
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- java.vm.version: "11.0-b15",
- sun.jnu.encoding: "UTF-8",
- java.vendor.url: "http://java.sun.com/",
- java.vm.info: "mixed mode",
- ...
-}
-```
-
-#### Oozie Configuration
-
-_Identical to the corresponding Oozie v0 WS API_
-
-A HTTP GET request returns the Oozie system configuration.
-
-**Request:**
-
-
-```
-GET /oozie/v1/admin/configuration
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- oozie.service.SchedulerService.threads: "5",
- oozie.service.ActionService.executor.classes: "
- org.apache.oozie.dag.action.decision.DecisionActionExecutor,
- org.apache.oozie.dag.action.hadoop.HadoopActionExecutor,
- org.apache.oozie.dag.action.hadoop.FsActionExecutor
- ",
- oozie.service.CallableQueueService.threads.min: "10",
- oozie.service.DBLiteWorkflowStoreService.oozie.autoinstall: "true",
- ...
-}
-```
-
-#### Oozie Instrumentation
-
-_Identical to the corresponding Oozie v0 WS API_
-
-Deprecated and by default disabled since 5.0.0.
-
-A HTTP GET request returns the Oozie instrumentation information. Keep in mind that timers and counters that the Oozie server
-hasn't incremented yet will not show up.
-
-**Note:** If Instrumentation is enabled, then Metrics is unavailable.
-
-**Request:**
-
-
-```
-GET /oozie/v1/admin/instrumentation
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- timers: [
- {
- group: "db",
- data: [
- {
- ownMinTime: 2,
- ownTimeStdDev: 0,
- totalTimeStdDev: 0,
- ownTimeAvg: 3,
- ticks: 117,
- name: "update-workflow",
- ownMaxTime: 32,
- totalMinTime: 2,
- totalMaxTime: 32,
- totalTimeAvg: 3
- },
- ...
- ]
- },
- ...
- ],
- samplers: [
- {
- group: "callablequeue",
- data: [
- {
- name: "threads.active",
- value: 1.8333333333333333
- },
- {
- name: "delayed.queue.size",
- value: 0
- },
- {
- name: "queue.size",
- value: 0
- }
- ]
- },
- ...
- ],
- variables: [
- {
- group: "jvm",
- data: [
- {
- name: "max.memory",
- value: 506920960
- },
- {
- name: "total.memory",
- value: 56492032
- },
- {
- name: "free.memory",
- value: 45776800
- }
- ]
- },
- ...
- ]
-}
-```
-
-#### Oozie Metrics
-
-_Available in the Oozie v2 WS API and later_
-
-A HTTP GET request returns the Oozie metrics information. Keep in mind that timers and counters that the Oozie server
-hasn't incremented yet will not show up.
-
-
-**Note:** If Metrics is enabled, then Instrumentation is unavailable.
-
-**Note:** by default enabled since 5.0.0.
-
-**Request:**
-
-
-```
-GET /oozie/v2/admin/metrics
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- "gauges" : {
- "jvm.memory.non-heap.committed" : {
- "value" : 62590976
- },
- "oozie.mode" : {
- "value" : "NORMAL"
- },
- ...
- },
- "timers" : {
- "commands.action.end.call.timer" : {
- "mean" : 108.5,
- "p50" : 111.5,
- "p75" : 165.5,
- "p999" : 169,
- "count" : 4,
- "p95" : 169,
- "max" : 169,
- "mean_rate" : 0,
- "duration_units" : "milliseconds",
- "p98" : 169,
- "m1_rate" : 0,
- "rate_units" : "calls/millisecond",
- "m15_rate" : 0,
- "stddev" : 62.9417720330995,
- "m5_rate" : 0,
- "p99" : 169,
- "min" : 42
- },
- ...
- },
- "histograms" : {
- "callablequeue.threads.active.histogram" : {
- "p999" : 1,
- "mean" : 0.0625,
- "min" : 0,
- "p75" : 0,
- "p95" : 1,
- "count" : 48,
- "p98" : 1,
- "stddev" : 0.24462302739504083,
- "max" : 1,
- "p99" : 1,
- "p50" : 0
- },
- ...
- },
- "counters" : {
- "commands.job.info.executions" : {
- "count" : 9
- },
- "jpa.CoordJobsGetPendingJPAExecutor" : {
- "count" : 1
- },
- "jpa.GET_WORKFLOW" : {
- "count" : 10
- },
- ...
- }
-}
-```
-
-#### Version
-
-_Identical to the corresponding Oozie v0 WS API_
-
-A HTTP GET request returns the Oozie build version.
-
-**Request:**
-
-
-```
-GET /oozie/v1/admin/build-version
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{buildVersion: "3.0.0-SNAPSHOT" }
-```
-
-#### Available Time Zones
-
-A HTTP GET request returns the available time zones.
-
-**Request:**
-
-
-```
-GET /oozie/v1/admin/available-timezones
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- "available-timezones":[
- {
- "timezoneDisplayName":"SST (Pacific\/Midway)",
- "timezoneId":"Pacific\/Midway"
- },
- {
- "timezoneDisplayName":"NUT (Pacific\/Niue)",
- "timezoneId":"Pacific\/Niue"
- },
- {
- "timezoneDisplayName":"SST (Pacific\/Pago_Pago)",
- "timezoneId":"Pacific\/Pago_Pago"
- },
- {
- "timezoneDisplayName":"SST (Pacific\/Samoa)",
- "timezoneId":"Pacific\/Samoa"
- },
- {
- "timezoneDisplayName":"SST (US\/Samoa)",
- "timezoneId":"US\/Samoa"
- },
- {
- "timezoneDisplayName":"HAST (America\/Adak)",
- "timezoneId":"America\/Adak"
- },
- {
- "timezoneDisplayName":"HAST (America\/Atka)",
- "timezoneId":"America\/Atka"
- },
- {
- "timezoneDisplayName":"HST (HST)",
- "timezoneId":"HST"
- },
- ...
- ]
-}
-```
-
-#### Queue Dump
-
-A HTTP GET request returns the queue dump of the Oozie system. This is an administrator debugging feature.
-
-**Request:**
-
-
-```
-
-GET /oozie/v1/admin/queue-dump
-```
-
-#### Available Oozie Servers
-
-A HTTP GET request returns the list of available Oozie Servers. This is useful when Oozie is configured
-for [High Availability](AG_Install.html#HA); if not, it will simply return the one Oozie Server.
-
-**Request:**
-
-
-```
-GET /oozie/v2/admin/available-oozie-servers
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- "hostA": "http://hostA:11000/oozie",
- "hostB": "http://hostB:11000/oozie",
- "hostC": "http://hostC:11000/oozie",
-}
-```
-
-#### List available sharelib
-A HTTP GET request to get list of available sharelib.
-If the name of the sharelib is passed as an argument (regex supported) then all corresponding files are also listed.
-
-**Request:**
-
-
-```
-GET /oozie/v2/admin/list_sharelib
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-{
- "sharelib":
- [
- "oozie",
- "hive",
- "distcp",
- "hcatalog",
- "sqoop",
- "mapreduce-streaming",
- "pig"
- ]
-}
-```
-
-**Request:**
-
-
-```
-GET /oozie/v2/admin/list_sharelib?lib=pig*
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-
-{
- "sharelib":
- [
- {
- "pig":
- {
- "sharelibFiles":
- [
- hdfs://localhost:9000/user/purushah/share/lib/lib_20131114095729/pig/pig.jar
- hdfs://localhost:9000/user/purushah/share/lib/lib_20131114095729/pig/piggybank.jar
- ]
- }
- }
- ]
-}
-```
-
-
-#### Update system sharelib
-This webservice call makes the oozie server(s) to pick up the latest version of sharelib present
-under oozie.service.WorkflowAppService.system.libpath directory based on the sharelib directory timestamp or reloads
-the sharelib metafile if one is configured. The main purpose is to update the sharelib on the oozie server without restarting.
-
-
-**Request:**
-
-
-```
-GET /oozie/v2/admin/update_sharelib
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-[
- {
- "sharelibUpdate":{
- "host":"server1",
- "status":"Server not found"
- }
- },
- {
- "sharelibUpdate":{
- "host":"server2",
- "status":"Successful",
- "sharelibDirOld":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218",
- "sharelibDirNew":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218"
- }
- },
- {
- "sharelibUpdate":{
- "host":"server3",
- "status":"Successful",
- "sharelibDirOld":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218",
- "sharelibDirNew":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218"
- }
- }
-]
-```
-
-#### Purge Command
-
-Oozie admin purge command cleans up the Oozie Workflow/Coordinator/Bundle records based on the parameters.
-The unit for parameters is day.
-
-Purge command will delete the workflow records (wf=30) older than 30 days, coordinator records (coord=7) older than 7 days and
-bundle records (bundle=7) older than 7 days. The limit (limit=10) defines, number of records to be fetch at a time. Turn
-(oldCoordAction`true/false) `on/off= coordinator action record purging for long running coordinators. If any of the parameter is
-not provided, then it will be taken from the `oozie-default/oozie-site` configuration.
-
-**Request:**
-
-
-```
-
-GET /oozie/v2/admin/purge?wf=30&coord=7&bundle=7&limit=10&oldCoordAction=true
-
-```
-
-**Response:**
-
-
-```
-
-{
- "purge": "Purge command executed successfully"
-}
-
-```
-
-### Job and Jobs End-Points
-
-_Modified in Oozie v1 WS API_
-
-These endpoints are for submitting, managing and retrieving information of workflow, coordinator, and bundle jobs.
-
-#### Job Submission
-
-#### Standard Job Submission
-
-An HTTP POST request with an XML configuration as payload creates a job.
-
-The type of job is determined by the presence of one of the following 3 properties:
-
- * `oozie.wf.application.path` : path to a workflow application directory, creates a workflow job
- * `oozie.coord.application.path` : path to a coordinator application file, creates a coordinator job
- * `oozie.bundle.application.path` : path to a bundle application file, creates a bundle job
-
-Or, if none of those are present, the jobtype parameter determines the type of job to run. It can either be mapreduce or pig.
-
-**Request:**
-
-
-```
-POST /oozie/v1/jobs
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
- <property>
- <name>user.name</name>
- <value>bansalm</value>
- </property>
- <property>
- <name>oozie.wf.application.path</name>
- <value>hdfs://foo:8020/user/bansalm/myapp/</value>
- </property>
- ...
-</configuration>
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 201 CREATED
-Content-Type: application/json;charset=UTF-8
-.
-{
- id: "job-3"
-}
-```
-
-A created job will be in `PREP` status. If the query string parameter 'action=start' is provided in
-the POST URL, the job will be started immediately and its status will be `RUNNING`.
-
-Coordinator jobs with start time in the future they will not create any action until the start time
-happens.
-
-A coordinator job will remain in `PREP` status until it's triggered, in which case it will change to `RUNNING` status.
-The 'action=start' parameter is not valid for coordinator jobs.
-
-#### Proxy MapReduce Job Submission
-
-You can submit a Workflow that contains a single MapReduce action without writing a workflow.xml. Any required Jars or other files
-must already exist in HDFS.
-
-The following properties are required; any additional parameters needed by the MapReduce job can also be specified here:
-
- * `fs.default.name`: The NameNode
- * `mapred.job.tracker`: The JobTracker
- * `mapred.mapper.class`: The map-task classname
- * `mapred.reducer.class`: The reducer-task classname
- * `mapred.input.dir`: The map-task input directory
- * `mapred.output.dir`: The reduce-task output directory
- * `user.name`: The username of the user submitting the job
- * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
- * `oozie.proxysubmission`: Must be set to `true`
-
-**Request:**
-
-
-```
-POST /oozie/v1/jobs?jobtype=mapreduce
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://localhost:8020</value>
- </property>
- <property>
- <name>mapred.job.tracker</name>
- <value>localhost:8021</value>
- </property>
- <property>
- <name>mapred.mapper.class</name>
- <value>org.apache.oozie.example.SampleMapper</value>
- </property>
- <property>
- <name>mapred.reducer.class</name>
- <value>org.apache.oozie.example.SampleReducer</value>
- </property>
- <property>
- <name>mapred.input.dir</name>
- <value>hdfs://localhost:8020/user/rkanter/examples/input-data/text</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>hdfs://localhost:8020/user/rkanter/examples/output-data/map-reduce</value>
- </property>
- <property>
- <name>user.name</name>
- <value>rkanter</value>
- </property>
- <property>
- <name>oozie.libpath</name>
- <value>hdfs://localhost:8020/user/rkanter/examples/apps/map-reduce/lib</value>
- </property>
- <property>
- <name>oozie.proxysubmission</name>
- <value>true</value>
- </property>
-</configuration>
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 201 CREATED
-Content-Type: application/json;charset=UTF-8
-.
-{
- id: "job-3"
-}
-```
-
-#### Proxy Pig Job Submission
-
-You can submit a Workflow that contains a single Pig action without writing a workflow.xml. Any required Jars or other files must
-already exist in HDFS.
-
-The following properties are required:
-
- * `fs.default.name`: The NameNode
- * `mapred.job.tracker`: The JobTracker
- * `user.name`: The username of the user submitting the job
- * `oozie.pig.script`: Contains the pig script you want to run (the actual script, not a file path)
- * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
- * `oozie.proxysubmission`: Must be set to `true`
-
-The following properties are optional:
-
- * `oozie.pig.script.params.size`: The number of parameters you'll be passing to Pig
- required
- * `oozie.pig.script.params.n`: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
- * `oozie.pig.options.size`: The number of options you'll be passing to Pig
- * `oozie.pig.options.n`: An argument to pass to Pig, the 'n' should be an integer starting with 0 to indicate the option number
-
-The `oozie.pig.options.n` parameters are sent directly to Pig without any modification unless they start with `-D`, in which case
-they are put into the `<configuration>` element of the action.
-
-In addition to passing parameters to Pig with `oozie.pig.script.params.n`, you can also create a properties file on HDFS and
-reference it with the `-param_file` option in `oozie.pig.script.options.n`; both are shown in the following example.
-
-
-```
-$ hadoop fs -cat /user/rkanter/pig_params.properties
-INPUT=/user/rkanter/examples/input-data/text
-```
-
-**Request:**
-
-
-```
-POST /oozie/v1/jobs?jobtype=pig
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://localhost:8020</value>
- </property>
- <property>
- <name>mapred.job.tracker</name>
- <value>localhost:8021</value>
- </property>
- <property>
- <name>user.name</name>
- <value>rkanter</value>
- </property>
- <property>
- <name>oozie.pig.script</name>
- <value>
- A = load '$INPUT' using PigStorage(':');
- B = foreach A generate $0 as id;
- store B into '$OUTPUT' USING PigStorage();
- </value>
- </property>
- <property>
- <name>oozie.pig.script.params.size</name>
- <value>1</value>
- </property>
- <property>
- <name>oozie.pig.script.params.0</name>
- <value>OUTPUT=/user/rkanter/examples/output-data/pig</value>
- </property>
- <property>
- <name>oozie.pig.options.size</name>
- <value>2</value>
- </property>
- <property>
- <name>oozie.pig.options.0</name>
- <value>-param_file</value>
- </property>
- <property>
- <name>oozie.pig.options.1</name>
- <value>hdfs://localhost:8020/user/rkanter/pig_params.properties</value>
- </property>
- <property>
- <name>oozie.libpath</name>
- <value>hdfs://localhost:8020/user/rkanter/share/lib/pig</value>
- </property>
- <property>
- <name>oozie.proxysubmission</name>
- <value>true</value>
- </property>
-</configuration>
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 201 CREATED
-Content-Type: application/json;charset=UTF-8
-.
-{
- id: "job-3"
-}
-```
-
-#### Proxy Hive Job Submission
-
-You can submit a Workflow that contains a single Hive action without writing a workflow.xml. Any required Jars or other files must
-already exist in HDFS.
-
-The following properties are required:
-
- * `fs.default.name`: The NameNode
- * `mapred.job.tracker`: The JobTracker
- * `user.name`: The username of the user submitting the job
- * `oozie.hive.script`: Contains the hive script you want to run (the actual script, not a file path)
- * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
- * `oozie.proxysubmission`: Must be set to `true`
-
-The following properties are optional:
-
- * `oozie.hive.script.params.size`: The number of parameters you'll be passing to Hive
- * `oozie.hive.script.params.n`: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
- * `oozie.hive.options.size`: The number of options you'll be passing to Hive
- * `oozie.hive.options.n`: An argument to pass to Hive, the 'n' should be an integer starting with 0 to indicate the option number
-
-The `oozie.hive.options.n` parameters are sent directly to Hive without any modification unless they start with `-D`, in which case
-they are put into the `<configuration>` element of the action.
-
-**Request:**
-
-
-```
-POST /oozie/v1/jobs?jobtype=hive
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://localhost:8020</value>
- </property>
- <property>
- <name>mapred.job.tracker</name>
- <value>localhost:8021</value>
- </property>
- <property>
- <name>user.name</name>
- <value>rkanter</value>
- </property>
- <property>
- <name>oozie.hive.script</name>
- <value>
- CREATE EXTERNAL TABLE test (a INT) STORED AS TEXTFILE LOCATION '${INPUT}';
- INSERT OVERWRITE DIRECTORY '${OUTPUT}' SELECT * FROM test;
- </value>
- </property>
- <property>
- <name>oozie.hive.script.params.size</name>
- <value>2</value>
- </property>
- <property>
- <name>oozie.hive.script.params.0</name>
- <value>OUTPUT=/user/rkanter/examples/output-data/hive</value>
- </property>
- <property>
- <name>oozie.hive.script.params.1</name>
- <value>INPUT=/user/rkanter/examples/input-data/table</value>
- </property>
- <property>
- <name>oozie.libpath</name>
- <value>hdfs://localhost:8020/user/rkanter/share/lib/hive</value>
- </property>
- <property>
- <name>oozie.proxysubmission</name>
- <value>true</value>
- </property>
-</configuration>
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 201 CREATED
-Content-Type: application/json;charset=UTF-8
-.
-{
- id: "job-3"
-}
-```
-
-#### Proxy Sqoop Job Submission
-
-You can submit a Workflow that contains a single Sqoop command without writing a workflow.xml. Any required Jars or other
- files must already exist in HDFS.
-
-The following properties are required:
-
- * `fs.default.name`: The NameNode
- * `mapred.job.tracker`: The JobTracker
- * `user.name`: The username of the user submitting the job
- * `oozie.sqoop.command`: The sqoop command you want to run where each argument occupies one line or separated by "\n"
- * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
- * `oozie.proxysubmission`: Must be set to `true`
-
-The following properties are optional:
-
- * `oozie.sqoop.options.size`: The number of options you'll be passing to Sqoop Hadoop job
- * `oozie.sqoop.options.n`: An argument to pass to Sqoop hadoop job conf, the 'n' should be an integer starting with 0 to indicate the option number
-
-**Request:**
-
-
-```
-POST /oozie/v1/jobs?jobtype=sqoop
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://localhost:8020</value>
- </property>
- <property>
- <name>mapred.job.tracker</name>
- <value>localhost:8021</value>
- </property>
- <property>
- <name>user.name</name>
- <value>bzhang</value>
- </property>
- <property>
- <name>oozie.sqoop.command</name>
- <value>
- import
- --connect
- jdbc:mysql://localhost:3306/oozie
- --username
- oozie
- --password
- oozie
- --table
- WF_JOBS
- --target-dir
- /user/${wf:user()}/${examplesRoot}/output-data/sqoop
- </value>
- </property>
- <name>oozie.libpath</name>
- <value>hdfs://localhost:8020/user/bzhang/share/lib/sqoop</value>
- </property>
- <property>
- <name>oozie.proxysubmission</name>
- <value>true</value>
- </property>
-</configuration>
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 201 CREATED
-Content-Type: application/json;charset=UTF-8
-.
-{
- id: "job-3"
-}
-```
-
-#### Managing a Job
-
-A HTTP PUT request starts, suspends, resumes, kills, update or dryruns a job.
-
-**Request:**
-
-
-```
-PUT /oozie/v1/job/job-3?action=start
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-```
-
-Valid values for the 'action' parameter are 'start', 'suspend', 'resume', 'kill', 'dryrun', 'rerun', and 'change'.
-
-Rerunning and changing a job require additional parameters, and are described below:
-
-##### Re-Running a Workflow Job
-
-A workflow job in `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun specifying a list
-of workflow nodes to skip during the rerun. All the nodes in the skip list must have complete its
-execution.
-
-The rerun job will have the same job ID.
-
-A rerun request is done with a HTTP PUT request with a `rerun` action.
-
-**Request:**
-
-
-```
-PUT /oozie/v1/job/job-3?action=rerun
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
- <property>
- <name>user.name</name>
- <value>tucu</value>
- </property>
- <property>
- <name>oozie.wf.application.path</name>
- <value>hdfs://foo:8020/user/tucu/myapp/</value>
- </property>
- <property>
- <name>oozie.wf.rerun.skip.nodes</name>
- <value>firstAction,secondAction</value>
- </property>
- ...
-</configuration>
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-```
-
-##### Re-Running a coordinator job
-
-A coordinator job in `RUNNING` `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun by specifying the coordinator actions
-to re-execute.
-
-A rerun request is done with an HTTP PUT request with a `coord-rerun` `action`.
-
-The `type` of the rerun can be `date` or `action`.
-
-The `scope` of the rerun depends on the type:
-* `date`: a comma-separated list of date ranges. Each date range element is specified with dates separated by `::`
-* `action`: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by `-`
-
-The `refresh` parameter can be `true` or `false` to specify if the user wants to refresh an action's input and output events.
-
-The `nocleanup` parameter can be `true` or `false` to specify is the user wants to cleanup output events for the rerun actions.
-
-**Request:**
-
-
-```
-PUT /oozie/v1/job/job-3?action=coord-rerun&type=action&scope=1-2&refresh=false&nocleanup=false
-.
-```
-
-or
-
-
-```
-PUT /oozie/v1/job/job-3?action=coord-rerun&type=date2009-02-01T00:10Z::2009-03-01T00:10Z&scope=&refresh=false&nocleanup=false
-.
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-```
-
-##### Re-Running a bundle job
-
-A coordinator job in `RUNNING` `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun by specifying the coordinators to
-re-execute.
-
-A rerun request is done with an HTTP PUT request with a `bundle-rerun` `action`.
-
-A comma separated list of coordinator job names (not IDs) can be specified in the `coord-scope` parameter.
-
-The `date-scope` parameter is a comma-separated list of date ranges. Each date range element is specified with dates separated
-by `::`. If empty or not included, Oozie will figure this out for you
-
-The `refresh` parameter can be `true` or `false` to specify if the user wants to refresh the coordinator's input and output events.
-
-The `nocleanup` parameter can be `true` or `false` to specify is the user wants to cleanup output events for the rerun coordinators.
-
-**Request:**
-
-
-```
-PUT /oozie/v1/job/job-3?action=bundle-rerun&coord-scope=coord-1&refresh=false&nocleanup=false
-.
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-```
-
-
-##### Changing endtime/concurrency/pausetime of a Coordinator Job
-
-A coordinator job not in `KILLED` status can have it's endtime, concurrency, or pausetime changed.
-
-A change request is done with an HTTP PUT request with a `change` `action`.
-
-The `value` parameter can contain any of the following:
-* endtime: the end time of the coordinator job.
-* concurrency: the concurrency of the coordinator job.
-* pausetime: the pause time of the coordinator job.
-
-Multiple arguments can be passed to the `value` parameter by separating them with a ';' character.
-
-If an already-succeeded job changes its end time, its status will become running.
-
-**Request:**
-
-
-```
-PUT /oozie/v1/job/job-3?action=change&value=endtime=2011-12-01T05:00Z
-.
-```
-
-or
-
-
-```
-PUT /oozie/v1/job/job-3?action=change&value=concurrency=100
-.
-```
-
-or
-
-
-```
-PUT /oozie/v1/job/job-3?action=change&value=pausetime=2011-12-01T05:00Z
-.
-```
-
-or
-
-
-```
-PUT /oozie/v1/job/job-3?action=change&value=endtime=2011-12-01T05:00Z;concurrency=100;pausetime=2011-12-01T05:00Z
-.
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-```
-
-##### Updating coordinator definition and properties
-Existing coordinator definition and properties will be replaced by new definition and properties. Refer [Updating coordinator definition and properties](DG_CommandLineTool.html#Updating_coordinator_definition_and_properties)
-
-
-```
-PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-{"update":
- {"diff":"**********Job definition changes**********\n******************************************\n**********Job conf changes****************\n@@ -8,16 +8,12 @@\n
- <value>hdfs:\/\/localhost:9000\/user\/purushah\/examples\/apps\/aggregator\/coordinator.xml<\/value>\r\n <\/property>\r\n <property>\r\n
- - <name>user.name<\/name>\r\n
- - <value>purushah<\/value>\r\n
- - <\/property>\r\n
- - <property>\r\n <name>start<\/name>\r\n
- <value>2010-01-01T01:00Z<\/value>\r\n <\/property>\r\n <property>\r\n
- - <name>newproperty<\/name>\r\n
- - <value>new<\/value>\r\n
- + <name>user.name<\/name>\r\n
- + <value>purushah<\/value>\r\n <\/property>\r\n <property>\r\n
- <name>queueName<\/name>\r\n******************************************\n"
- }
-}
-```
-
-
-#### Job Information
-
-A HTTP GET request retrieves the job information.
-
-**Request:**
-
-
-```
-GET /oozie/v1/job/job-3?show=info&timezone=GMT
-```
-
-**Response for a workflow job:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- id: "0-200905191240-oozie-W",
- appName: "indexer-workflow",
- appPath: "hdfs://user/bansalm/indexer.wf",
- externalId: "0-200905191230-oozie-pepe",
- user: "bansalm",
- status: "RUNNING",
- conf: "<configuration> ... </configuration>",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: null,
- run: 0,
- actions: [
- {
- id: "0-200905191240-oozie-W@indexer",
- name: "indexer",
- type: "map-reduce",
- conf: "<configuration> ...</configuration>",
- startTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- status: "OK",
- externalId: "job-123-200903101010",
- externalStatus: "SUCCEEDED",
- trackerUri: "foo:8021",
- consoleUrl: "http://foo:50040/jobdetailshistory.jsp?jobId=...",
- transition: "reporter",
- data: null,
- errorCode: null,
- errorMessage: null,
- retries: 0
- },
- ...
- ]
-}
-```
-
-**Response for a coordinator job:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- id: "0-200905191240-oozie-C",
- appName: "indexer-Coord",
- appPath: "hdfs://user/bansalm/myapp/logprocessor-coord.xml",
- externalId: "0-200905191230-oozie-pepe",
- user: "bansalm",
- status: "RUNNING",
- conf: "<configuration> ... </configuration>",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
- frequency: "${days(1)}"
- actions: [
- {
- id: "0000010-130426111815091-oozie-bansalm-C@1",
- createdTime: "Fri, 26 Apr 2013 20:57:07 GMT",
- externalId: "",
- missingDependencies: "",
- runConf: null,
- createdConf: null,
- consoleUrl: null,
- nominalTime: "Fri, 01 Jan 2010 01:00:00 GMT",
- ...
-}
-```
-
-**Response for a bundle job:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- jobType: "bundle",
- id: "0-200905191240-oozie-B",
- appName: "new-bundle",
- appPath: "hdfs://user/bansalm/myapp/logprocessor-bundle.xml",
- externalId: "0-200905191230-oozie-pepe",
- user: "bansalm",
- status: "RUNNING",
- conf: "<configuration> ... </configuration>",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 31 Dec 2009 00:00:00 GMT"
- bundleCoordJobs: [
- {
- status: "RUNNING",
- concurrency: 1,
- conf: "<configuration> ... </configuration>",
- executionPolicy: "FIFO",
- toString: "Coordinator application id[0000010-130426111815091-oozie-bansalm-C] status[RUNNING]",
- coordJobName: "coord-1",
- endTime: "Fri, 01 Jan 2010 03:00:00 GMT",
- ...
- }
- ...
-}
-```
-
-**Getting all the Workflows corresponding to a Coordinator Action:**
-
-A coordinator action kicks off different workflows for its original run and all subsequent reruns.
-Getting a list of those workflow ids is a useful tool to keep track of your actions' runs and
-to go debug the workflow job logs if required. Along with ids, it also lists their statuses,
-and start and end times for quick reference.
-
-Both v1 and v2 API are supported. v0 is not supported.
-
-
-```
-GET /oozie/v2/job/0000001-111219170928042-oozie-joe-C@1?show=allruns
-```
-
-**Response**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{"workflows":[
- {
- "startTime":"Mon, 24 Mar 2014 23:40:53 GMT",
- "id":"0000001-140324163709596-oozie-chit-W",
- "status":"SUCCEEDED",
- "endTime":"Mon, 24 Mar 2014 23:40:54 GMT"
- },
- {
- "startTime":"Mon, 24 Mar 2014 23:44:01 GMT",
- "id":"0000000-140324164318985-oozie-chit-W",
- "status":"SUCCEEDED",
- "endTime":"Mon, 24 Mar 2014 23:44:01 GMT"
- },
- {
- "startTime":"Mon, 24 Mar 2014 23:44:24 GMT",
- "id":"0000001-140324164318985-oozie-chit-W",
- "status":"SUCCEEDED",
- "endTime":"Mon, 24 Mar 2014 23:44:24 GMT"
- }
-]}
-```
-
-An alternate API is also available for the same output. With this API, one can pass the coordinator **JOB** Id
-followed by query params - `type=action` and `scope=<action-number>`. One single action number can be passed at a time.
-
-
-```
-GET /oozie/v2/job/0000001-111219170928042-oozie-joe-C?show=allruns&type=action&scope=1
-```
-
-**Retrieve a subset of actions**
-
-Query parameters, `offset` and `length` can be specified with a workflow job to retrieve specific actions. Default is offset=0, len=1000
-
-```
-GET /oozie/v1/job/0000002-130507145349661-oozie-joe-W?show=info&offset=5&len=10
-```
-Query parameters, `offset`, `length`, `filter` can be specified with a coordinator job to retrieve specific actions.
-Query parameter, `order` with value "desc" can be used to retrieve the latest coordinator actions materialized instead of actions from @1.
-Query parameters `filter` can be used to retrieve coordinator actions matching specific status.
-Default is offset=0, len=0 for v2/job (i.e., does not return any coordinator actions) and offset=0, len=1000 with v1/job and v0/job.
-So if you need actions to be returned with v2 API, specifying `len` parameter is necessary.
-Default `order` is "asc".
-
-```
-GET /oozie/v1/job/0000001-111219170928042-oozie-joe-C?show=info&offset=5&len=10&filter=status%3DKILLED&order=desc
-```
-Note that the filter is URL encoded, its decoded value is `status=KILLED`.
-
-```
-GET /oozie/v1/job/0000001-111219170928042-oozie-joe-C?show=info&filter=status%21%3DSUCCEEDED&order=desc
-```
-This retrieves coordinator actions except for SUCCEEDED status, which is useful for debugging.
-
-**Retrieve information of the retry attempts of the workflow action:**
-
-
-```
-GET oozie/v2/job/0000000-161212175234862-oozie-puru-W@pig-node?show=retries
-```
-
-**Response**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- "retries":
- [
- {
- "startTime": "Tue, 13 Dec 2016 01:54:13 GMT",
- "consoleUrl": "http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2648",
- "endTime": "Tue, 13 Dec 2016 01:54:20 GMT",
- "attempt": "1"
- },
- {
- "startTime": "Tue, 13 Dec 2016 01:55:20 GMT",
- "consoleUrl": "http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2649",
- "endTime": "Tue, 13 Dec 2016 01:55:24 GMT",
- "attempt": "2"
- }
- ]
-}
-```
-
-#### Job Application Definition
-
-A HTTP GET request retrieves the workflow or a coordinator job definition file.
-
-**Request:**
-
-
-```
-GET /oozie/v1/job/job-3?show=definition
-```
-
-**Response for a workflow job:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<workflow-app name='xyz-app' xmlns="uri:oozie:workflow:0.1">
- <start to='firstaction' />
- ...
- <end name='end' />
-</workflow-app>
-```
-
-**Response for a coordinator job:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<coordinator-app name='abc-app' xmlns="uri:oozie:coordinator:0.1" frequency="${days(1)}
- start="2009-01-01T00:00Z" end="2009-12-31T00:00Z" timezone="America/Los_Angeles">
- <datasets>
- ...
- </datasets>
- ...
-</coordinator-app>
-```
-
-**Response for a bundle job:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8"?>
-<bundle-app name='abc-app' xmlns="uri:oozie:coordinator:0.1"
- start="2009-01-01T00:00Z" end="2009-12-31T00:00Z"">
- <datasets>
- ...
- </datasets>
- ...
-</bundle-app>
-```
-
-#### Job Log
-
-An HTTP GET request retrieves the job log.
-
-**Request:**
-
-
-```
-GET /oozie/v1/job/job-3?show=log
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: text/plain;charset=UTF-8
-.
-...
-23:21:31,272 TRACE oozieapp:526 - USER[bansalm] GROUP[other] TOKEN[-] APP[test-wf] JOB[0-20090518232130-oozie-tucu] ACTION[mr-1] Start
-23:21:31,305 TRACE oozieapp:526 - USER[bansalm] GROUP[other] TOKEN[-] APP[test-wf] JOB[0-20090518232130-oozie-tucu] ACTION[mr-1] End
-...
-```
-
-#### Job Error Log
-
-An HTTP GET request retrieves the job error log.
-
-**Request:**
-
-
-```
-GET /oozie/v2/job/0000000-150121110331712-oozie-puru-B?show=errorlog
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: text/plain;charset=UTF-8
-2015-01-21 11:33:29,090 WARN CoordSubmitXCommand:523 - SERVER[-] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150121110331712-oozie-puru-B] ACTION[] SAXException :
-org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-type.2.4.a: Invalid content was found starting with element 'concurrency'. One of '{"uri:oozie:coordinator:0.2":controls, "uri:oozie:coordinator:0.2":datasets, "uri:oozie:coordinator:0.2":input-events, "uri:oozie:coordinator:0.2":output-events, "uri:oozie:coordinator:0.2":action}' is expected.
- at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
- at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
-...
-```
-
-
-#### Job Audit Log
-
-An HTTP GET request retrieves the job audit log.
-
-**Request:**
-
-
-```
-GET /oozie/v2/job/0000000-150322000230582-oozie-puru-C?show=auditlog
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: text/plain;charset=UTF-8
-2015-03-22 00:04:35,494 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 00:05:13,823 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 00:06:59,561 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 23:22:20,012 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-```
-
-
-#### Filtering the server logs with logfilter options
-User can provide multiple option to filter logs using -logfilter opt1=val1;opt2=val1;opt3=val1. This can be used to fetch only just logs of interest faster as fetching Oozie server logs is slow due to the overhead of pattern matching.
-
-
-```
-GET /oozie/v1/job/0000003-140319184715726-oozie-puru-C?show=log&logfilter=limit=3;loglevel=WARN
-```
-
-
-Refer to the [Filtering the server logs with logfilter options](DG_CommandLineTool.html#Filtering_the_server_logs_with_logfilter_options) for more details.
-
-
-#### Job graph
-
-An `HTTP GET` request returns the image of the workflow DAG (rendered as a PNG or SVG image, or as a DOT string).
-
- * The nodes that are being executed are painted yellow
- * The nodes that have successfully executed are painted green
- * The nodes that have failed execution are painted red
- * The nodes that are yet to be executed are pained gray
- * An arc painted green marks the successful path taken so far
- * An arc painted red marks the failure of the node and highlights the _error_ action
- * An arc painted gray marks a path not taken yet
-
-**PNG request:**
-
-```
-GET /oozie/v1/job/job-3?show=graph[&show-kill=true][&format=png]
-```
-
-**PNG response:**
-
-```
-HTTP/1.1 200 OK
-Content-Type: image/png
-Content-Length: {image_size_in_bytes}
-
-{image_bits}
-```
-
-**SVG request:**
-
-```
-GET /oozie/v1/job/job-3?show=graph[&show-kill=true]&format=svg
-```
-
-**SVG response:**
-
-```
-HTTP/1.1 200 OK
-Content-Type: image/svg+xml
-Content-Length: {image_size_in_bytes}
-
-{image_bits}
-```
-
-**DOT request:**
-
-```
-GET /oozie/v1/job/job-3?show=graph[&show-kill=true]&format=dot
-```
-
-**DOT response:**
-
-```
-HTTP/1.1 200 OK
-Content-Type: text/plain
-Content-Length: {dot_size_in_bytes}
-
-{dot_bytes}
-```
-
-The optional `show-kill` parameter shows `kill` node in the graph. Valid values for this parameter are `1`, `yes`, and `true`.
-This parameter has no effect when workflow fails and the failure node leads to the `kill` node; in that case `kill` node is shown
-always.
-
-The optional `format` parameter describes whether the response has to be rendered as a PNG image, or an SVG image, or a DOT string.
-When omitted, `format` is considered as `png` for backwards compatibility. Oozie Web UI uses the `svg` `format`.
-
-The node labels are the node names provided in the workflow XML.
-
-This API returns `HTTP 400` when run on a resource other than a workflow, viz. bundle and coordinator.
-
-#### Job Status
-
-An `HTTP GET` request that returns the current status (e.g. `SUCCEEDED`, `KILLED`, etc) of a given job. If you are only interested
-in the status, and don't want the rest of the information that the `info` query provides, it is recommended to use this call
-as it is more efficient.
-
-**Request**
-
-```
-GET /oozie/v2/job/0000000-140908152307821-oozie-rkan-C?show=status
-```
-
-**Response**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- "status" : "SUCCEEDED"
-}
-```
-
-It accepts any valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.
-
-#### Changing job SLA definition and alerting
-An `HTTP PUT` request to change job SLA alert status/SLA definition.
-
- * All sla commands takes actions-list or date parameter.
- * `date`: a comma-separated list of date ranges. Each date range element is specified with dates separated by `::`
- * `action-list`: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by `-`
- * For bundle jobs additional `coordinators` (coord_name/id) parameter can be passed.
- * Sla change command need extra parameter `value` to specify new sla definition.
- * Changing SLA definition
-
- SLA definition of should-start, should-end, nominal-time and max-duration can be changed.
-
-
-```
-PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-change&value=<key>=<value>;...;<key>=<value>
-```
-
- * Disabling SLA alert
-
-
-```
-PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-disable&action-list=3-4
-```
-Will disable SLA alert for actions 3 and 4.
-
-
-```
-PUT /oozie/v1/job/0000003-140319184715726-oozie-puru-C?action=sla-disable&date=2009-02-01T00:10Z::2009-03-01T00:10Z
-```
-Will disable SLA alert for actions whose nominal time is in-between 2009-02-01T00:10Z 2009-03-01T00:10Z (inclusive).
-
-
-
-```
-PUT /oozie/v1/job/0000004-140319184715726-oozie-puru-B?action=sla-disable&date=2009-02-01T00:10Z::2009-03-01T00:10Z&coordinators=abc
-```
-For bundle jobs additional coordinators (list of comma separated coord_name/id) parameter can be passed.
-
- * Enabling SLA alert
-
-
-```
-PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-enable&action-list=1,14,17-20
-```
-Will enable SLA alert for actions 1,14,17,18,19,20.
-
-### Getting missing dependencies of coordinator action(s)
-
-
-```
-GET oozie/v2/job/0000000-170104115137443-oozie-puru-C?show=missing-dependencies&action-list=1,20
-```
-
-**Response**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-
-{
-"missingDependencies":
-[{
- "blockedOn": "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS",
- "dataSets":
- [
- {
- "missingDependencies":
- [
- "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS"
- ],
- "dataSet": "input-2"
- }
- ],
- "id": 1
- },
- {
- "blockedOn": "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/20/00/_SUCCESS",
- "dataSets":
- [
- {
- "missingDependencies":
- [
- "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/20/00/_SUCCESS",
- "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/19/40/_SUCCESS",
- "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/19/20/_SUCCESS",
- "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/19/00/_SUCCESS",
- "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/18/40/_SUCCESS",
- "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/18/20/_SUCCESS"
- ],
- "dataSet": "input-2"
- }
- ],
- "id": 20
- }]
-}
-```
-#### Jobs Information
-
-A HTTP GET request retrieves workflow and coordinator jobs information.
-
-**Request:**
-
-
-```
-GET /oozie/v1/jobs?filter=user%3Dbansalm&offset=1&len=50&timezone=GMT
-```
-
-Note that the filter is URL encoded, its decoded value is `user=bansalm`.
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- offset: 1,
- len: 50,
- total: 1002,
-**jobs: [
- {
-** jobType: "workflow"
- id: "0-200905191240-oozie-W",
- appName: "indexer-workflow",
- appPath: "hdfs://user/tucu/indexer-wf",
- user: "bansalm",
- group: "other",
- status: "RUNNING",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: null,
- info: "run=0",
- },
- {
-** jobType: "coordinator"
- id: "0-200905191240-oozie-C",
- appName: "logprocessor-coord",
- appPath: "hdfs://user/bansalm/myapp/logprocessor-coord.xml",
- user: "bansalm",
- group: "other",
- status: "RUNNING",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
- info: "nextAction=5",
- },
- {
-** jobType: "bundle"
- id: "0-200905191240-oozie-B",
- appName: "logprocessor-bundle",
- appPath: "hdfs://user/bansalm/myapp/logprocessor-bundle.xml",
- user: "bansalm",
- group: "other",
- status: "RUNNING",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
- },
- ...
- ]
-}
-```
-
-No action information is returned when querying for multiple jobs.
-
-
-The syntax for the filter is `[NAME=VALUE][;NAME=VALUE]*`
-
-Valid filter names are:
-
- * text: any text that might be a part of application name or a part of user name or a complete job ID
- * name: the application name from the workflow/coordinator/bundle definition
- * user: the user who submitted the job
- * group: the group for the job (support for the group filter is discontinued. version: 3.2.0 OOZIE-228).
- * id: the id of the workflow/coordinator/bundle job
- * status: the status of the job
- * startCreatedTime : the start of the window about workflow job's created time
- * endCreatedTime : the end of above window
- * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
-
-The query will do an AND among all the filter names.
-
-The query will do an OR among all the filter values for the same name. Multiple values must be specified as different
-name value pairs.
-
-Additionally the `offset` and `len` parameters can be used for pagination. The start parameter is base 1.
-
-Moreover, the `jobtype` parameter could be used to determine what type of job is looking for.
-The valid values of job type are: `wf`, `coordinator` or `bundle`.
-
-startCreatedTime and endCreatedTime should be specified either in **ISO8601 (UTC)** format **(yyyy-MM-dd'T'HH:mm'Z')** or
-a offset value in days or hours or minutes from the current time. For example, -2d means the (current time - 2 days),
--3h means the (current time - 3 hours), -5m means the (current time - 5 minutes).
-
-#### Bulk modify jobs
-
-A HTTP PUT request can kill, suspend, or resume all jobs that satisfy the url encoded parameters.
-
-**Request:**
-
-
-```
-PUT /oozie/v1/jobs?action=kill&filter=name%3Dcron-coord&offset=1&len=50&jobtype=coordinator
-```
-
-This request will kill all the coordinators with name=cron-coord up to 50 of them.
-
-Note that the filter is URL encoded, its decoded value is `name=cron-coord`.
-
-The syntax for the filter is `[NAME=VALUE][;NAME=VALUE]*`
-
-Valid filter names are:
-
- * name: the application name from the workflow/coordinator/bundle definition
- * user: the user that submitted the job
- * group: the group for the job
- * status: the status of the job
-
-The query will do an AND among all the filter names.
-
-The query will do an OR among all the filter values for the same name. Multiple values must be specified as different
-name value pairs.
-
-Additionally the `offset` and `len` parameters can be used for pagination. The start parameter is base 1.
-
-Moreover, the `jobtype` parameter could be used to determine what type of job is looking for.
-The valid values of job type are: `wf`, `coordinator` or `bundle`
-
-**Response:**
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- offset: 1,
- len: 50,
- total: 2,
-**jobs: [
- {
-** jobType: "coordinator"
- id: "0-200905191240-oozie-C",
- appName: "cron-coord",
- appPath: "hdfs://user/bansalm/app/cron-coord.xml",
- user: "bansalm",
- group: "other",
- status: "KILLED",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
- info: "nextAction=5",
- },
- {
-** jobType: "coordinator"
- id: "0-200905191240-oozie-C",
- appName: "cron-coord",
- appPath: "hdfs://user/bansalm/myapp/cron-coord.xml",
- user: "bansalm",
- group: "other",
- status: "KILLED",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
- },
- ...
- ]
-}
-```
-
-
-```
-PUT /oozie/v1/jobs?action=suspend&filter=status%3Drunning&offset=1&len=50&jobtype=wf
-```
-
-This request will suspend all the workflows with status=running up to 50 of them.
-Note that the filter is URL encoded, its decoded value is `status=running`.
-
-**Response:**
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- offset: 1,
- len: 50,
- total: 50,
-**jobs: [
- {
-** jobType: "workflow"
- id: "0-200905191240-oozie-W",
- appName: "indexer-workflow",
- appPath: "hdfs://user/tucu/indexer-wf",
- user: "bansalm",
- group: "other",
- status: "SUSPENDED",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: null,
- info: "run=0",
- },
- {
-** jobType: "workflow"
- id: "0-200905191240-oozie-W",
- appName: "logprocessor-wf",
- appPath: "hdfs://user/bansalm/myapp/workflow.xml",
- user: "bansalm",
- group: "other",
- status: "SUSPENDED",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: null,
- info: "run=0",
- },
- ...
- ]
-}
-```
-
-#### Jobs information using Bulk API
-
-A HTTP GET request retrieves a bulk response for all actions, corresponding to a particular bundle, that satisfy user specified criteria.
-This is useful for monitoring purposes, where user can find out about the status of downstream jobs with a single bulk request.
-The criteria are used for filtering the actions returned. Valid options (_case insensitive_) for these request criteria are:
-
- * **bundle**: the application name from the bundle definition
- * **coordinators**: the application name(s) from the coordinator definition.
- * **actionStatus**: the status of coordinator action (Valid values are WAITING, READY, SUBMITTED, RUNNING, SUSPENDED, TIMEDOUT, SUCCEEDED, KILLED, FAILED)
- * **startCreatedTime**: the start of the window you want to look at, of the actions' created time
- * **endCreatedTime**: the end of above window
- * **startScheduledTime**: the start of the window you want to look at, of the actions' scheduled i.e. nominal time.
- * **endScheduledTime**: the end of above window
-
-Specifying 'bundle' is REQUIRED. All the rest are OPTIONAL but that might result in thousands of results depending on the size of your job. (pagination comes into play then)
-
-If no 'actionStatus' values provided, by default KILLED,FAILED will be used.
-For e.g if the query string is only "bundle=MyBundle", the response will have all actions (across all coordinators) whose status is KILLED or FAILED
-
-The query will do an AND among all the filter names, and OR among each filter name's values.
-
-
-The syntax for the request criteria is `[NAME=VALUE][;NAME=VALUE]*`
-
-For 'coordinators' and 'actionStatus', if user wants to check for multiple values, they can be passed in a comma-separated manner.
-**Note**: The query will do an OR among them. Hence no need to repeat the criteria name
-
-All the time values should be specified in **ISO8601 (UTC)** format i.e. **yyyy-MM-dd'T'HH:mm'Z'**
-
-Additionally the `offset` and `len` parameters can be used as usual for pagination. The start parameter is base 1.
-
-If you specify a coordinator in the list, that does not exist, no error is thrown; simply the response will be empty or pertaining to the other valid coordinators.
-However, if bundle name provided does not exist, an error is thrown.
-
-**Request:**
-
-
-```
-GET /oozie/v1/jobs?bulk=bundle%3Dmy-bundle-app;coordinators%3Dmy-coord-1,my-coord-5;actionStatus%3DKILLED&offset=1&len=50
-```
-
-Note that the filter is URL encoded, its decoded value is `user=chitnis`. If typing in browser URL, one can type decoded value itself i.e. using '='
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- offset: 1,
- len: 50,
- total: 1002,
-** bulkresponses: [
-** {
- bulkbundle:
- {
- bundleJobName: "my-bundle-app",
- bundleJobId: "0-200905191240-oozie-B",
- status: "SUSPENDED",
- },
- bulkcoord:
- {
- coordJobName: "my-coord-1",
- status: "SUSPENDED",
- },
- bulkaction:
- {
- id: "0-200905191240-oozie-C@21",
- coordJobId: "0-200905191240-oozie-C",
- actionNumber: 21,
- externalId: "job_00076_0009",
- status: "KILLED",
- externalStatus: "FAILED",
- errorCode: "E0902",
- errorMessage: "Input file corrupt",
- createdTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- nominalTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- missingDependencies: "hdfs://nn:port/user/joe/file.txt"
- },
- },
-** {
- bulkbundle:
- {
- bundleJobName: "my-bundle-app",
- bundleJobId: "0-200905191240-oozie-B",
- status: "SUSPENDED",
- }
- bulkcoord:
- {
- coordJobName: "my-coord-5",
- status: "SUSPENDED",
- }
- bulkaction:
- {
- id: "0-200905191245-oozie-C@114",
- coordJobId: "0-200905191245-oozie-C",
- actionNumber: 114,
- externalId: "job_00076_0265",
- status: "KILLED",
- externalStatus: "KILLED",
- errorCode: "E0603",
- errorMessage: "SQL error in operation ...",
- createdTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- nominalTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- missingDependencies:
- }
- }
- ...
- ]
-}
-```
-
-## Oozie Web Services API, V2 (Workflow , Coordinator And Bundle)
-
-The Oozie Web Services API is a HTTP REST JSON API.
-
-All responses are in `UTF-8`.
-
-Assuming Oozie is running at `OOZIE_URL`, the following web services end points are supported:
-
- * \<OOZIE_URL\>/versions
- * \<OOZIE_URL\>/v2/admin
- * \<OOZIE_URL\>/v2/job
- * \<OOZIE_URL\>/v2/jobs
-
-**Changes in v2 job API:**
-
-There is a difference in the JSON format of Job Information API (*/job) particularly for map-reduce action.
-No change for other actions.
-In v1, externalId and consoleUrl point to spawned child job ID, and externalChildIDs is null in map-reduce action.
-In v2, externalId and consoleUrl point to launcher job ID, and externalChildIDs is spawned child job ID in map-reduce action.
-
-v2 supports retrieving of JMS topic on which job notifications are sent
-
-**REST API URL:**
-
-
-```
-GET http://localhost:11000/oozie/v2/job/0000002-130507145349661-oozie-vira-W?show=jmstopic
-```
-
-**Changes in v2 admin API:**
-
-v2 adds support for retrieving JMS connection information related to JMS notifications.
-
-**REST API URL:**
-
-
-```
-GET http://localhost:11000/oozie/v2/admin/jmsinfo
-```
-
-v2/jobs remain the same as v1/jobs
-
-### Job and Jobs End-Points
-
-#### Job Information
-
-A HTTP GET request retrieves the job information.
-
-**Request:**
-
-
-```
-GET /oozie/v2/job/job-3?show=info&timezone=GMT
-```
-
-**Response for a workflow job:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
-**jobType: "workflow",
- id: "0-200905191240-oozie-W",
- appName: "indexer-workflow",
- appPath: "hdfs://user/bansalm/indexer.wf",
- externalId: "0-200905191230-oozie-pepe",
- user: "bansalm",
- group: "other",
- status: "RUNNING",
- conf: "<configuration> ... </configuration>",
- createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- endTime: null,
- run: 0,
- actions: [
- {
- id: "0-200905191240-oozie-W@indexer",
- name: "indexer",
- type: "map-reduce",
- conf: "<configuration> ...</configuration>",
- startTime: "Thu, 01 Jan 2009 00:00:00 GMT",
- endTime: "Fri, 02 Jan 2009 00:00:00 GMT",
- status: "OK",
- externalId: "job-123-200903101010",
- externalStatus: "SUCCEEDED",
- trackerUri: "foo:8021",
- consoleUrl: "http://foo:50040/jobdetailshistory.jsp?jobId=job-123-200903101010",
- transition: "reporter",
- data: null,
- stats: null,
- externalChildIDs: "job-123-200903101011"
- errorCode: null,
- errorMessage: null,
- retries: 0
- },
- ...
- ]
-}
-```
-
-#### Managing a Job
-##### Ignore a Coordinator Job or Action
-
-A ignore request is done with an HTTP PUT request with a `ignore`
-
-The `type` parameter supports `action` only.
-The `scope` parameter can contain coordinator action id(s) to be ignored.
-Multiple action ids can be passed to the `scope` parameter
-
-**Request:**
-
-Ignore a coordinator job
-
-```
-PUT /oozie/v2/job/job-3?action=ignore
-```
-
-Ignore coordinator actions
-
-```
-PUT /oozie/v2/job/job-3?action=ignore&type=action&scope=3-4
-```
-
-### Validate End-Point
-
-This endpoint is to validate a workflow, coordinator, bundle XML file.
-
-#### Validate a local file
-
-**Request:**
-
-
-```
-POST /oozie/v2/validate?file=/home/test/myApp/workflow.xml
-Content-Type: application/xml;charset=UTF-8
-.
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<workflow-app xmlns="uri:oozie:workflow:0.3" name="test">
- <start to="shell"/>
- <action name="shell">
- <shell xmlns="uri:oozie:shell-action:0.3">
- <job-tracker>${jobTracker}</job-tracker>
- <name-node>${nameNode}</name-node>
- <exec>script.sh</exec>
- <argument></argument>
- <file>script.sh</file>
- <capture-output/>
- </shell>
- <ok to="end"/>
- <error to="fail"/>
- </action>
- <kill name="fail">
- <message>failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
- </kill>
- <end name="end"/>
-</workflow-app>
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- validate: "Valid workflow-app"
-}
-```
-
-#### Validate a file in HDFS
-
-You can validate a workflow, coordinator, bundle XML file in HDFS. The XML file must already exist in HDFS.
-
-**Request:**
-
-
-```
-POST /oozie/v2/validate?file=hdfs://localhost:8020/user/test/myApp/workflow.xml
-Content-Type: application/xml;charset=UTF-8
-.
-```
-
-**Response:**
-
-
-```
-HTTP/1.1 200 OK
-Content-Type: application/json;charset=UTF-8
-.
-{
- validate: "Valid workflow-app"
-}
-```
-
-
-
-
-
-
-
-
[08/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_ActionAuthentication.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_ActionAuthentication.twiki b/docs/src/site/twiki/DG_ActionAuthentication.twiki
deleted file mode 100644
index d280baf..0000000
--- a/docs/src/site/twiki/DG_ActionAuthentication.twiki
+++ /dev/null
@@ -1,132 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Action Authentication
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Background
-
-A secure cluster requires that actions have been authenticated (typically via Kerberos). However, due to the way that Oozie runs
-actions, Kerberos credentials are not easily made available to actions launched by Oozie. For many action types, this is not a
-problem because they are self contained (beyond core Hadoop components). For example, a Pig action typically only talks to
-MapReduce and HDFS. However, some actions require talking to external services (e.g. HCatalog, HBase Region Server, Hive Server 2)
-and in these cases, the actions require some extra configuration in Oozie to authenticate. To be clear, this extra configuration
-is only required if an action will be talking to these types of external services; running a typical MapReduce, Pig, Hive, etc
-action will not require any of this.
-
-For these situations, Oozie will have to use its Kerberos credentials to obtain "delegation tokens" (think of it like a cookie) on
-behalf of the user from the service in question. The details of what this means is beyond the scope of this documentation, but
-basically, Oozie needs some extra configuration in the workflow so that it can obtain this delegation token.
-
-## Oozie Server Configuration
-
-The code to obtain delegation tokens is pluggable so that it is easy to add support for different services by simply subclassing
-org.apache.oozie.action.hadoop.Credentials to retrieve a delegation token from the service and add it to the Configuration.
-
-Out of the box, Oozie already comes with support for some credential types
-(see [Built-in Credentials Implementations](DG_ActionAuthentication.html#Built-in_Credentials_Implementations)).
-The credential classes that Oozie should load are specified by the following property in oozie-site.xml. The left hand side of the
-equals sign is the type for the credential type, while the right hand side is the class.
-
-
-```
- <property>
- <name>oozie.credentials.credentialclasses</name>
- <value>
- hcat=org.apache.oozie.action.hadoop.HCatCredentials,
- hbase=org.apache.oozie.action.hadoop.HbaseCredentials,
- hive2=org.apache.oozie.action.hadoop.Hive2Credentials
- </value>
- </property>
-```
-
-## Workflow Changes
-
-The user should add a `credentials` section to the top of their workflow that contains 1 or more `credential` sections. Each of
-these `credential` sections contains a name for the credential, the type for the credential, and any configuration properties
-needed by that type of credential for obtaining a delegation token. The `credentials` section is available in workflow schema
-version 0.3 and later.
-
-For example, the following workflow is configured to obtain an HCatalog delegation token, which is given to a Pig action so that the
-Pig action can talk to a secure HCatalog:
-
-
-```
- <workflow-app xmlns='uri:oozie:workflow:0.4' name='pig-wf'>
- <credentials>
- <credential name='my-hcat-creds' type='hcat'>
- <property>
- <name>hcat.metastore.uri</name>
- <value>HCAT_URI</value>
- </property>
- <property>
- <name>hcat.metastore.principal</name>
- <value>HCAT_PRINCIPAL</value>
- </property>
- </credential>
- </credentials>
- ...
- <action name='pig' cred='my-hcat-creds'>
- <pig>
- <job-tracker>JT</job-tracker>
- <name-node>NN</name-node>
- <configuration>
- <property>
- <name>TESTING</name>
- <value>${start}</value>
- </property>
- </configuration>
- </pig>
- </action>
- ...
- </workflow-app>
-```
-
-The type of the `credential` is "hcat", which is the type name we gave for the HCatCredentials class in oozie-site.xml. We gave
-the `credential` a name, "my-hcat-creds", which can be whatever you want; we then specify cred='my-hcat-creds' in the Pig action,
-so that Oozie will include these credentials with the action. You can include multiple credentials with an action by specifying
-a comma-separated list of `credential` names. And finally, the HCatCredentials required two properties (the metastore URI and
-principal), which we also specified.
-
-Adding the `credentials` section to a workflow and referencing it in an action will make Oozie always try to obtain that delegation
-token. Ordinarily, this would mean that you cannot re-use this workflow in a non-secure cluster without editing it because trying
-to obtain the delegation token will likely fail. However, you can tell Oozie to ignore the `credentials` for a workflow by setting
-the job-level property `oozie.credentials.skip` to `true`; this will allow you to use the same workflow.xml in a secure and
-non-secure cluster by simply changing the job-level property at runtime. If omitted or set to `false`, Oozie will handle
-the `credentials` section normally. In addition, you can also set this property at the action-level or server-level to skip getting
-credentials for just that action or for all workflows, respectively. The order of priority is this:
-
- 1. `oozie.credentials.skip` in the `configuration` section of an action, if set
- 1. `oozie.credentials.skip` in the job.properties for a workflow, if set
- 1. `oozie.credentials.skip` in oozie-site.xml for all workflows, if set
- 1. (don't skip)
-
-## Built-in Credentials Implementations
-
-Oozie currently comes with the following Credentials implementations:
-
- 1. HCatalog and Hive Metastore: `org.apache.oozie.action.hadoop.HCatCredentials`
- 1. HBase: `org.apache.oozie.action.hadoop.HBaseCredentials`
- 1. Hive Server 2: `org.apache.oozie.action.hadoop.Hive2Credentials`
-
-HCatCredentials requires these two properties:
-
- 1. `hcat.metastore.principal` or hive.metastore.kerberos.principal
- 1. `hcat.metastore.uri` or hive.metastore.uris
-
-**Note:** The HCatalog Metastore and Hive Metastore are one and the same and so the "hcat" type credential can also be used to talk
-to a secure Hive Metastore, though the property names would still start with "hcat.".
-
-HBase does not require any additional properties since the hbase-site.xml on the Oozie server provides necessary information
-to obtain a delegation token; though properties can be overwritten here if desired.
-
-Hive2Credentials requires these two properties:
-
- 1. `hive2.server.principal`
- 1. `hive2.jdbc.url`
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
[10/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/BundleFunctionalSpec.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/BundleFunctionalSpec.twiki b/docs/src/site/twiki/BundleFunctionalSpec.twiki
deleted file mode 100644
index 5301429..0000000
--- a/docs/src/site/twiki/BundleFunctionalSpec.twiki
+++ /dev/null
@@ -1,418 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Bundle Specification
-
-The goal of this document is to define a new oozie abstraction called bundle system specialized in submitting and maintaining a set of coordinator applications.
-
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Changelog
-
-## 1. Bundle Overview
-
-Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun in the bundle level resulting a better and easy operational control.
-
-More specifically, the oozie **Bundle** system allows the user to define and execute a bunch of coordinator applications often called a data pipeline. There is no explicit dependency among the coordinator applications in a bundle. However, a user could use the data dependency of coordinator applications to create an implicit data application pipeline.
-
-
-## 2. Definitions
-
-**Kick-off-time:** The time when a bundle should start and submit coordinator applications.
-
-**Bundle Application:** A bundle application defines a set of coordinator applications and when to start those. Normally, bundle applications are parameterized. A bundle application is written in XML.
-
-**Bundle Job:** A bundle job is an executable instance of a bundle application. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
-
-**Bundle Definition Language:** The language used to describe bundle applications.
-
-## 3. Expression Language for Parameterization
-
-Bundle application definitions can be parameterized with variables.
-
-At job submission time all the parameters are resolved into concrete values.
-
-The parameterization of bundle definitions is done using JSP Expression Language syntax from the [JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to support variables as parameters but also complex expressions.
-
-EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.
-
-
-## 4. Bundle Job
-
-### 4.1. Bundle Job Status
-
-At any time, a bundle job is in one of the following status: **PREP, RUNNING, RUNNINGWITHERROR, SUSPENDED, PREPSUSPENDED, SUSPENDEDWITHERROR, PAUSED, PAUSEDWITHERROR, PREPPAUSED, SUCCEEDED, DONEWITHERROR, KILLED, FAILED**.
-
-### 4.2. Transitions of Bundle Job Status
-
-Valid bundle job status transitions are:
-
- * **PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED**
- * **RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED**
- * **RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | FAILED | KILLED**
- * **PREPSUSPENDED --> PREP | KILLED**
- * **SUSPENDED --> RUNNING | KILLED**
- * **SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED**
- * **PREPPAUSED --> PREP | KILLED**
- * **PAUSED --> SUSPENDED | RUNNING | KILLED**
- * **PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED**
-
-### 4.3. Details of Status Transitions
-When a bundle job is submitted, oozie parses the bundle job XML. Oozie then creates a record for the bundle with status **PREP** and returns a unique ID.
-
-When a user requests to suspend a bundle job that is in **PREP** state, oozie puts the job in status **PREPSUSPENDED**. Similarly, when pause time reaches for a bundle job with **PREP** status, oozie puts the job in status **PREPPAUSED**.
-
-Conversely, when a user requests to resume a **PREPSUSPENDED** bundle job, oozie puts the job in status **PREP**. And when pause time is reset for a bundle job that is in **PREPPAUSED** state, oozie puts the job in status **PREP**.
-
-There are two ways a bundle job could be started.
-
-* If `kick-off-time` (defined in the bundle xml) reaches. The default value is null which means starts coordinators NOW.
-
-* If user sends a start request to START the bundle.
-
-When a bundle job starts, oozie puts the job in status **RUNNING** and it submits all the coordinator jobs. If any coordinator job goes to **FAILED/KILLED/DONEWITHERROR** state, the bundle job is put in **RUNNINGWITHERROR**
-
-When a user requests to kill a bundle job, oozie puts the job in status **KILLED** and it sends kill to all submitted coordinator jobs.
-
-When a user requests to suspend a bundle job that is in **RUNNING** status, oozie puts the job in status **SUSPENDED** and it suspends all submitted coordinator jobs. Similarly, when a user requests to suspend a bundle job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **SUSPENDEDWITHERROR** and it suspends all submitted coordinator jobs.
-
-When pause time reaches for a bundle job that is in **RUNNING** status, oozie puts the job in status **PAUSED**. When pause time reaches for a bundle job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **PAUSEDWITHERROR**.
-
-Conversely, when a user requests to resume a **SUSPENDED** bundle job, oozie puts the job in status **RUNNING**. Similarly, when a user requests to resume a **SUSPENDEDWITHERROR** bundle job, oozie puts the job in status **RUNNINGWITHERROR**. And when pause time is reset for a bundle job and job status is **PAUSED**, oozie puts the job in status **RUNNING**. Similarly, when the pause time is reset for a bundle job and job status is **PAUSEDWITHERROR**, oozie puts the job in status **RUNNINGWITHERROR**
-
-When all the coordinator jobs finish, oozie updates the bundle status accordingly. If all coordinators reaches to the _same_ terminal state, bundle job status also move to the same status. For example, if all coordinators are **SUCCEEDED**, oozie puts the bundle job into **SUCCEEDED** status. However, if all coordinator jobs don't finish with the same status, oozie puts the bundle job into **DONEWITHERROR**.
-
-
-### 4.3. Bundle Application Definition
-A bundle definition is defined in XML by a name, controls and one or more coordinator application specifications:
-
- * **<font color="#0000ff"> name: </font>** The name for the bundle job.
- * **<font color="#0000ff"> controls: </font>** The control specification for the bundle.
- * **<font color="#0000ff"> kick-off-time: </font>** It defines when the bundle job should start and submit the coordinator applications. This field is optional and the default is **NOW** that means the job should start right-a-way.
- * **<font color="#0000ff"> coordinator: </font>** Coordinator application specification. There should be at least one coordinator application in any bundle.
- * **<font color="#0000ff"> name: </font>** Name of the coordinator application. It can be used for referring this application through bundle to control such as kill, suspend, rerun.
- * **<font color="#0000ff"> enabled: </font>** Enabled can be used to enable or disable a coordinator. It is optional. The default value for enabled is true.
- * **<font color="#0000ff"> app-path: </font>** Path of the coordinator application definition in hdfs. This is a mandatory element.
- * **<font color="#0000ff"> configuration: </font>** A hadoop like configuration to parameterize corresponding coordinator application. This is optional.
- * **<font color="#0000ff"> Parameterization: </font>** Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as `${NAME}` variables within the bundle application definition. Configuration properties that are not a valid Java identifier, for example `job.tracker`, are available via the `${bundle:conf(String name)}` function. Valid Java identifier properties are available via this function as well.
-
-
-**<font color="#800080">Syntax: </font>**
-
-
-```
- <bundle-app name=[NAME] xmlns='uri:oozie:bundle:0.1'>
- <controls>
- <kick-off-time>[DATETIME]</kick-off-time>
- </controls>
- <coordinator name=[NAME] enabled=[TRUE | FALSE] >
- <app-path>[COORD-APPLICATION-PATH]</app-path>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- </coordinator>
- ...
-</bundle-app>
-```
-
-
-**<font color="#008000"> Examples: </font>**
-
-**A Bundle Job that maintains two coordinator applications:**
-
-
-```
-<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.1'>
- <controls>
- <kick-off-time>${kickOffTime}</kick-off-time>
- </controls>
- <coordinator name="${bundle:conf('coordName1')}" >
- <app-path>${appPath}</app-path>
- <configuration>
- <property>
- <name>startTime1</name>
- <value>${bundle:conf('coord1.startTime1')}</value>
- </property>
- <property>
- <name>endTime1</name>
- <value>${END_TIME}</value>
- </property>
- </configuration>
- </coordinator>
- <coordinator name='coordJobFromBundle2' >
- <app-path>${appPath2}</app-path>
- <configuration>
- <property>
- <name>startTime2</name>
- <value>${START_TIME2}</value>
- </property>
- <property>
- <name>endTime2</name>
- <value>${END_TIME2}</value>
- </property>
- </configuration>
- </coordinator>
-</bundle-app>
-```
-
-### 4.4. Bundle Formal Parameters
-As of schema 0.2, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
-properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
-
-**Example:**
-
-The previous Bundle Job application definition with formal parameters:
-
-
-```
-<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.2'>
- <parameters>
- <property>
- <name>appPath</name>
- </property>
- <property>
- <name>appPath2</name>
- <value>hdfs://foo:8020/user/joe/job/job.properties</value>
- </property>
- </parameters>
- <controls>
- <kick-off-time>${kickOffTime}</kick-off-time>
- </controls>
- <coordinator name='coordJobFromBundle1' >
- <app-path>${appPath}</app-path>
- <configuration>
- <property>
- <name>startTime1</name>
- <value>${START_TIME}</value>
- </property>
- <property>
- <name>endTime1</name>
- <value>${END_TIME}</value>
- </property>
- </configuration>
- </coordinator>
- <coordinator name='coordJobFromBundle2' >
- <app-path>${appPath2}</app-path>
- <configuration>
- <property>
- <name>startTime2</name>
- <value>${START_TIME2}</value>
- </property>
- <property>
- <name>endTime2</name>
- <value>${END_TIME2}</value>
- </property>
- </configuration>
- </coordinator>
-</bundle-app>
-```
-
-In the above example, if `appPath` is not specified, Oozie will print an error message instead of submitting the job. If
-`appPath2` is not specified, Oozie will use the default value, `hdfs://foo:8020/user/joe/job/job.properties`.
-
-
-## 5. User Propagation
-
-When submitting a bundle job, the configuration must contain a `user.name` property. If security is enabled, Oozie must ensure that the value of the `user.name` property in the configuration match the user credentials present in the protocol (web services) request.
-
-When submitting a bundle job, the configuration may contain the `oozie.job.acl` property (the `group.name` property
-has been deprecated). If authorization is enabled, this property is treated as as the ACL for the job, it can contain
-user and group IDs separated by commas.
-
-The specified user and ACL are assigned to the created bundle job.
-
-Oozie must propagate the specified user and ACL to the system executing its children jobs (coordinator jobs).
-
-## 6. Bundle Application Deployment
-
-A bundle application consist exclusively of bundle application definition and associated coordinator application specifications. They must be installed in an HDFS directory. To submit a job for a bundle application, the full HDFS path to bundle application definition must be specified.
-
-### 6.1. Organizing Bundle Applications
-
-TBD.
-
-## 7. Bundle Job Submission
-
-When a bundle job is submitted to Oozie, the submitter must specified all the required job properties plus the HDFS path to the bundle application definition for the job.
-
-The bundle application definition HDFS path must be specified in the 'oozie.bundle.application.path' job property.
-
-All the bundle job properties, the HDFS path for the bundle application, the 'user.name' and 'oozie.job.acl' must be
-submitted to the Oozie using an XML configuration file (Hadoop XML configuration file).
-
-**<font color="#008000"> Example: </font>**:
-
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<configuration>
- <property>
- <name>user.name</name>
- <value>joe</value>
- </property>
- <property>
- <name>oozie.bundle.application.path</name>
- <value>hdfs://foo:8020/user/joe/mybundles/hello-bundle1.xml</value>
- </property>
- ...
-</configuration>
-```
-
-## 8. Bundle Rerun
-### 8.1 Rerunning a Bundle Job
-Oozie provides a way of rerunning a bundle job. The user could request to rerun a subset of coordinators within a bundle by defining a list of coordinator's names. In addition, a user could define a list of dates or ranges of dates (in UTC format) to rerun for those time windows.
-There is a way of asking whether to cleanup all output directories before rerun. By default, oozie will remove all output directories. Moreover, there is an option by which a user could ask to re-calculate the dynamic input directories defined by latest function in coordinators.
-
-### 8.2 Rerun Arguments
-
-
-```
-$oozie job -rerun <bundle_Job_id> [-coordinator <list of coordinator name separate by comma>
-[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
- [-nocleanup] [-refresh]
-```
-
- * The `rerun` option reruns a bundle job that is *not* in (`KILLED`, `FAILED`, `PREP`, `PREPPAUSED`, `PREPSUSPENDED`).
- * Rerun a bundle job that is in `PAUSED` state will reset the paused time.
- * The option -coordinator determines the name of coordinator that will be rerun. By default all coordinators are rerun.
- * Multiple ranges can be used in -date. See the above examples.
- * The dates specified in -date must be UTC.
- * If -nocleanup is given, corresponding coordinator directories will not be removed; otherwise the 'output-event' will be deleted.
- * If -refresh is set, new dataset is re-evaluated for latest() and future() for the corresponding coordinators.
- * If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked for the corresponding coordinators.
-
-
-After the command is executed the rerun bundle job will be in `RUNNING` status.
-
-Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details on rerun of coordinator job.
-
-
-## Appendixes
-
-### Appendix A, Oozie Bundle XML-Schema
-
-#### Oozie Bundle Schema 0.1
-
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.1"
- elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.1">
-
- <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/>
- <xs:simpleType name="IDENTIFIER">
- <xs:restriction base="xs:string">
- <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39})"/>
- </xs:restriction>
- </xs:simpleType>
- <xs:complexType name="BUNDLE-APP">
- <xs:sequence>
- <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/>
- <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/>
- </xs:sequence>
- <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/>
- </xs:complexType>
- <xs:complexType name="CONTROLS">
- <xs:sequence minOccurs="0" maxOccurs="1">
- <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
- <xs:complexType name="COORDINATOR">
- <xs:sequence minOccurs="1" maxOccurs="1">
- <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/>
- <xs:attribute name="critical" type="xs:string" use="optional"/>
- </xs:complexType>
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-</xs:schema>
-```
-
-#### Oozie Bundle Schema 0.2
-
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.2"
- elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.2">
-
- <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/>
- <xs:simpleType name="IDENTIFIER">
- <xs:restriction base="xs:string">
- <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}"/>
- </xs:restriction>
- </xs:simpleType>
- <xs:complexType name="BUNDLE-APP">
- <xs:sequence>
- <xs:element name="parameters" type="bundle:PARAMETERS" minOccurs="0" maxOccurs="1"/>
- <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/>
- <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/>
- </xs:sequence>
- <xs:attribute name="name" type="xs:string" use="required"/>
- </xs:complexType>
- <xs:complexType name="PARAMETERS">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="0" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
- <xs:complexType name="CONTROLS">
- <xs:sequence minOccurs="0" maxOccurs="1">
- <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
- <xs:complexType name="COORDINATOR">
- <xs:sequence minOccurs="1" maxOccurs="1">
- <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/>
- <xs:attribute name="critical" type="xs:string" use="optional"/>
- <xs:attribute name="enabled" type="xs:string" use="optional"/>
- </xs:complexType>
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-</xs:schema>
-```
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
[20/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/AG_Monitoring.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/AG_Monitoring.md b/docs/src/site/markdown/AG_Monitoring.md
new file mode 100644
index 0000000..4877dfc
--- /dev/null
+++ b/docs/src/site/markdown/AG_Monitoring.md
@@ -0,0 +1,199 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Monitoring
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Oozie Instrumentation
+
+Oozie code is instrumented in several places to collect runtime metrics. The instrumentation data can be used to
+determine the health of the system, performance of the system, and to tune the system.
+
+This comes in two flavors:
+
+ * metrics (by default enabled since 5.0.0)
+ * instrumentation (deprecated and by default disabled since 5.0.0)
+
+The instrumentation is accessible via the Admin web-services API (see the [metrics](WebServicesAPI.html#Oozie_Metrics) and
+[instrumentation](WebServicesAPI.html#Oozie_Instrumentation) Web Services API documentations for more details) and is also written on
+regular intervals to an instrumentation log.
+
+Instrumentation data includes variables, samplers, timers and counters.
+
+### Variables
+
+ * oozie
+ * version: Oozie build version.
+
+ * configuration
+ * config.dir: directory from where the configuration files are loaded. If null, all configuration files are loaded from the classpath. [Configuration files are described here](AG_Install.html#Oozie_Configuration).
+ * config.file: the Oozie custom configuration for the instance.
+
+ * jvm
+ * free.memory
+ * max.memory
+ * total.memory
+
+ * locks
+ * locks: Locks are used by Oozie to synchronize access to workflow and action entries when the database being used does not support 'select for update' queries. (MySQL supports 'select for update').
+
+ * logging
+ * config.file: Log4j '.properties' configuration file.
+ * from.classpath: whether the config file has been read from the classpath or from the config directory.
+ * reload.interval: interval at which the config file will be reloaded. 0 if the config file will never be reloaded, when loaded from the classpath is never reloaded.
+
+### Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).
+
+Poll for data over fixed interval and generate an average over the time interval. Unless specified, all samplers in
+Oozie work on a 1 minute interval.
+
+ * callablequeue
+ * delayed.queue.size: The size of the delayed command queue.
+ * queue.size: The size of the command queue.
+ * threads.active: The number of threads processing callables.
+
+ * jdbc:
+ * connections.active: Active Connections over the past minute.
+
+ * webservices: Requests to the Oozie HTTP endpoints over the last minute.
+ * admin
+ * callback
+ * job
+ * jobs
+ * requests
+ * version
+
+### Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.
+
+ * action.executors - Counters related to actions.
+ * [action_type]#action.[operation_performed] (start, end, check, kill)
+ * [action_type]#ex.[exception_type] (transient, non-transient, error, failed)
+ * e.g.
+```
+ssh#action.end: 306
+ssh#action.start: 316
+```
+
+ * callablequeue - count of events in various execution queues.
+ * delayed.queued: Number of commands queued with a delay.
+ * executed: Number of executions from the queue.
+ * failed: Number of queue attempts which failed.
+ * queued: Number of queued commands.
+
+ * commands: Execution Counts for various commands. This data is generated for all commands.
+ * action.end
+ * action.notification
+ * action.start
+ * callback
+ * job.info
+ * job.notification
+ * purge
+ * signal
+ * start
+ * submit
+
+ * jobs: Job Statistics
+ * start: Number of started jobs.
+ * submit: Number of submitted jobs.
+ * succeeded: Number of jobs which succeeded.
+ * kill: Number of killed jobs.
+
+ * authorization
+ * failed: Number of failed authorization attempts.
+
+ * webservices: Number of request to various web services along with the request type.
+ * failed: total number of failed requests.
+ * requests: total number of requests.
+ * admin
+ * admin-GET
+ * callback
+ * callback-GET
+ * jobs
+ * jobs-GET
+ * jobs-POST
+ * version
+ * version-GET
+
+### Timers - Maintain information about the time spent in various operations.
+
+ * action.executors - Counters related to actions.
+ * [action_type]#action.[operation_performed] (start, end, check, kill)
+
+ * callablequeue
+ * time.in.queue: Time a callable spent in the queue before being processed.
+
+ * commands: Generated for all Commands.
+ * action.end
+ * action.notification
+ * action.start
+ * callback
+ * job.info
+ * job.notification
+ * purge
+ * signal
+ * start
+ * submit
+
+ * db - Timers related to various database operations.
+ * create-workflow
+ * load-action
+ * load-pending-actions
+ * load-running-actions
+ * load-workflow
+ * load-workflows
+ * purge-old-workflows
+ * save-action
+ * update-action
+ * update-workflow
+
+ * webservices
+ * admin
+ * admin-GET
+ * callback
+ * callback-GET
+ * jobs
+ * jobs-GET
+ * jobs-POST
+ * version
+ * version-GET
+
+## Oozie JVM Thread Dump
+The `admin/jvminfo.jsp` servlet can be used to get some basic jvm stats and thread dump.
+For eg: `http://localhost:11000/oozie/admin/jvminfo.jsp?cpuwatch=1000&threadsort=cpu`. It takes the following optional
+query parameters:
+
+ * threadsort - The order in which the threads are sorted for display. Valid values are name, cpu, state. Default is state.
+ * cpuwatch - Time interval in milliseconds to monitor cpu usage of threads. Default value is 0.
+
+## Monitoring Database Schema Integrity
+
+Oozie stores all of its state in a database. Hence, ensuring that the database schema is correct is very important to ensuring that
+Oozie is healthy and behaves correctly. To help with this, Oozie includes a `SchemaCheckerService` which periodically runs and
+performs a series of checks on the database schema. More specifically, it checks the following:
+
+ * Existence of the required tables
+ * Existence of the required columns in each table
+ * Each column has the correct type and default value
+ * Existence of the required primary keys and indexes
+
+After each run, the `SchemaCheckerService` writes the result of the checks to the Oozie log and to the "schema-checker.status"
+instrumentation variable. If there's a problem, it will be logged at the ERROR level, while correct checks are logged at the DEBUG
+level.
+
+By default, the `SchemaCheckerService` runs every 7 days. This can be configured
+by `oozie.service.SchemaCheckerService.check.interval`
+
+By default, the `SchemaCheckerService` will consider "extra" tables, columns, and indexes to be incorrect. Advanced users who have
+added additional tables, columns, and indexes can tell Oozie to ignore these by
+setting `oozie.service.SchemaCheckerService.ignore.extras` to `false`.
+
+The `SchemaCheckerService` currently only supports MySQL, PostgreSQL, and Oracle databases. SQL Server and Derby are currently not
+supported.
+
+When Oozie HA is enabled, only one of the Oozie servers will perform the checks.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/AG_OozieUpgrade.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/AG_OozieUpgrade.md b/docs/src/site/markdown/AG_OozieUpgrade.md
new file mode 100644
index 0000000..ec24011
--- /dev/null
+++ b/docs/src/site/markdown/AG_OozieUpgrade.md
@@ -0,0 +1,86 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Upgrade
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Preparation
+
+Make sure there are not Workflows in RUNNING or SUSPENDED status, otherwise the database upgrade will fail.
+
+Shutdown Oozie and backup the Oozie database.
+
+Copy the oozie-site.xml from your current setup.
+
+## Oozie Server Upgrade
+
+Expand the new Oozie tarball in a new location.
+
+Edit the new `oozie-site.xml` setting all custom properties values from the old `oozie-site.xml`
+
+IMPORTANT: From Oozie 2.x to Oozie 3.x the names of the database configuration properties have
+changed. Their prefix has changed from `oozie.service.StoreService.*` to `oozie.service.JPAService.*`.
+Make sure you are using the new prefix.
+
+After upgrading the Oozie server, the `oozie-setup.sh` MUST be rerun before starting the
+upgraded Oozie server.
+
+Oozie database migration is required when there Oozie database schema changes, like
+upgrading from Oozie 2.x to Oozie 3.x.
+
+Configure the oozie-site.xml with the correct database configuration properties as
+explained in the 'Database Configuration' section in [Oozie Install](AG_Install.html).
+
+Once `oozie-site.xml` has been configured with the database configuration execute the `ooziedb.sh`
+command line tool to upgrade the database:
+
+
+```
+$ bin/ooziedb.sh upgrade -run
+
+Validate DB Connection.
+DONE
+Check DB schema exists
+DONE
+Check OOZIE_SYS table does not exist
+DONE
+Verify there are not active Workflow Jobs
+DONE
+Create SQL schema
+DONE
+DONE
+Create OOZIE_SYS table
+DONE
+Upgrade COORD_JOBS new columns default values.
+DONE
+Upgrade COORD_JOBS & COORD_ACTIONS status values.
+DONE
+Table 'WF_ACTIONS' column 'execution_path', length changed to 1024
+DONE
+
+Oozie DB has been upgraded to Oozie version '3.2.0'
+
+The SQL commands have been written to: /tmp/ooziedb-5737263881793872034.sql
+
+$
+```
+
+The new version of the Oozie server is ready to be started.
+
+NOTE: If using MySQL or Oracle, copy the corresponding JDBC driver JAR file to the `libext/` directory before running
+the `ooziedb.sh` command line tool.
+
+NOTE: If instead using the '-run' option, the `-sqlfile <FILE>` option is used, then all the
+database changes will be written to the specified file and the database won't be modified.
+
+## Oozie Client Upgrade
+
+While older Oozie clients work with newer Oozie server, to have access to all the
+functionality of the Oozie server the same version of Oozie client should be installed
+and used by users.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/BundleFunctionalSpec.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/BundleFunctionalSpec.md b/docs/src/site/markdown/BundleFunctionalSpec.md
new file mode 100644
index 0000000..5301429
--- /dev/null
+++ b/docs/src/site/markdown/BundleFunctionalSpec.md
@@ -0,0 +1,418 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Bundle Specification
+
+The goal of this document is to define a new oozie abstraction called bundle system specialized in submitting and maintaining a set of coordinator applications.
+
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Changelog
+
+## 1. Bundle Overview
+
+Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun in the bundle level resulting a better and easy operational control.
+
+More specifically, the oozie **Bundle** system allows the user to define and execute a bunch of coordinator applications often called a data pipeline. There is no explicit dependency among the coordinator applications in a bundle. However, a user could use the data dependency of coordinator applications to create an implicit data application pipeline.
+
+
+## 2. Definitions
+
+**Kick-off-time:** The time when a bundle should start and submit coordinator applications.
+
+**Bundle Application:** A bundle application defines a set of coordinator applications and when to start those. Normally, bundle applications are parameterized. A bundle application is written in XML.
+
+**Bundle Job:** A bundle job is an executable instance of a bundle application. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
+
+**Bundle Definition Language:** The language used to describe bundle applications.
+
+## 3. Expression Language for Parameterization
+
+Bundle application definitions can be parameterized with variables.
+
+At job submission time all the parameters are resolved into concrete values.
+
+The parameterization of bundle definitions is done using JSP Expression Language syntax from the [JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to support variables as parameters but also complex expressions.
+
+EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.
+
+
+## 4. Bundle Job
+
+### 4.1. Bundle Job Status
+
+At any time, a bundle job is in one of the following status: **PREP, RUNNING, RUNNINGWITHERROR, SUSPENDED, PREPSUSPENDED, SUSPENDEDWITHERROR, PAUSED, PAUSEDWITHERROR, PREPPAUSED, SUCCEEDED, DONEWITHERROR, KILLED, FAILED**.
+
+### 4.2. Transitions of Bundle Job Status
+
+Valid bundle job status transitions are:
+
+ * **PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED**
+ * **RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED**
+ * **RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | FAILED | KILLED**
+ * **PREPSUSPENDED --> PREP | KILLED**
+ * **SUSPENDED --> RUNNING | KILLED**
+ * **SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED**
+ * **PREPPAUSED --> PREP | KILLED**
+ * **PAUSED --> SUSPENDED | RUNNING | KILLED**
+ * **PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED**
+
+### 4.3. Details of Status Transitions
+When a bundle job is submitted, oozie parses the bundle job XML. Oozie then creates a record for the bundle with status **PREP** and returns a unique ID.
+
+When a user requests to suspend a bundle job that is in **PREP** state, oozie puts the job in status **PREPSUSPENDED**. Similarly, when pause time reaches for a bundle job with **PREP** status, oozie puts the job in status **PREPPAUSED**.
+
+Conversely, when a user requests to resume a **PREPSUSPENDED** bundle job, oozie puts the job in status **PREP**. And when pause time is reset for a bundle job that is in **PREPPAUSED** state, oozie puts the job in status **PREP**.
+
+There are two ways a bundle job could be started.
+
+* If `kick-off-time` (defined in the bundle xml) reaches. The default value is null which means starts coordinators NOW.
+
+* If user sends a start request to START the bundle.
+
+When a bundle job starts, oozie puts the job in status **RUNNING** and it submits all the coordinator jobs. If any coordinator job goes to **FAILED/KILLED/DONEWITHERROR** state, the bundle job is put in **RUNNINGWITHERROR**
+
+When a user requests to kill a bundle job, oozie puts the job in status **KILLED** and it sends kill to all submitted coordinator jobs.
+
+When a user requests to suspend a bundle job that is in **RUNNING** status, oozie puts the job in status **SUSPENDED** and it suspends all submitted coordinator jobs. Similarly, when a user requests to suspend a bundle job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **SUSPENDEDWITHERROR** and it suspends all submitted coordinator jobs.
+
+When pause time reaches for a bundle job that is in **RUNNING** status, oozie puts the job in status **PAUSED**. When pause time reaches for a bundle job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **PAUSEDWITHERROR**.
+
+Conversely, when a user requests to resume a **SUSPENDED** bundle job, oozie puts the job in status **RUNNING**. Similarly, when a user requests to resume a **SUSPENDEDWITHERROR** bundle job, oozie puts the job in status **RUNNINGWITHERROR**. And when pause time is reset for a bundle job and job status is **PAUSED**, oozie puts the job in status **RUNNING**. Similarly, when the pause time is reset for a bundle job and job status is **PAUSEDWITHERROR**, oozie puts the job in status **RUNNINGWITHERROR**
+
+When all the coordinator jobs finish, oozie updates the bundle status accordingly. If all coordinators reaches to the _same_ terminal state, bundle job status also move to the same status. For example, if all coordinators are **SUCCEEDED**, oozie puts the bundle job into **SUCCEEDED** status. However, if all coordinator jobs don't finish with the same status, oozie puts the bundle job into **DONEWITHERROR**.
+
+
+### 4.3. Bundle Application Definition
+A bundle definition is defined in XML by a name, controls and one or more coordinator application specifications:
+
+ * **<font color="#0000ff"> name: </font>** The name for the bundle job.
+ * **<font color="#0000ff"> controls: </font>** The control specification for the bundle.
+ * **<font color="#0000ff"> kick-off-time: </font>** It defines when the bundle job should start and submit the coordinator applications. This field is optional and the default is **NOW** that means the job should start right-a-way.
+ * **<font color="#0000ff"> coordinator: </font>** Coordinator application specification. There should be at least one coordinator application in any bundle.
+ * **<font color="#0000ff"> name: </font>** Name of the coordinator application. It can be used for referring this application through bundle to control such as kill, suspend, rerun.
+ * **<font color="#0000ff"> enabled: </font>** Enabled can be used to enable or disable a coordinator. It is optional. The default value for enabled is true.
+ * **<font color="#0000ff"> app-path: </font>** Path of the coordinator application definition in hdfs. This is a mandatory element.
+ * **<font color="#0000ff"> configuration: </font>** A hadoop like configuration to parameterize corresponding coordinator application. This is optional.
+ * **<font color="#0000ff"> Parameterization: </font>** Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as `${NAME}` variables within the bundle application definition. Configuration properties that are not a valid Java identifier, for example `job.tracker`, are available via the `${bundle:conf(String name)}` function. Valid Java identifier properties are available via this function as well.
+
+
+**<font color="#800080">Syntax: </font>**
+
+
+```
+ <bundle-app name=[NAME] xmlns='uri:oozie:bundle:0.1'>
+ <controls>
+ <kick-off-time>[DATETIME]</kick-off-time>
+ </controls>
+ <coordinator name=[NAME] enabled=[TRUE | FALSE] >
+ <app-path>[COORD-APPLICATION-PATH]</app-path>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ </coordinator>
+ ...
+</bundle-app>
+```
+
+
+**<font color="#008000"> Examples: </font>**
+
+**A Bundle Job that maintains two coordinator applications:**
+
+
+```
+<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.1'>
+ <controls>
+ <kick-off-time>${kickOffTime}</kick-off-time>
+ </controls>
+ <coordinator name="${bundle:conf('coordName1')}" >
+ <app-path>${appPath}</app-path>
+ <configuration>
+ <property>
+ <name>startTime1</name>
+ <value>${bundle:conf('coord1.startTime1')}</value>
+ </property>
+ <property>
+ <name>endTime1</name>
+ <value>${END_TIME}</value>
+ </property>
+ </configuration>
+ </coordinator>
+ <coordinator name='coordJobFromBundle2' >
+ <app-path>${appPath2}</app-path>
+ <configuration>
+ <property>
+ <name>startTime2</name>
+ <value>${START_TIME2}</value>
+ </property>
+ <property>
+ <name>endTime2</name>
+ <value>${END_TIME2}</value>
+ </property>
+ </configuration>
+ </coordinator>
+</bundle-app>
+```
+
+### 4.4. Bundle Formal Parameters
+As of schema 0.2, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
+properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
+
+**Example:**
+
+The previous Bundle Job application definition with formal parameters:
+
+
+```
+<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.2'>
+ <parameters>
+ <property>
+ <name>appPath</name>
+ </property>
+ <property>
+ <name>appPath2</name>
+ <value>hdfs://foo:8020/user/joe/job/job.properties</value>
+ </property>
+ </parameters>
+ <controls>
+ <kick-off-time>${kickOffTime}</kick-off-time>
+ </controls>
+ <coordinator name='coordJobFromBundle1' >
+ <app-path>${appPath}</app-path>
+ <configuration>
+ <property>
+ <name>startTime1</name>
+ <value>${START_TIME}</value>
+ </property>
+ <property>
+ <name>endTime1</name>
+ <value>${END_TIME}</value>
+ </property>
+ </configuration>
+ </coordinator>
+ <coordinator name='coordJobFromBundle2' >
+ <app-path>${appPath2}</app-path>
+ <configuration>
+ <property>
+ <name>startTime2</name>
+ <value>${START_TIME2}</value>
+ </property>
+ <property>
+ <name>endTime2</name>
+ <value>${END_TIME2}</value>
+ </property>
+ </configuration>
+ </coordinator>
+</bundle-app>
+```
+
+In the above example, if `appPath` is not specified, Oozie will print an error message instead of submitting the job. If
+`appPath2` is not specified, Oozie will use the default value, `hdfs://foo:8020/user/joe/job/job.properties`.
+
+
+## 5. User Propagation
+
+When submitting a bundle job, the configuration must contain a `user.name` property. If security is enabled, Oozie must ensure that the value of the `user.name` property in the configuration match the user credentials present in the protocol (web services) request.
+
+When submitting a bundle job, the configuration may contain the `oozie.job.acl` property (the `group.name` property
+has been deprecated). If authorization is enabled, this property is treated as as the ACL for the job, it can contain
+user and group IDs separated by commas.
+
+The specified user and ACL are assigned to the created bundle job.
+
+Oozie must propagate the specified user and ACL to the system executing its children jobs (coordinator jobs).
+
+## 6. Bundle Application Deployment
+
+A bundle application consist exclusively of bundle application definition and associated coordinator application specifications. They must be installed in an HDFS directory. To submit a job for a bundle application, the full HDFS path to bundle application definition must be specified.
+
+### 6.1. Organizing Bundle Applications
+
+TBD.
+
+## 7. Bundle Job Submission
+
+When a bundle job is submitted to Oozie, the submitter must specified all the required job properties plus the HDFS path to the bundle application definition for the job.
+
+The bundle application definition HDFS path must be specified in the 'oozie.bundle.application.path' job property.
+
+All the bundle job properties, the HDFS path for the bundle application, the 'user.name' and 'oozie.job.acl' must be
+submitted to the Oozie using an XML configuration file (Hadoop XML configuration file).
+
+**<font color="#008000"> Example: </font>**:
+
+
+```
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>user.name</name>
+ <value>joe</value>
+ </property>
+ <property>
+ <name>oozie.bundle.application.path</name>
+ <value>hdfs://foo:8020/user/joe/mybundles/hello-bundle1.xml</value>
+ </property>
+ ...
+</configuration>
+```
+
+## 8. Bundle Rerun
+### 8.1 Rerunning a Bundle Job
+Oozie provides a way of rerunning a bundle job. The user could request to rerun a subset of coordinators within a bundle by defining a list of coordinator's names. In addition, a user could define a list of dates or ranges of dates (in UTC format) to rerun for those time windows.
+There is a way of asking whether to cleanup all output directories before rerun. By default, oozie will remove all output directories. Moreover, there is an option by which a user could ask to re-calculate the dynamic input directories defined by latest function in coordinators.
+
+### 8.2 Rerun Arguments
+
+
+```
+$oozie job -rerun <bundle_Job_id> [-coordinator <list of coordinator name separate by comma>
+[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
+ [-nocleanup] [-refresh]
+```
+
+ * The `rerun` option reruns a bundle job that is *not* in (`KILLED`, `FAILED`, `PREP`, `PREPPAUSED`, `PREPSUSPENDED`).
+ * Rerun a bundle job that is in `PAUSED` state will reset the paused time.
+ * The option -coordinator determines the name of coordinator that will be rerun. By default all coordinators are rerun.
+ * Multiple ranges can be used in -date. See the above examples.
+ * The dates specified in -date must be UTC.
+ * If -nocleanup is given, corresponding coordinator directories will not be removed; otherwise the 'output-event' will be deleted.
+ * If -refresh is set, new dataset is re-evaluated for latest() and future() for the corresponding coordinators.
+ * If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked for the corresponding coordinators.
+
+
+After the command is executed the rerun bundle job will be in `RUNNING` status.
+
+Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details on rerun of coordinator job.
+
+
+## Appendixes
+
+### Appendix A, Oozie Bundle XML-Schema
+
+#### Oozie Bundle Schema 0.1
+
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.1"
+ elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.1">
+
+ <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/>
+ <xs:simpleType name="IDENTIFIER">
+ <xs:restriction base="xs:string">
+ <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39})"/>
+ </xs:restriction>
+ </xs:simpleType>
+ <xs:complexType name="BUNDLE-APP">
+ <xs:sequence>
+ <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/>
+ </xs:sequence>
+ <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/>
+ </xs:complexType>
+ <xs:complexType name="CONTROLS">
+ <xs:sequence minOccurs="0" maxOccurs="1">
+ <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+ <xs:complexType name="COORDINATOR">
+ <xs:sequence minOccurs="1" maxOccurs="1">
+ <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/>
+ <xs:attribute name="critical" type="xs:string" use="optional"/>
+ </xs:complexType>
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+</xs:schema>
+```
+
+#### Oozie Bundle Schema 0.2
+
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.2"
+ elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.2">
+
+ <xs:element name="bundle-app" type="bundle:BUNDLE-APP"/>
+ <xs:simpleType name="IDENTIFIER">
+ <xs:restriction base="xs:string">
+ <xs:pattern value="([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}"/>
+ </xs:restriction>
+ </xs:simpleType>
+ <xs:complexType name="BUNDLE-APP">
+ <xs:sequence>
+ <xs:element name="parameters" type="bundle:PARAMETERS" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="controls" type="bundle:CONTROLS" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="coordinator" type="bundle:COORDINATOR" minOccurs="1" maxOccurs="unbounded"/>
+ </xs:sequence>
+ <xs:attribute name="name" type="xs:string" use="required"/>
+ </xs:complexType>
+ <xs:complexType name="PARAMETERS">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ <xs:complexType name="CONTROLS">
+ <xs:sequence minOccurs="0" maxOccurs="1">
+ <xs:element name="kick-off-time" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+ <xs:complexType name="COORDINATOR">
+ <xs:sequence minOccurs="1" maxOccurs="1">
+ <xs:element name="app-path" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="configuration" type="bundle:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ <xs:attribute name="name" type="bundle:IDENTIFIER" use="required"/>
+ <xs:attribute name="critical" type="xs:string" use="optional"/>
+ <xs:attribute name="enabled" type="xs:string" use="optional"/>
+ </xs:complexType>
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+</xs:schema>
+```
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
[02/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
deleted file mode 100644
index 463635b..0000000
--- a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
+++ /dev/null
@@ -1,5457 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Specification, a Hadoop Workflow System
-**<center>(v5.0)</center>**
-
-The goal of this document is to define a workflow engine system specialized in coordinating the execution of Hadoop
-Map/Reduce and Pig jobs.
-
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Changelog
-
-
-**2016FEB19**
-
- * 3.2.7 Updated notes on System.exit(int n) behavior
-
-**2015APR29**
-
- * 3.2.1.4 Added notes about Java action retries
- * 3.2.7 Added notes about Java action retries
-
-**2014MAY08**
-
- * 3.2.2.4 Added support for fully qualified job-xml path
-
-**2013JUL03**
-
- * Appendix A, Added new workflow schema 0.5 and SLA schema 0.2
-
-**2012AUG30**
-
- * 4.2.2 Added two EL functions (replaceAll and appendAll)
-
-**2012JUL26**
-
- * Appendix A, updated XML schema 0.4 to include `parameters` element
- * 4.1 Updated to mention about `parameters` element as of schema 0.4
-
-**2012JUL23**
-
- * Appendix A, updated XML schema 0.4 (Fs action)
- * 3.2.4 Updated to mention that a `name-node`, a `job-xml`, and a `configuration` element are allowed in the Fs action as of
-schema 0.4
-
-**2012JUN19**
-
- * Appendix A, added XML schema 0.4
- * 3.2.2.4 Updated to mention that multiple `job-xml` elements are allowed as of schema 0.4
- * 3.2.3 Updated to mention that multiple `job-xml` elements are allowed as of schema 0.4
-
-**2011AUG17**
-
- * 3.2.4 fs 'chmod' xml closing element typo in Example corrected
-
-**2011AUG12**
-
- * 3.2.4 fs 'move' action characteristics updated, to allow for consistent source and target paths and existing target path only if directory
- * 18, Update the doc for user-retry of workflow action.
-
-**2011FEB19**
-
- * 10, Update the doc to rerun from the failed node.
-
-**2010OCT31**
-
- * 17, Added new section on Shared Libraries
-
-**2010APR27**
-
- * 3.2.3 Added new "arguments" tag to PIG actions
- * 3.2.5 SSH actions are deprecated in Oozie schema 0.1 and removed in Oozie schema 0.2
- * Appendix A, Added schema version 0.2
-
-
-**2009OCT20**
-
- * Appendix A, updated XML schema
-
-
-**2009SEP15**
-
- * 3.2.6 Removing support for sub-workflow in a different Oozie instance (removing the 'oozie' element)
-
-
-**2009SEP07**
-
- * 3.2.2.3 Added Map Reduce Pipes specifications.
- * 3.2.2.4 Map-Reduce Examples. Previously was 3.2.2.3.
-
-
-**2009SEP02**
-
- * 10 Added missing skip nodes property name.
- * 3.2.1.4 Reworded action recovery explanation.
-
-
-**2009AUG26**
-
- * 3.2.9 Added `java` action type
- * 3.1.4 Example uses EL constant to refer to counter group/name
-
-
-**2009JUN09**
-
- * 12.2.4 Added build version resource to admin end-point
- * 3.2.6 Added flag to propagate workflow configuration to sub-workflows
- * 10 Added behavior for workflow job parameters given in the rerun
- * 11.3.4 workflows info returns pagination information
-
-
-**2009MAY18**
-
- * 3.1.4 decision node, 'default' element, 'name' attribute changed to 'to'
- * 3.1.5 fork node, 'transition' element changed to 'start', 'to' attribute change to 'path'
- * 3.1.5 join node, 'transition' element remove, added 'to' attribute to 'join' element
- * 3.2.1.4 Rewording on action recovery section
- * 3.2.2 map-reduce action, added 'job-tracker', 'name-node' actions, 'file', 'file' and 'archive' elements
- * 3.2.2.1 map-reduce action, remove from 'streaming' element 'file', 'file' and 'archive' elements
- * 3.2.2.2 map-reduce action, reorganized streaming section
- * 3.2.3 pig action, removed information about implementation (SSH), changed elements names
- * 3.2.4 fs action, removed 'fs-uri' and 'user-name' elements, file system URI is now specified in path, user is propagated
- * 3.2.6 sub-workflow action, renamed elements 'oozie-url' to 'oozie' and 'workflow-app' to 'app-path'
- * 4 Properties that are valid Java identifiers can be used as ${NAME}
- * 4.1 Renamed default properties file from 'configuration.xml' to 'default-configuration.xml'
- * 4.2 Changes in EL Constants and Functions
- * 5 Updated notification behavior and tokens
- * 6 Changed user propagation behavior
- * 7 Changed application packaging from ZIP to HDFS directory
- * Removed application lifecycle and self containment model sections
- * 10 Changed workflow job recovery, simplified recovery behavior
- * 11 Detailed Web Services API
- * 12 Updated Client API section
- * 15 Updated Action Executor API section
- * Appendix A XML namespace updated to 'uri:oozie:workflow:0.1'
- * Appendix A Updated XML schema to changes in map-reduce/pig/fs/ssh actions
- * Appendix B Updated workflow example to schema changes
-
-
-**2009MAR25**
-
- * Changing all references of HWS to Oozie (project name)
- * Typos, XML Formatting
- * XML Schema URI correction
-
-
-**2009MAR09**
-
- * Changed `CREATED` job state to `PREP` to have same states as Hadoop
- * Renamed 'hadoop-workflow' element to 'workflow-app'
- * Decision syntax changed to be 'switch/case' with no transition indirection
- * Action nodes common root element 'action', with the action type as sub-element (using a single built-in XML schema)
- * Action nodes have 2 explicit transitions 'ok to' and 'error to' enforced by XML schema
- * Renamed 'fail' action element to 'kill'
- * Renamed 'hadoop' action element to 'map-reduce'
- * Renamed 'hdfs' action element to 'fs'
- * Updated all XML snippets and examples
- * Made user propagation simpler and consistent
- * Added Oozie XML schema to Appendix A
- * Added workflow example to Appendix B
-
-
-**2009FEB22**
-
- * Opened [JIRA HADOOP-5303](https://issues.apache.org/jira/browse/HADOOP-5303)
-
-
-**27/DEC/2012:**
-
- * Added information on dropping hcatalog table partitions in prepare block
- * Added hcatalog EL functions section
-
-## 0 Definitions
-
-**Action:** An execution/computation task (Map-Reduce job, Pig job, a shell command). It can also be referred as task or
-'action node'.
-
-**Workflow:** A collection of actions arranged in a control dependency DAG (Directed Acyclic Graph). "control dependency"
-from one action to another means that the second action can't run until the first action has completed.
-
-**Workflow Definition:** A programmatic description of a workflow that can be executed.
-
-**Workflow Definition Language:** The language used to define a Workflow Definition.
-
-**Workflow Job:** An executable instance of a workflow definition.
-
-**Workflow Engine:** A system that executes workflows jobs. It can also be referred as a DAG engine.
-
-## 1 Specification Highlights
-
-A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, and
-sub-workflows.
-
-Flow control operations within the workflow applications can be done using decision, fork and join nodes. Cycles in
-workflows are not supported.
-
-Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow
-definition as `${VAR}` variables.
-
-A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to
-run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig
-scripts, and other resource files.
-
-Before running a workflow job, the corresponding workflow application must be deployed in Oozie.
-
-Deploying workflow application and running workflow jobs can be done via command line tools, a WS API and a Java API.
-
-Monitoring the system and workflow jobs can be done via a web console, command line tools, a WS API and a Java API.
-
-When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions
-must be provided. This set of properties is a Hadoop configuration.
-
-Possible states for a workflow jobs are: `PREP`, `RUNNING`, `SUSPENDED`, `SUCCEEDED`, `KILLED` and `FAILED`.
-
-In the case of a action start failure in a workflow job, depending on the type of failure, Oozie will attempt automatic
-retries, it will request a manual retry or it will fail the workflow job.
-
-Oozie can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.
-
-In the case of workflow job failure, the workflow job can be resubmitted skipping previously completed actions.
-Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow
-application code.
-
-<a name="WorkflowDefinition"></a>
-## 2 Workflow Definition
-
-A workflow definition is a DAG with control flow nodes (start, end, decision, fork, join, kill) or action nodes
-(map-reduce, pig, etc.), nodes are connected by transitions arrows.
-
-The workflow definition language is XML based and it is called hPDL (Hadoop Process Definition Language).
-
-Refer to the Appendix A for the[Oozie Workflow Definition XML Schema](WorkflowFunctionalSpec.html#OozieWFSchema). Appendix
-B has [Workflow Definition Examples](WorkflowFunctionalSpec.html#OozieWFExamples).
-
-### 2.1 Cycles in Workflow Definitions
-
-Oozie does not support cycles in workflow definitions, workflow definitions must be a strict DAG.
-
-At workflow application deployment time, if Oozie detects a cycle in the workflow definition it must fail the
-deployment.
-
-## 3 Workflow Nodes
-
-Workflow nodes are classified in control flow nodes and action nodes:
-
- * **Control flow nodes:** nodes that control the start and end of the workflow and workflow job execution path.
- * **Action nodes:** nodes that trigger the execution of a computation/processing task.
-
-Node names and transitions must be conform to the following pattern `[a-zA-Z][\-_a-zA-Z0-0]*`, of up to 20 characters
-long.
-
-### 3.1 Control Flow Nodes
-
-Control flow nodes define the beginning and the end of a workflow (the `start`, `end` and `kill` nodes) and provide a
-mechanism to control the workflow execution path (the `decision`, `fork` and `join` nodes).
-
-<a name="StartNode"></a>
-#### 3.1.1 Start Control Node
-
-The `start` node is the entry point for a workflow job, it indicates the first workflow node the workflow job must
-transition to.
-
-When a workflow is started, it automatically transitions to the node specified in the `start`.
-
-A workflow definition must have one `start` node.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <start to="[NODE-NAME]"/>
- ...
-</workflow-app>
-```
-
-The `to` attribute is the name of first workflow node to execute.
-
-**Example:**
-
-
-```
-<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <start to="firstHadoopJob"/>
- ...
-</workflow-app>
-```
-
-<a name="EndNode"></a>
-#### 3.1.2 End Control Node
-
-The `end` node is the end for a workflow job, it indicates that the workflow job has completed successfully.
-
-When a workflow job reaches the `end` it finishes successfully (SUCCEEDED).
-
-If one or more actions started by the workflow job are executing when the `end` node is reached, the actions will be
-killed. In this scenario the workflow job is still considered as successfully run.
-
-A workflow definition must have one `end` node.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <end name="[NODE-NAME]"/>
- ...
-</workflow-app>
-```
-
-The `name` attribute is the name of the transition to do to end the workflow job.
-
-**Example:**
-
-
-```
-<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <end name="end"/>
-</workflow-app>
-```
-
-<a name="KillNode"></a>
-#### 3.1.3 Kill Control Node
-
-The `kill` node allows a workflow job to kill itself.
-
-When a workflow job reaches the `kill` it finishes in error (KILLED).
-
-If one or more actions started by the workflow job are executing when the `kill` node is reached, the actions will be
-killed.
-
-A workflow definition may have zero or more `kill` nodes.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <kill name="[NODE-NAME]">
- <message>[MESSAGE-TO-LOG]</message>
- </kill>
- ...
-</workflow-app>
-```
-
-The `name` attribute in the `kill` node is the name of the Kill action node.
-
-The content of the `message` element will be logged as the kill reason for the workflow job.
-
-A `kill` node does not have transition elements because it ends the workflow job, as `KILLED`.
-
-**Example:**
-
-
-```
-<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <kill name="killBecauseNoInput">
- <message>Input unavailable</message>
- </kill>
- ...
-</workflow-app>
-```
-
-<a name="DecisionNode"></a>
-#### 3.1.4 Decision Control Node
-
-A `decision` node enables a workflow to make a selection on the execution path to follow.
-
-The behavior of a `decision` node can be seen as a switch-case statement.
-
-A `decision` node consists of a list of predicates-transition pairs plus a default transition. Predicates are evaluated
-in order or appearance until one of them evaluates to `true` and the corresponding transition is taken. If none of the
-predicates evaluates to `true` the `default` transition is taken.
-
-Predicates are JSP Expression Language (EL) expressions (refer to section 4.2 of this document) that resolve into a
-boolean value, `true` or `false`. For example:
-
-
-```
- ${fs:fileSize('/usr/foo/myinputdir') gt 10 * GB}
-```
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <decision name="[NODE-NAME]">
- <switch>
- <case to="[NODE_NAME]">[PREDICATE]</case>
- ...
- <case to="[NODE_NAME]">[PREDICATE]</case>
- <default to="[NODE_NAME]"/>
- </switch>
- </decision>
- ...
-</workflow-app>
-```
-
-The `name` attribute in the `decision` node is the name of the decision node.
-
-Each `case` elements contains a predicate and a transition name. The predicate ELs are evaluated
-in order until one returns `true` and the corresponding transition is taken.
-
-The `default` element indicates the transition to take if none of the predicates evaluates
-to `true`.
-
-All decision nodes must have a `default` element to avoid bringing the workflow into an error
-state if none of the predicates evaluates to true.
-
-**Example:**
-
-
-```
-<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <decision name="mydecision">
- <switch>
- <case to="reconsolidatejob">
- ${fs:fileSize(secondjobOutputDir) gt 10 * GB}
- </case> <case to="rexpandjob">
- ${fs:fileSize(secondjobOutputDir) lt 100 * MB}
- </case>
- <case to="recomputejob">
- ${ hadoop:counters('secondjob')[RECORDS][REDUCE_OUT] lt 1000000 }
- </case>
- <default to="end"/>
- </switch>
- </decision>
- ...
-</workflow-app>
-```
-
-<a name="ForkJoinNodes"></a>
-#### 3.1.5 Fork and Join Control Nodes
-
-A `fork` node splits one path of execution into multiple concurrent paths of execution.
-
-A `join` node waits until every concurrent execution path of a previous `fork` node arrives to it.
-
-The `fork` and `join` nodes must be used in pairs. The `join` node assumes concurrent execution paths are children of
-the same `fork` node.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <fork name="[FORK-NODE-NAME]">
- <path start="[NODE-NAME]" />
- ...
- <path start="[NODE-NAME]" />
- </fork>
- ...
- <join name="[JOIN-NODE-NAME]" to="[NODE-NAME]" />
- ...
-</workflow-app>
-```
-
-The `name` attribute in the `fork` node is the name of the workflow fork node. The `start` attribute in the `path`
-elements in the `fork` node indicate the name of the workflow node that will be part of the concurrent execution paths.
-
-The `name` attribute in the `join` node is the name of the workflow join node. The `to` attribute in the `join` node
-indicates the name of the workflow node that will executed after all concurrent execution paths of the corresponding
-fork arrive to the join node.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <fork name="forking">
- <path start="firstparalleljob"/>
- <path start="secondparalleljob"/>
- </fork>
- <action name="firstparallejob">
- <map-reduce>
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <job-xml>job1.xml</job-xml>
- </map-reduce>
- <ok to="joining"/>
- <error to="kill"/>
- </action>
- <action name="secondparalleljob">
- <map-reduce>
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <job-xml>job2.xml</job-xml>
- </map-reduce>
- <ok to="joining"/>
- <error to="kill"/>
- </action>
- <join name="joining" to="nextaction"/>
- ...
-</workflow-app>
-```
-
-By default, Oozie performs some validation that any forking in a workflow is valid and won't lead to any incorrect behavior or
-instability. However, if Oozie is preventing a workflow from being submitted and you are very certain that it should work, you can
-disable forkjoin validation so that Oozie will accept the workflow. To disable this validation just for a specific workflow, simply
-set `oozie.wf.validate.ForkJoin` to `false` in the job.properties file. To disable this validation for all workflows, simply set
-`oozie.validate.ForkJoin` to `false` in the oozie-site.xml file. Disabling this validation is determined by the AND of both of
-these properties, so it will be disabled if either or both are set to false and only enabled if both are set to true (or not
-specified).
-
-<a name="ActionNodes"></a>
-### 3.2 Workflow Action Nodes
-
-Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task.
-
-#### 3.2.1 Action Basis
-
-The following sub-sections define common behavior and capabilities for all action types.
-
-##### 3.2.1.1 Action Computation/Processing Is Always Remote
-
-All computation/processing tasks triggered by an action node are remote to Oozie. No workflow application specific
-computation/processing task is executed within Oozie.
-
-##### 3.2.1.2 Actions Are Asynchronous
-
-All computation/processing tasks triggered by an action node are executed asynchronously by Oozie. For most types of
-computation/processing tasks triggered by workflow action, the workflow job has to wait until the
-computation/processing task completes before transitioning to the following node in the workflow.
-
-The exception is the `fs` action that is handled as a synchronous action.
-
-Oozie can detect completion of computation/processing tasks by two different means, callbacks and polling.
-
-When a computation/processing tasks is started by Oozie, Oozie provides a unique callback URL to the task, the task
-should invoke the given URL to notify its completion.
-
-For cases that the task failed to invoke the callback URL for any reason (i.e. a transient network failure) or when
-the type of task cannot invoke the callback URL upon completion, Oozie has a mechanism to poll computation/processing
-tasks for completion.
-
-##### 3.2.1.3 Actions Have 2 Transitions, `ok` and `error`
-
-If a computation/processing task -triggered by a workflow- completes successfully, it transitions to `ok`.
-
-If a computation/processing task -triggered by a workflow- fails to complete successfully, its transitions to `error`.
-
-If a computation/processing task exits in error, there computation/processing task must provide `error-code` and
- `error-message` information to Oozie. This information can be used from `decision` nodes to implement a fine grain
-error handling at workflow application level.
-
-Each action type must clearly define all the error codes it can produce.
-
-##### 3.2.1.4 Action Recovery
-
-Oozie provides recovery capabilities when starting or ending actions.
-
-Once an action starts successfully Oozie will not retry starting the action if the action fails during its execution.
-The assumption is that the external system (i.e. Hadoop) executing the action has enough resilience to recover jobs
-once it has started (i.e. Hadoop task retries).
-
-Java actions are a special case with regard to retries. Although Oozie itself does not retry Java actions
-should they fail after they have successfully started, Hadoop itself can cause the action to be restarted due to a
-map task retry on the map task running the Java application. See the Java Action section below for more detail.
-
-For failures that occur prior to the start of the job, Oozie will have different recovery strategies depending on the
-nature of the failure.
-
-If the failure is of transient nature, Oozie will perform retries after a pre-defined time interval. The number of
-retries and timer interval for a type of action must be pre-configured at Oozie level. Workflow jobs can override such
-configuration.
-
-Examples of a transient failures are network problems or a remote system temporary unavailable.
-
-If the failure is of non-transient nature, Oozie will suspend the workflow job until an manual or programmatic
-intervention resumes the workflow job and the action start or end is retried. It is the responsibility of an
-administrator or an external managing system to perform any necessary cleanup before resuming the workflow job.
-
-If the failure is an error and a retry will not resolve the problem, Oozie will perform the error transition for the
-action.
-
-<a name="MapReduceAction"></a>
-#### 3.2.2 Map-Reduce Action
-
-The `map-reduce` action starts a Hadoop map/reduce job from a workflow. Hadoop jobs can be Java Map/Reduce jobs or
-streaming jobs.
-
-A `map-reduce` action can be configured to perform file system cleanup and directory creation before starting the
-map reduce job. This capability enables Oozie to retry a Hadoop job in the situation of a transient failure (Hadoop
-checks the non-existence of the job output directory and then creates it when the Hadoop job is starting, thus a retry
-without cleanup of the job output directory would fail).
-
-The workflow job will wait until the Hadoop map/reduce job completes before continuing to the next action in the
-workflow execution path.
-
-The counters of the Hadoop job and job exit status (`FAILED`, `KILLED` or `SUCCEEDED`) must be available to the
-workflow job after the Hadoop jobs ends. This information can be used from within decision nodes and other actions
-configurations.
-
-The `map-reduce` action has to be configured with all the necessary Hadoop JobConf properties to run the Hadoop
-map/reduce job.
-
-Hadoop JobConf properties can be specified as part of
-
- * the `config-default.xml` or
- * JobConf XML file bundled with the workflow application or
- * \<global\> tag in workflow definition or
- * Inline `map-reduce` action configuration or
- * An implementation of OozieActionConfigurator specified by the \<config-class\> tag in workflow definition.
-
-The configuration properties are loaded in the following above order i.e. `streaming`, `job-xml`, `configuration`,
-and `config-class`, and the precedence order is later values override earlier values.
-
-Streaming and inline property values can be parameterized (templatized) using EL expressions.
-
-The Hadoop `mapred.job.tracker` and `fs.default.name` properties must not be present in the job-xml and inline
-configuration.
-
-<a name="FilesArchives"></a>
-##### 3.2.2.1 Adding Files and Archives for the Job
-
-The `file`, `archive` elements make available, to map-reduce jobs, files and archives. If the specified path is
-relative, it is assumed the file or archiver are within the application directory, in the corresponding sub-path.
-If the path is absolute, the file or archive it is expected in the given absolute path.
-
-Files specified with the `file` element, will be symbolic links in the home directory of the task.
-
-If a file is a native library (an '.so' or a '.so.#' file), it will be symlinked as and '.so' file in the task running
-directory, thus available to the task JVM.
-
-To force a symlink for a file on the task running directory, use a '#' followed by the symlink name. For example
-'mycat.sh#cat'.
-
-Refer to Hadoop distributed cache documentation for details more details on files and archives.
-
-##### 3.2.2.2 Configuring the MapReduce action with Java code
-
-Java code can be used to further configure the MapReduce action. This can be useful if you already have "driver" code for your
-MapReduce action, if you're more familiar with MapReduce's Java API, if there's some configuration that requires logic, or some
-configuration that's difficult to do in straight XML (e.g. Avro).
-
-Create a class that implements the org.apache.oozie.action.hadoop.OozieActionConfigurator interface from the "oozie-sharelib-oozie"
-artifact. It contains a single method that receives a `JobConf` as an argument. Any configuration properties set on this `JobConf`
-will be used by the MapReduce action.
-
-The OozieActionConfigurator has this signature:
-
-```
-public interface OozieActionConfigurator {
- public void configure(JobConf actionConf) throws OozieActionConfiguratorException;
-}
-```
-where `actionConf` is the `JobConf` you can update. If you need to throw an Exception, you can wrap it in
-an `OozieActionConfiguratorException`, also in the "oozie-sharelib-oozie" artifact.
-
-For example:
-
-```
-package com.example;
-
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.mapred.FileInputFormat;
-import org.apache.hadoop.mapred.FileOutputFormat;
-import org.apache.hadoop.mapred.JobConf;
-import org.apache.oozie.action.hadoop.OozieActionConfigurator;
-import org.apache.oozie.action.hadoop.OozieActionConfiguratorException;
-import org.apache.oozie.example.SampleMapper;
-import org.apache.oozie.example.SampleReducer;
-
-public class MyConfigClass implements OozieActionConfigurator {
-
- @Override
- public void configure(JobConf actionConf) throws OozieActionConfiguratorException {
- if (actionConf.getUser() == null) {
- throw new OozieActionConfiguratorException("No user set");
- }
- actionConf.setMapperClass(SampleMapper.class);
- actionConf.setReducerClass(SampleReducer.class);
- FileInputFormat.setInputPaths(actionConf, new Path("/user/" + actionConf.getUser() + "/input-data"));
- FileOutputFormat.setOutputPath(actionConf, new Path("/user/" + actionConf.getUser() + "/output"));
- ...
- }
-}
-```
-
-To use your config class in your MapReduce action, simply compile it into a jar, make the jar available to your action, and specify
-the class name in the `config-class` element (this requires at least schema 0.5):
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <map-reduce>
- ...
- <job-xml>[JOB-XML-FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <config-class>com.example.MyConfigClass</config-class>
- ...
- </map-reduce>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-Another example of this can be found in the "map-reduce" example that comes with Oozie.
-
-A useful tip: The initial `JobConf` passed to the `configure` method includes all of the properties listed in the `configuration`
-section of the MR action in a workflow. If you need to pass any information to your OozieActionConfigurator, you can simply put
-them here.
-
-<a name="StreamingMapReduceAction"></a>
-##### 3.2.2.3 Streaming
-
-Streaming information can be specified in the `streaming` element.
-
-The `mapper` and `reducer` elements are used to specify the executable/script to be used as mapper and reducer.
-
-User defined scripts must be bundled with the workflow application and they must be declared in the `files` element of
-the streaming configuration. If the are not declared in the `files` element of the configuration it is assumed they
-will be available (and in the command PATH) of the Hadoop slave machines.
-
-Some streaming jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using
-the `file` and `archive` elements described in the previous section.
-
-The Mapper/Reducer can be overridden by a `mapred.mapper.class` or `mapred.reducer.class` properties in the `job-xml`
-file or `configuration` elements.
-
-<a name="PipesMapReduceAction"></a>
-##### 3.2.2.4 Pipes
-
-Pipes information can be specified in the `pipes` element.
-
-A subset of the command line options which can be used while using the Hadoop Pipes Submitter can be specified
-via elements - `map`, `reduce`, `inputformat`, `partitioner`, `writer`, `program`.
-
-The `program` element is used to specify the executable/script to be used.
-
-User defined program must be bundled with the workflow application.
-
-Some pipe jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using
-the `file` and `archive` elements described in the previous section.
-
-Pipe properties can be overridden by specifying them in the `job-xml` file or `configuration` element.
-
-##### 3.2.2.5 Syntax
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <map-reduce>
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <streaming>
- <mapper>[MAPPER-PROCESS]</mapper>
- <reducer>[REDUCER-PROCESS]</reducer>
- <record-reader>[RECORD-READER-CLASS]</record-reader>
- <record-reader-mapping>[NAME=VALUE]</record-reader-mapping>
- ...
- <env>[NAME=VALUE]</env>
- ...
- </streaming>
- <!-- Either streaming or pipes can be specified for an action, not both -->
- <pipes>
- <map>[MAPPER]</map>
- <reduce>[REDUCER]</reducer>
- <inputformat>[INPUTFORMAT]</inputformat>
- <partitioner>[PARTITIONER]</partitioner>
- <writer>[OUTPUTFORMAT]</writer>
- <program>[EXECUTABLE]</program>
- </pipes>
- <job-xml>[JOB-XML-FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <config-class>com.example.MyConfigClass</config-class>
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- </map-reduce>
-
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete before starting the job. This should be used
-exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation
-will be performed in the `fs.default.name` filesystem for hdfs URIs. The format for specifying hcatalog table URI is
-hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
-`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
-In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
-need to be placed in workflow lib directory or specified using `archive` tag.
-
-The `job-xml` element, if present, must refer to a Hadoop JobConf `job.xml` file bundled in the workflow application.
-By default the `job.xml` file is taken from the workflow application namenode, regardless the namenode specified for the action.
-To specify a `job.xml` on another namenode use a fully qualified file path.
-The `job-xml` element is optional and as of schema 0.4, multiple `job-xml` elements are allowed in order to specify multiple Hadoop JobConf `job.xml` files.
-
-The `configuration` element, if present, contains JobConf properties for the Hadoop job.
-
-Properties specified in the `configuration` element override properties specified in the file specified in the
- `job-xml` element.
-
-As of schema 0.5, the `config-class` element, if present, contains a class that implements OozieActionConfigurator that can be used
-to further configure the MapReduce job.
-
-Properties specified in the `config-class` class override properties specified in `configuration` element.
-
-External Stats can be turned on/off by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in the configuration element of workflow.xml. The default value for this property is _false_.
-
-The `file` element, if present, must specify the target symbolic link for binaries by separating the original file and target with a # (file#target-sym-link). This is not required for libraries.
-
-The `mapper` and `reducer` process for streaming jobs, should specify the executable command with URL encoding. e.g. '%' should be replaced by '%25'.
-
-**Example:**
-
-
-```
-<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="myfirstHadoopJob">
- <map-reduce>
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="hdfs://foo:8020/usr/tucu/output-data"/>
- </prepare>
- <job-xml>/myfirstjob.xml</job-xml>
- <configuration>
- <property>
- <name>mapred.input.dir</name>
- <value>/usr/tucu/input-data</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>/usr/tucu/input-data</value>
- </property>
- <property>
- <name>mapred.reduce.tasks</name>
- <value>${firstJobReducers}</value>
- </property>
- <property>
- <name>oozie.action.external.stats.write</name>
- <value>true</value>
- </property>
- </configuration>
- </map-reduce>
- <ok to="myNextAction"/>
- <error to="errorCleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-In the above example, the number of Reducers to be used by the Map/Reduce job has to be specified as a parameter of
-the workflow job configuration when creating the workflow job.
-
-**Streaming Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="firstjob">
- <map-reduce>
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${output}"/>
- </prepare>
- <streaming>
- <mapper>/bin/bash testarchive/bin/mapper.sh testfile</mapper>
- <reducer>/bin/bash testarchive/bin/reducer.sh</reducer>
- </streaming>
- <configuration>
- <property>
- <name>mapred.input.dir</name>
- <value>${input}</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>${output}</value>
- </property>
- <property>
- <name>stream.num.map.output.key.fields</name>
- <value>3</value>
- </property>
- </configuration>
- <file>/users/blabla/testfile.sh#testfile</file>
- <archive>/users/blabla/testarchive.jar#testarchive</archive>
- </map-reduce>
- <ok to="end"/>
- <error to="kill"/>
- </action>
- ...
-</workflow-app>
-```
-
-
-**Pipes Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="firstjob">
- <map-reduce>
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${output}"/>
- </prepare>
- <pipes>
- <program>bin/wordcount-simple#wordcount-simple</program>
- </pipes>
- <configuration>
- <property>
- <name>mapred.input.dir</name>
- <value>${input}</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>${output}</value>
- </property>
- </configuration>
- <archive>/users/blabla/testarchive.jar#testarchive</archive>
- </map-reduce>
- <ok to="end"/>
- <error to="kill"/>
- </action>
- ...
-</workflow-app>
-```
-
-
-<a name="PigAction"></a>
-#### 3.2.3 Pig Action
-
-The `pig` action starts a Pig job.
-
-The workflow job will wait until the pig job completes before continuing to the next action.
-
-The `pig` action has to be configured with the resource-manager, name-node, pig script and the necessary parameters and
-configuration to run the Pig job.
-
-A `pig` action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
-starting the Pig job. This capability enables Oozie to retry a Pig job in the situation of a transient failure (Pig
-creates temporary directories for intermediate data, thus a retry without cleanup would fail).
-
-Hadoop JobConf properties can be specified as part of
-
- * the `config-default.xml` or
- * JobConf XML file bundled with the workflow application or
- * \<global\> tag in workflow definition or
- * Inline `pig` action configuration.
-
-The configuration properties are loaded in the following above order i.e. `job-xml` and `configuration`, and
-the precedence order is later values override earlier values.
-
-Inline property values can be parameterized (templatized) using EL expressions.
-
-The YARN `yarn.resourcemanager.address` and HDFS `fs.default.name` properties must not be present in the job-xml and inline
-configuration.
-
-As with Hadoop map-reduce jobs, it is possible to add files and archives to be available to the Pig job, refer to
-section [#FilesArchives][Adding Files and Archives for the Job].
-
-
-**Syntax for Pig actions in Oozie schema 1.0:**
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <pig>
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[JOB-XML-FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <script>[PIG-SCRIPT]</script>
- <param>[PARAM-VALUE]</param>
- ...
- <param>[PARAM-VALUE]</param>
- <argument>[ARGUMENT-VALUE]</argument>
- ...
- <argument>[ARGUMENT-VALUE]</argument>
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- </pig>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-**Syntax for Pig actions in Oozie schema 0.2:**
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.2">
- ...
- <action name="[NODE-NAME]">
- <pig>
- <job-tracker>[JOB-TRACKER]</job-tracker>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[JOB-XML-FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <script>[PIG-SCRIPT]</script>
- <param>[PARAM-VALUE]</param>
- ...
- <param>[PARAM-VALUE]</param>
- <argument>[ARGUMENT-VALUE]</argument>
- ...
- <argument>[ARGUMENT-VALUE]</argument>
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- </pig>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-**Syntax for Pig actions in Oozie schema 0.1:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1">
- ...
- <action name="[NODE-NAME]">
- <pig>
- <job-tracker>[JOB-TRACKER]</job-tracker>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[JOB-XML-FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <script>[PIG-SCRIPT]</script>
- <param>[PARAM-VALUE]</param>
- ...
- <param>[PARAM-VALUE]</param>
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- </pig>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete before starting the job. This should be used
-exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation
-will be performed in the `fs.default.name` filesystem for hdfs URIs. The format for specifying hcatalog table URI is
-hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
-`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
-In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
-need to be placed in workflow lib directory or specified using `archive` tag.
-
-The `job-xml` element, if present, must refer to a Hadoop JobConf `job.xml` file bundled in the workflow application.
-The `job-xml` element is optional and as of schema 0.4, multiple `job-xml` elements are allowed in order to specify multiple Hadoop JobConf `job.xml` files.
-
-The `configuration` element, if present, contains JobConf properties for the underlying Hadoop jobs.
-
-Properties specified in the `configuration` element override properties specified in the file specified in the
- `job-xml` element.
-
-External Stats can be turned on/off by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in the configuration element of workflow.xml. The default value for this property is _false_.
-
-The inline and job-xml configuration properties are passed to the Hadoop jobs submitted by Pig runtime.
-
-The `script` element contains the pig script to execute. The pig script can be templatized with variables of the
-form `${VARIABLE}`. The values of these variables can then be specified using the `params` element.
-
-NOTE: Oozie will perform the parameter substitution before firing the pig job. This is different from the
-[parameter substitution mechanism provided by Pig](http://wiki.apache.org/pig/ParameterSubstitution), which has a
-few limitations.
-
-The `params` element, if present, contains parameters to be passed to the pig script.
-
-**In Oozie schema 0.2:**
-The `arguments` element, if present, contains arguments to be passed to the pig script.
-
-
-All the above elements can be parameterized (templatized) using EL expressions.
-
-**Example for Oozie schema 0.2:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.2">
- ...
- <action name="myfirstpigjob">
- <pig>
- <job-tracker>foo:8021</job-tracker>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- <property>
- <name>oozie.action.external.stats.write</name>
- <value>true</value>
- </property>
- </configuration>
- <script>/mypigscript.pig</script>
- <argument>-param</argument>
- <argument>INPUT=${inputDir}</argument>
- <argument>-param</argument>
- <argument>OUTPUT=${outputDir}/pig-output3</argument>
- </pig>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-
-**Example for Oozie schema 0.1:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
- ...
- <action name="myfirstpigjob">
- <pig>
- <job-tracker>foo:8021</job-tracker>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- </configuration>
- <script>/mypigscript.pig</script>
- <param>InputDir=/home/tucu/input-data</param>
- <param>OutputDir=${jobOutput}</param>
- </pig>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-<a name="FsAction"></a>
-#### 3.2.4 Fs (HDFS) action
-
-The `fs` action allows to manipulate files and directories in HDFS from a workflow application. The supported commands
-are `move`, `delete`, `mkdir`, `chmod`, `touchz`, `setrep` and `chgrp`.
-
-The FS commands are executed synchronously from within the FS action, the workflow job will wait until the specified
-file commands are completed before continuing to the next action.
-
-Path names specified in the `fs` action can be parameterized (templatized) using EL expressions.
-Path name should be specified as a absolute path. In case of `move`, `delete`, `chmod` and `chgrp` commands, a glob pattern can also be specified instead of an absolute path.
-For `move`, glob pattern can only be specified for source path and not the target.
-
-Each file path must specify the file system URI, for move operations, the target must not specify the system URI.
-
-**IMPORTANT:** For the purposes of copying files within a cluster it is recommended to refer to the `distcp` action
-instead. Refer to [`distcp`](DG_DistCpActionExtension.html) action to copy files within a cluster.
-
-**IMPORTANT:** All the commands within `fs` action do not happen atomically, if a `fs` action fails half way in the
-commands being executed, successfully executed commands are not rolled back. The `fs` action, before executing any
-command must check that source paths exist and target paths don't exist (constraint regarding target relaxed for the `move` action. See below for details), thus failing before executing any command.
-Therefore the validity of all paths specified in one `fs` action are evaluated before any of the file operation are
-executed. Thus there is less chance of an error occurring while the `fs` action executes.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <fs>
- <delete path='[PATH]' skip-trash='[true/false]'/>
- ...
- <mkdir path='[PATH]'/>
- ...
- <move source='[SOURCE-PATH]' target='[TARGET-PATH]'/>
- ...
- <chmod path='[PATH]' permissions='[PERMISSIONS]' dir-files='false' />
- ...
- <touchz path='[PATH]' />
- ...
- <chgrp path='[PATH]' group='[GROUP]' dir-files='false' />
- ...
- <setrep path='[PATH]' replication-factor='2'/>
- </fs>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `delete` command deletes the specified path, if it is a directory it deletes recursively all its content and then
-deletes the directory. By default it does skip trash. It can be moved to trash by setting the value of skip-trash to
-'false'. It can also be used to drop hcat tables/partitions. This is the only FS command which supports HCatalog URIs as well.
-For eg:
-
-```
-<delete path='hcat://[metastore server]:[port]/[database name]/[table name]'/>
-OR
-<delete path='hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...'/>
-```
-
-The `mkdir` command creates the specified directory, it creates all missing directories in the path. If the directory
-already exist it does a no-op.
-
-In the `move` command the `source` path must exist. The following scenarios are addressed for a `move`:
-
- * The file system URI(e.g. `hdfs://{nameNode}`) can be skipped in the `target` path. It is understood to be the same as that of the source. But if the target path does contain the system URI, it cannot be different than that of the source.
- * The parent directory of the `target` path must exist
- * For the `target` path, if it is a file, then it must not already exist.
- * However, if the `target` path is an already existing directory, the `move` action will place your `source` as a child of the `target` directory.
-
-The `chmod` command changes the permissions for the specified path. Permissions can be specified using the Unix Symbolic
-representation (e.g. -rwxrw-rw-) or an octal representation (755).
-When doing a `chmod` command on a directory, by default the command is applied to the directory and the files one level
-within the directory. To apply the `chmod` command to the directory, without affecting the files within it,
-the `dir-files` attribute must be set to `false`. To apply the `chmod` command
-recursively to all levels within a directory, put a `recursive` element inside the \<chmod\> element.
-
-The `touchz` command creates a zero length file in the specified path if none exists. If one already exists, then touchz will perform a touch operation.
-Touchz works only for absolute paths.
-
-The `chgrp` command changes the group for the specified path.
-When doing a `chgrp` command on a directory, by default the command is applied to the directory and the files one level
-within the directory. To apply the `chgrp` command to the directory, without affecting the files within it,
-the `dir-files` attribute must be set to `false`.
-To apply the `chgrp` command recursively to all levels within a directory, put a `recursive` element inside the \<chgrp\> element.
-
-The `setrep` command changes replication factor of an hdfs file(s). Changing RF of directories or symlinks is not
-supported; this action requires an argument for RF.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="hdfscommands">
- <fs>
- <delete path='hdfs://foo:8020/usr/tucu/temp-data'/>
- <mkdir path='archives/${wf:id()}'/>
- <move source='${jobInput}' target='archives/${wf:id()}/processed-input'/>
- <chmod path='${jobOutput}' permissions='-rwxrw-rw-' dir-files='true'><recursive/></chmod>
- <chgrp path='${jobOutput}' group='testgroup' dir-files='true'><recursive/></chgrp>
- <setrep path='archives/${wf:id()/filename(s)}' replication-factor='2'/>
- </fs>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-In the above example, a directory named after the workflow job ID is created and the input of the job, passed as
-workflow configuration parameter, is archived under the previously created directory.
-
-As of schema 0.4, if a `name-node` element is specified, then it is not necessary for any of the paths to start with the file system
-URI as it is taken from the `name-node` element. This is also true if the name-node is specified in the global section
-(see [Global Configurations](WorkflowFunctionalSpec.html#GlobalConfigurations))
-
-As of schema 0.4, zero or more `job-xml` elements can be specified; these must refer to Hadoop JobConf `job.xml` formatted files
-bundled in the workflow application. They can be used to set additional properties for the FileSystem instance.
-
-As of schema 0.4, if a `configuration` element is specified, then it will also be used to set additional JobConf properties for the
-FileSystem instance. Properties specified in the `configuration` element override properties specified in the files specified
-by any `job-xml` elements.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.4">
- ...
- <action name="hdfscommands">
- <fs>
- <name-node>hdfs://foo:8020</name-node>
- <job-xml>fs-info.xml</job-xml>
- <configuration>
- <property>
- <name>some.property</name>
- <value>some.value</value>
- </property>
- </configuration>
- <delete path='/usr/tucu/temp-data'/>
- </fs>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-<a name="SubWorkflowAction"></a>
-#### 3.2.5 Sub-workflow Action
-
-The `sub-workflow` action runs a child workflow job, the child workflow job can be in the same Oozie system or in
-another Oozie system.
-
-The parent workflow job will wait until the child workflow job has completed.
-
-There can be several sub-workflows defined within a single workflow, each under its own action element.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <sub-workflow>
- <app-path>[WF-APPLICATION-PATH]</app-path>
- <propagate-configuration/>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- </sub-workflow>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The child workflow job runs in the same Oozie system instance where the parent workflow job is running.
-
-The `app-path` element specifies the path to the workflow application of the child workflow job.
-
-The `propagate-configuration` flag, if present, indicates that the workflow job configuration should be propagated to
-the child workflow.
-
-The `configuration` section can be used to specify the job properties that are required to run the child workflow job.
-
-The configuration of the `sub-workflow` action can be parameterized (templatized) using EL expressions.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="a">
- <sub-workflow>
- <app-path>child-wf</app-path>
- <configuration>
- <property>
- <name>input.dir</name>
- <value>${wf:id()}/second-mr-output</value>
- </property>
- </configuration>
- </sub-workflow>
- <ok to="end"/>
- <error to="kill"/>
- </action>
- ...
-</workflow-app>
-```
-
-In the above example, the workflow definition with the name `child-wf` will be run on the Oozie instance at
- `.http://myhost:11000/oozie`. The specified workflow application must be already deployed on the target Oozie instance.
-
-A configuration parameter `input.dir` is being passed as job property to the child workflow job.
-
-The subworkflow can inherit the lib jars from the parent workflow by setting `oozie.subworkflow.classpath.inheritance` to true
-in oozie-site.xml or on a per-job basis by setting `oozie.wf.subworkflow.classpath.inheritance` to true in a job.properties file.
-If both are specified, `oozie.wf.subworkflow.classpath.inheritance` has priority. If the subworkflow and the parent have
-conflicting jars, the subworkflow's jar has priority. By default, `oozie.wf.subworkflow.classpath.inheritance` is set to false.
-
-To prevent errant workflows from starting infinitely recursive subworkflows, `oozie.action.subworkflow.max.depth` can be specified
-in oozie-site.xml to set the maximum depth of subworkflow calls. For example, if set to 3, then a workflow can start subwf1, which
-can start subwf2, which can start subwf3; but if subwf3 tries to start subwf4, then the action will fail. The default is 50.
-
-<a name="JavaAction"></a>
-#### 3.2.6 Java Action
-
-The `java` action will execute the `public static void main(String[] args)` method of the specified main Java class.
-
-Java applications are executed in the Hadoop cluster as map-reduce job with a single Mapper task.
-
-The workflow job will wait until the java application completes its execution before continuing to the next action.
-
-The `java` action has to be configured with the resource-manager, name-node, main Java class, JVM options and arguments.
-
-To indicate an `ok` action transition, the main Java class must complete gracefully the `main` method invocation.
-
-To indicate an `error` action transition, the main Java class must throw an exception.
-
-The main Java class can call `System.exit(int n)`. Exit code zero is regarded as OK, while non-zero exit codes will
-cause the `java` action to do an `error` transition and exit.
-
-A `java` action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
-starting the Java application. This capability enables Oozie to retry a Java application in the situation of a transient
-or non-transient failure (This can be used to cleanup any temporary data which may have been created by the Java
-application in case of failure).
-
-A `java` action can create a Hadoop configuration for interacting with a cluster (e.g. launching a map-reduce job).
-Oozie prepares a Hadoop configuration file which includes the environments site configuration files (e.g. hdfs-site.xml,
-mapred-site.xml, etc) plus the properties added to the `<configuration>` section of the `java` action. The Hadoop configuration
-file is made available as a local file to the Java application in its running directory. It can be added to the `java` actions
-Hadoop configuration by referencing the system property: `oozie.action.conf.xml`. For example:
-
-
-```
-// loading action conf prepared by Oozie
-Configuration actionConf = new Configuration(false);
-actionConf.addResource(new Path("file:///", System.getProperty("oozie.action.conf.xml")));
-```
-
-If `oozie.action.conf.xml` is not added then the job will pick up the mapred-default properties and this may result
-in unexpected behaviour. For repeated configuration properties later values override earlier ones.
-
-Inline property values can be parameterized (templatized) using EL expressions.
-
-The YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties must not be present
-in the `job-xml` and in the inline configuration.
-
-As with `map-reduce` and `pig` actions, it is possible to add files and archives to be available to the Java
-application. Refer to section [#FilesArchives][Adding Files and Archives for the Job].
-
-The `capture-output` element can be used to propagate values back into Oozie context, which can then be accessed via
-EL-functions. This needs to be written out as a java properties format file. The filename is obtained via a System
-property specified by the constant `oozie.action.output.properties`
-
-**IMPORTANT:** In order for a Java action to succeed on a secure cluster, it must propagate the Hadoop delegation token like in the
-following code snippet (this is benign on non-secure clusters):
-
-```
-// propagate delegation related props from launcher job to MR job
-if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
- jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
-}
-```
-
-**IMPORTANT:** Because the Java application is run from within a Map-Reduce job, from Hadoop 0.20. onwards a queue must
-be assigned to it. The queue name must be specified as a configuration property.
-
-**IMPORTANT:** The Java application from a Java action is executed in a single map task. If the task is abnormally terminated,
-such as due to a TaskTracker restart (e.g. during cluster maintenance), the task will be retried via the normal Hadoop task
-retry mechanism. To avoid workflow failure, the application should be written in a fashion that is resilient to such retries,
-for example by detecting and deleting incomplete outputs or picking back up from complete outputs. Furthermore, if a Java action
-spawns asynchronous activity outside the JVM of the action itself (such as by launching additional MapReduce jobs), the
-application must consider the possibility of collisions with activity spawned by the new instance.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <java>
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[JOB-XML]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <main-class>[MAIN-CLASS]</main-class>
- <java-opts>[JAVA-STARTUP-OPTS]</java-opts>
- <arg>ARGUMENT</arg>
- ...
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- <capture-output />
- </java>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete before starting the Java application. This should
-be used exclusively for directory cleanup or dropping of hcatalog table or table partitions for the Java application to be executed.
-In case of `delete`, a glob pattern can be used to specify path.
-The format for specifying hcatalog table URI is
-hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
-`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
-In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
-need to be placed in workflow lib directory or specified using `archive` tag.
-
-The `java-opts` and `java-opt` elements, if present, contains the command line parameters which are to be used to start the JVM that
-will execute the Java application. Using this element is equivalent to using the `mapred.child.java.opts`
-or `mapreduce.map.java.opts` configuration properties, with the advantage that Oozie will append to these properties instead of
-simply setting them (e.g. if you have one of these properties specified in mapred-site.xml, setting it again in
-the `configuration` element will override it, but using `java-opts` or `java-opt` will instead append to it, preserving the original
-value). You can have either one `java-opts`, multiple `java-opt`, or neither; you cannot use both at the same time. In any case,
-Oozie will set both `mapred.child.java.opts` and `mapreduce.map.java.opts` to the same value based on combining them. In other
-words, after Oozie is finished:
-
-```
-mapred.child.java.opts <-- "<mapred.child.java.opts> <mapreduce.map.java.opts> <java-opt...|java-opts>"
-mapreduce.map.java.opts <-- "<mapred.child.java.opts> <mapreduce.map.java.opts> <java-opt...|java-opts>"
-```
-In the case that parameters are repeated, the latest instance of the parameter is used by Java. This means that `java-opt`
-(or `java-opts`) has the highest priority, followed by `mapreduce.map.java.opts`, and finally `mapred.child.java.opts`. When
-multiple `java-opt` are specified, they are included from top to bottom ordering, where the bottom has highest priority.
-
-The `arg` elements, if present, contains arguments for the main function. The value of each `arg` element is considered
-a single argument and they are passed to the `main` method in the same order.
-
-All the above elements can be parameterized (templatized) using EL expressions.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="myfirstjavajob">
- <java>
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.queue.name</name>
- <value>default</value>
- </property>
- </configuration>
- <main-class>org.apache.oozie.MyFirstMainClass</main-class>
- <java-opts>-Dblah</java-opts>
- <arg>argument1</arg>
- <arg>argument2</arg>
- </java>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-##### 3.2.6.1 Overriding an action's Main class
-
-This feature is useful for developers to change the Main classes without having to recompile or redeploy Oozie.
-
-For most actions (not just the Java action), you can override the Main class it uses by specifying the following `configuration`
-property and making sure that your class is included in the workflow's classpath. If you specify this in the Java action,
-the main-class element has priority.
-
-
-```
-<property>
- <name>oozie.launcher.action.main.class</name>
- <value>org.my.CustomMain</value>
-</property>
-```
-
-**Note:** Most actions typically pass information to their corresponding Main in specific ways; you should look at the action's
-existing Main to see how it works before creating your own. In fact, its probably simplest to just subclass the existing Main and
-add/modify/overwrite any behavior you want to change.
-
-<a name="GitAction"></a>
-#### 3.2.7 Git action
-
-The `git` action allows one to clone a Git repository into HDFS. The supported options are `git-uri`, `branch`, `key-path`
-and `destination-uri`.
-
-The `git clone` action is executed asynchronously by one of the YARN containers assigned to run on the cluster. If an SSH key is
-specified it will be created on the file system in a YARN container's local directory, relying on YARN NodeManager to remove the
-file after the action has run.
-
-Path names specified in the `git` action should be able to be parameterized (templatized) using EL expressions,
-e.g. `${wf:user()}` . Path name should be specified as an absolute path. Each file path must specify the file system URI.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <git>
- <git-uri>[SOURCE-URI]</git-uri>
- ...
- <branch>[BRANCH]</branch>
- ...
- <key-path>[HDFS-PATH]</key-path>
- ...
- <destination-uri>[HDFS-PATH]</destination-uri>
- </git>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
- ...
- <action name="clone_oozie">
- <git>
- <git-uri>https://github.com/apache/oozie</git-uri>
- <destination-uri>hdfs://my_git_repo_directory</destination-uri>
- </git>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-In the above example, a Git repository on e.g. GitHub.com is cloned to the HDFS directory `my_git_repo_directory` which should not
-exist previously on the filesystem. Note that repository addresses outside of GitHub.com but accessible to the YARN container
-running the Git action may also be used.
-
-If a `name-node` element is specified, then it is not necessary for any of the paths to start with the file system URI as it is
-taken from the `name-node` element.
-
-The `resource-manager` (Oozie 5.x) element has to be specified to name the YARN ResourceManager address.
-
-If any of the paths need to be served from another HDFS namenode, its address has to be part of
-that filesystem URI prefix:
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <git>
- ...
- <name-node>hdfs://name-node.first.company.com:8020</name-node>
- ...
- <key-path>hdfs://name-node.second.company.com:8020/[HDFS-PATH]</key-path>
- ...
- </git>
- ...
- </action>
- ...
-</workflow-app>
-```
-
-This is also true if the name-node is specified in the global section (see
-[Global Configurations](WorkflowFunctionalSpec.html#GlobalConfigurations)).
-
-Be aware that `key-path` might point to a secure object store location other than the current `fs.defaultFS`. In that case,
-appropriate file permissions are still necessary (readable by submitting user), credentials provided, etc.
-
-As of workflow schema 1.0, zero or more `job-xml` elements can be specified; these must refer to Hadoop JobConf `job.xml` formatted
-files bundled in the workflow application. They can be used to set additional properties for the `FileSystem` instance.
-
-As of schema workflow schema 1.0, if a `configuration` element is specified, then it will also be used to set additional `JobConf`
-properties for the `FileSystem` instance. Properties specified in the `configuration` element are overridden by properties
-specified in the files specified by any `job-xml` elements.
-
-**Example:**
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <git>
- ...
- <name-node>hdfs://foo:8020</name-node>
- <job-xml>fs-info.xml</job-xml>
- <configuration>
- <property>
- <name>some.property</name>
- <value>some.value</value>
- </property>
- </configuration>
- </git>
- ...
- </action>
- ...
-</workflow>
-```
-
-<a name="WorkflowParameterization"></a>
-## 4 Parameterization of Workflows
-
-Workflow definitions can be parameterized.
-
-When workflow node is executed by Oozie all the ELs are resolved into concrete values.
-
-The parameterization of workflow definitions it done using JSP Expression Language syntax from the
-[JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to
-support variables as parameters but also functions and complex expressions.
-
-EL expressions can be used in the configuration values of action and decision nodes. They can be used in XML attribute
-values and in XML element and attribute values.
-
-They cannot be used in XML element and attribute names. They cannot be used in the name of a node and they cannot be
-used within the `transition` elements of a node.
-
-<a name="WorkflowProperties"></a>
-### 4.1 Workflow Job Properties (or Parameters)
-
-When a workflow job is submitted to Oozie, the submitter may specify as many workflow job properties as required
-(similar to Hadoop JobConf properties).
-
-Workflow applications may define default values for the workflow job or action parameters. They must be defined in a
- `config-default.xml` file bundled with the workflow application archive (refer to section '7 Workflow
- Applications Packaging'). Job or action properties specified in the workflow definition have precedence over the default values.
-
-Properties that are a valid Java identifier, `[A-Za-z_][0-9A-Za-z_]*`, are available as '${NAME}'
-variables within the workflow definition.
-
-Properties that are not valid Java Identifier, for example 'job.tracker', are available via the `String
-wf:conf(String name)` function. Valid identifier properties are available via this function as well.
-
-Using properties that are valid Java identifiers result in a more readable and compact definition.
-
-By using properties
-**Example:**
-
-Parameterized Workflow definition:
-
-
-```
-<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:1.0">
- ...
- <action name='firstjob'>
- <map-reduce>
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode}</name-node>
- <configuration>
- <property>
- <name>mapred.mapper.class</name>
- <value>com.foo.FirstMapper</value>
- </property>
- <property>
- <name>mapred.reducer.class</name>
- <value>com.foo.FirstReducer</value>
- </property>
- <property>
- <name>mapred.input.dir</name>
- <value>${inputDir}</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>${outputDir}</value>
- </property>
- </configuration>
- </map-reduce>
- <ok to='secondjob'/>
- <error to='killcleanup'/>
- </action>
- ...
-</workflow-app>
-```
-
-When submitting a workflow job for the workflow definition above, 3 workflow job properties must be specified:
-
- * `resourceManager:`
- * `inputDir:`
- * `outputDir:`
-
-If the above 3 properties are not specified, the job will fail.
-
-As of schema 0.4, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
-properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
-
-**Example:**
-
-The previous parameterized workflow definition with formal parameters:
-
-
-```
-<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:1.0">
- <parameters>
- <property>
- <name>inputDir</name>
- </property>
- <property>
- <name>outputDir</name>
- <value>out-dir</value>
- </property>
- </parameters>
- ...
- <action name='firstjob'>
- <map-reduce>
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode}</name-node>
- <configuration>
- <property>
- <name>mapred.mapper.class</name>
- <value>com.foo.FirstMapper</value>
- </property>
- <property>
- <name>mapred.reducer.class</name>
- <value>com.foo.FirstReducer</value>
- </property>
- <property>
- <name>mapred.input.dir</name>
- <value>${inputDir}</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>${outputDir}</value>
- </property>
- </configuration>
- </map-reduce>
- <ok to='secondjob'/>
- <error to='killcleanup'/>
- </action>
- ...
-</workflow-app>
-```
-
-In the above example, if `inputDir` is not specified, Oozie will print an error message instead of submitting the job. If
-`outputDir` is not specified, Oozie will use the default value, `out-dir`.
-
-<a name="WorkflowELSupport"></a>
-### 4.2 Expression Language Functions
-
-Oozie, besides allowing the use of workflow job properties to parameterize workflow jobs, it provides a set of build
-in EL functions that enable a more complex parameterization of workflow action nodes as well as the predicates in
-decision nodes.
-
-#### 4.2.1 Basic EL Constants
-
- * **KB:** 1024, one kilobyte.
- * **MB:** 1024 *** KB, one megabyte.
- * **GB:** 1024 *** MB, one gigabyte.
- * **TB:** 1024 *** GB, one terabyte.
- * **PB:** 1024 *** TG, one petabyte.
-
-All the above constants are of type `long`.
-
-#### 4.2.2 Basic EL Functions
-
-**String firstNotNull(String value1, String value2)**
-
-It returns the first not `null` value, or `null` if both are `null`.
-
-Note that if the output of this function is `null` and it is used as string, the EL library converts it to an empty
-string. This is the common behavior when using `firstNotNull()` in node configuration sections.
-
-**String concat(String s1, String s2)**
-
-It returns the concatenation of 2 strings. A string with `null` value is considered as an empty string.
-
-**String replaceAll(String src, String regex, String replacement)**
-
-Replace each occurrence of regular expression match in the first string with the `replacement` string and return the replaced string. A 'regex' string with `null` value is considered as no change. A 'replacement' string with `null` value is consider as an empty string.
-
-**String appendAll(String src, String append, String delimeter)**
-
- Add the `append` string into each splitted sub-strings of the first string(`src`). The split is performed into `src` string using the `delimiter`. E.g. `appendAll("/a/b/,/c/b/,/c/d/", "ADD", ",")` will return `/a/b/ADD,/c/b/ADD,/c/d/ADD`. A `append` string with `null` value is consider as an empty string. A `delimiter` string with value `null` is considered as no append in the string.
-
-**String trim(String s)**
-
-It returns the trimmed value of the given string. A string with `null` value is considered as an empty string.
-
-**String urlEncode(String s)**
-
-It returns the URL UTF-8 encoded value of the given string. A string with `null` value is considered as an empty string.
-
-**String timestamp()**
-
-It returns the current datetime in ISO8601 format, down to minutes (yyyy-MM-ddTHH:mmZ), in the Oozie's processing timezone,
-i.e. 1997-07-16T19:20Z
-
-**String toJsonStr(Map<String, String>)** (since Oozie 3.3)
-
-It returns an XML encoded JSON representation of a Map<String, String>. This function is useful to encode as
-a single property the complete action-data of an action, **wf:actionData(String actionName)**, in order to pass
-it in full to another action.
-
-**String toPropertiesStr(Map<String, String>)** (since Oozie 3.3)
-
-It returns an XML encoded Properties representation of a Map<String, String>. This function is useful to encode as
-a single property the complete action-data of an action, **wf:actionData(String actionName)**, in order to pass
-it in full to another action.
-
-**String toConfigurationStr(Map<String, String>)** (since Oozie 3.3)
-
-It returns an XML encoded Configuration representation of a Map<String, String>. This function is useful to encode as
-a single property the complete action-data of an action, **wf:actionData(String actionName)**, in order to pass
-it in full to another action.
-
-#### 4.2.3 Workflow EL Functions
-
-**String wf:id()**
-
-It returns the workflow job ID for the current workflow job.
-
-**String wf:name()**
-
-It returns the workflow application name for the current workflow job.
-
-**String wf:appPath()**
-
-It returns the workflow application path for the current workflow job.
-
-**String wf:conf(String name)**
-
-It returns the value of the workflow job configuration property for the current workflow job, or an empty string if
-undefined.
-
-**String wf:user()**
-
-It returns the user name that started the current workflow job.
-
-**String wf:group()**
-
-It returns the group/ACL for the current workflow job.
-
-**String wf:callback(String stateVar)**
-
-It returns the callback URL for the current workflow action node, `stateVar` can be a valid exit state (`OK` or
- `ERROR`) for the action or a token to be replaced with the exit state by the remote system executing the task.
-
-**String wf:transition(String node)**
-
-It returns the transition taken by the specified workflow action node, or an empty string if the action has not being
-executed or it has not completed yet.
-
-**String wf:lastErrorNode()**
-
-It returns the name of the last workflow action node that exit with an `ERROR` exit state, or an empty string if no
-action has exited with `ERROR` state in the current workflow job.
-
-**String wf:errorCode(String node)**
-
-It returns the error code for the specified action node, or an empty string if the action node has not exited
-with `ERROR` state.
-
-Each type of action node must define its complete error code list.
-
-**String wf:errorMessage(String message)**
-
-It returns the error message for the specified action node, or an empty string if no action node has not exited
-with `ERROR` state.
-
-The error message can be useful for debugging and notification purposes.
-
-**int wf:run()**
-
-It returns the run number for the current workflow job, normally `0` unless the workflow job is re-run, in which case
-indicates the current run.
-
-**Map<String, String> wf:actionData(String node)**
-
-This function is only applicable to action nodes that produce output data on completion.
-
-The output data is in a Java Properties format and via this EL function it is available as a `Map<String, String>`.
-
-**String wf:actionExternalId(String node)**
-
-It returns the external Id for an action node, or an empty string if the action has not being executed or it has not
-completed yet.
-
-**String wf:actionTrackerUri(String node)**
-
-It returns the tracker URI for an action node, or an empty string if the action has not being executed or it has not
-completed yet.
-
-**String wf:actionExternalStatus(String node)**
-
-It returns the external status for an action node, or an empty string if the action has not being executed or it has
-not completed yet.
-
-#### 4.2.4 Hadoop EL Constants
-
- * **RECORDS:** Hadoop record counters group name.
- * **MAP_IN:** Hadoop mapper input records counter name.
- * **MAP_OUT:** Hadoop mapper output records counter name.
- * **REDUCE_IN:** Hadoop reducer input records counter name.
- * **REDUCE_OUT:** Hadoop reducer input record counter name.
- * **GROUPS:** 1024 *** Hadoop mapper/reducer record groups counter name.
-
-#### 4.2.5 Hadoop EL Functions
-
-<a name="HadoopCountersEL"></a>
-**Map < String, Map < String, Long > > hadoop:counters(String node)**
-
-It returns the counters for a job submitted by a Hadoop action node. It returns `0` if the if the Hadoop job has not
-started yet and for undefined counters.
-
-The outer Map key is a counter group name. The inner Map value is a Map of counter names and counter values.
-
-The Hadoop EL constants defined in the previous section provide access to the Hadoop built in record counters.
-
-This function can also be used to access specific action statistics information. Examples of action stats and their usage through EL Functions (referenced in workflow xml) are given below.
-
-**Example of MR action stats:**
-
-```
-{
- "ACTION_TYPE": "MAP_REDUCE",
- "org.apache.hadoop.mapred.JobInProgress$Counter": {
- "TOTAL_LAUNCHED_REDUCES": 1,
- "TOTAL_LAUNCHED_MAPS": 1,
- "DATA_LOCAL_MAPS": 1
- },
- "FileSystemCounters": {
- "FILE_BYTES_READ": 1746,
- "HDFS_BYTES_READ": 1409,
- "FILE_BYTES_WRITTEN": 3524,
- "HDFS_BYTES_WRITTEN": 1547
- },
- "org.apache.hadoop.mapred.Task$Counter": {
- "REDUCE_INPUT_GROUPS": 33,
- "COMBINE_OUTPUT_RECORDS": 0,
- "MAP_INPUT_RECORDS": 33,
- "REDUCE_SHUFFLE_BYTES": 0,
- "REDUCE_OUTPUT_RECORDS": 33,
- "SPILLED_RECORDS": 66,
- "MAP_OUTPUT_BYTES": 1674,
- "MAP_INPUT_BYTES": 1409,
- "MAP_OUTPUT_RECORDS": 33,
- "COMBINE_INPUT_RECORDS": 0,
- "REDUCE_INPUT_RECORDS": 33
- }
-}
-```
-
-Below is the workflow that describes how to access specific information using hadoop:counters() EL function from the MR stats.
-**Workflow xml:**
-
-```
-<workflow-app xmlns="uri:oozie:workflow:1.0" name="map-reduce-wf">
- <start to="mr-node"/>
- <action name="mr-node">
- <map-reduce>
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode}</name-node>
- <prepare>
- <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.job.queue.name</name>
- <value>${queueName}</value>
- </property>
- <property>
- <name>mapred.mapper.class</name>
- <value>org.apache.oozie.example.SampleMapper</value>
- </property>
- <property>
- <name>mapred.reducer.class</name>
- <value>org.apache.oozie.example.SampleReducer</value>
- </property>
- <property>
- <name>mapred.map.tasks</name>
- <value>1</value>
- </property>
- <property>
- <name>mapred.input.dir</name>
- <value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
- </property>
- <property>
- <name>oozie.action.external.stats.write</name>
- <value>true</value>
- </property>
- </configuration>
- </map-reduce>
- <ok to="java1"/>
- <error to="fail"/>
- </action>
- <action name="java1">
- <java>
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode}</name-node>
- <configuration>
- <property>
- <name>mapred.job.queue.name</name>
- <value>${queueName}</value>
- </property>
- </configuration>
- <main-class>MyTest</main-class>
- <arg> ${hadoop:counters("mr-node")["FileSystemCounters"]["FILE_BYTES_READ"]}</arg>
- <capture-output/>
- </java>
- <ok to="end" />
- <error to="fail" />
- </action>
- <kill name="fail">
- <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
- </kill>
- <end name="end"
<TRUNCATED>
[11/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/AG_ActionConfiguration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_ActionConfiguration.twiki b/docs/src/site/twiki/AG_ActionConfiguration.twiki
deleted file mode 100644
index e020d1a..0000000
--- a/docs/src/site/twiki/AG_ActionConfiguration.twiki
+++ /dev/null
@@ -1,87 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Action Configuration
-
-Oozie supports providing default configuration for actions of a particular
-action type and default configuration for all actions
-
-## Hadoop Default Configuration Values
-
-Oozie supports action configuration equivalent to the component's `*-site.xml` and `*.properties` files.
-
-The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.action.configurations`
-and its value must follow the pattern _\[AUTHORITY=ACTION-CONF-DIR_,\]*. Where _AUTHORITY_ is the _HOST:PORT_ of
-the Hadoop service (JobTracker/ResourceManager or HDFS). The _ACTION-CONF-DIR_ is the action configuration directory. If the specified directory is a relative path, it will be looked under the Oozie configuration directory. An absolute path can
-also be specified. Oozie will load and process the action configuration files in the following order.
-
- 1. All files in _default_/*.xml (sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.
- 1. _default_.xml, if present.
- 1. All supported files in _actionname_/\*, e.g. _actionname_/\*.xml and _actionname_/*.properties (based on filename extension, sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.
- 1. _actionname_.xml, if present.
-
-
-For example, for _Hive_ action (which has the _actionname_ defined as _hive_ ), the list of files (under relevant _ACTION-CONF-DIR_ ) processed would be,
-
- 1. All files in _default_/*.xml, if present
- 1. _default_.xml, if present.
- 1. All files in _hive_/\*.xml and _hive_/\*.properties, if present
- 1. _hive_.xml, if present.
-
-
-Files processed earlier for an action have the lowest precedence and can have the configuration parameters redefined. All files and directories are relative to the _ACTION-CONF-DIR_ directory.
-
-In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard
-will be used as default if there is no action configuration for the requested Hadoop service.
-
-For example, the configuration in the `oozie-site.xml` would look like:
-
-
-```
-...
- <property>
- <name>oozie.service.HadoopAccessorService.action.configurations</name>
- <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value>
- </property>
-...
-```
-
-The action configuration files use the Hadoop configuration syntax.
-
-By default Oozie does not define any default action configurations.
-
-## Dependency deduplication
-
-Using Oozie with Hadoop 3 may require to have dependency file names distinguishable,
- which means having two files on sharelib and in your app's dependencies with identical names, leads to job submission failure.
-To avoid this you can enable the deduplicator by setting oozie.action.dependency.deduplicate=true in oozie-site.xml
-(false, by default).
-Dependencies which are closer to your application has higher priority: action jar > user workflow libs > action libs > system lib,
-where dependency with greater prio is used.
-
-Real world example:
-You have an application workflow which is uploaded to HDFS in /apps/app directory. You have your app.jar and dependency jars.
-You also define a spark action in your workflow and set use system libs; the HDFS tree is similar to this:
-
-```
- + /apps/app/
- - app.jar
- - workflow.xml
- + libs
- - app.jar
- - jackson-annotations-1.0.jar
- + share/lib/
- + spark
- - app.jar
- - jackson-annotations-1.0.jar
- + oozie
- - jackson-annotations-1.0.jar
-```
-The deduplicator code will create the following list of files:
-`/apps/app/app.jar,/apps/app/libs/jackson-annotations-1.0.jar`
-And no other files will be passed at job submission.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/AG_HadoopConfiguration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_HadoopConfiguration.twiki b/docs/src/site/twiki/AG_HadoopConfiguration.twiki
deleted file mode 100644
index ab71d7c..0000000
--- a/docs/src/site/twiki/AG_HadoopConfiguration.twiki
+++ /dev/null
@@ -1,94 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Hadoop Configuration
-
-## Hadoop Services Whitelisting
-
-Oozie supports whitelisting Hadoop services (JobTracker, HDFS), via 2 configuration properties:
-
-
-```
-...
- <property>
- <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
- <value> </value>
- <description>
- Whitelisted job tracker for Oozie service.
- </description>
- </property>
- <property>
- <name>oozie.service.HadoopAccessorService.nameNode.whitelist</name>
- <value> </value>
- <description>
- Whitelisted job tracker for Oozie service.
- </description>
- </property>
-...
-```
-
-The value must follow the pattern `[AUTHORITY,...]`. Where `AUTHORITY` is the `HOST:PORT` of
-the Hadoop service (JobTracker, HDFS).
-
-If the value is empty any HOST:PORT is accepted. Empty is the default value.
-
-## Hadoop Default Configuration Values
-
-Oozie supports Hadoop configuration equivalent to the Hadoop `*-site.xml` files.
-
-The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.hadoop.configurations`
-and its value must follow the pattern `[<AUTHORITY>=<HADOOP_CONF_DIR>,]*`. Where `<AUTHORITY>` is the `HOST:PORT` of
-the Hadoop service (JobTracker, HDFS). The `<HADOOP_CONF_DIR>` is a Hadoop configuration directory. If the specified
- directory is a relative path, it will be looked under the Oozie configuration directory. And absolute path can
- also be specified. Oozie will load the Hadoop `*-site.xml` files in the following order: core-site.xml, hdfs-site.xml,
- mapred-site.xml, yarn-site.xml, hadoop-site.xml, ssl-client.xml.
-
-In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard
-will be used as default if there is no configuration for the requested Hadoop service.
-
-For example, the configuration in the `oozie-site.xml` would look like:
-
-
-```
-...
- <property>
- <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
- <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value>
- </property>
-...
-```
-
-The Hadoop configuration files use the Hadoop configuration syntax.
-
-By default Oozie defines `*=hadoop-conf` and the default values of the `hadoop-site.xml` file are:
-
-
-```
-<configuration>
- <property>
- <name>mapreduce.jobtracker.kerberos.principal</name>
- <value>mapred/_HOST@LOCALREALM</value>
- </property>
- <property>
- <name>yarn.resourcemanager.principal</name>
- <value>yarn/_HOST@LOCALREALM</value>
- </property>
- <property>
- <name>dfs.namenode.kerberos.principal</name>
- <value>hdfs/_HOST@LOCALREALM</value>
- </property>
- <property>
- <name>mapreduce.framework.name</name>
- <value>yarn</value>
- </property>
-</configuration>
-```
-
-## Limitations
-
-All actions in a workflow application must interact with the same Hadoop JobTracker and NameNode.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/AG_Install.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_Install.twiki b/docs/src/site/twiki/AG_Install.twiki
deleted file mode 100644
index 270b98f..0000000
--- a/docs/src/site/twiki/AG_Install.twiki
+++ /dev/null
@@ -1,1447 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Oozie Installation and Configuration
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Basic Setup
-
-Follow the instructions at [Oozie Quick Start](DG_QuickStart.html).
-
-## Environment Setup
-
-**IMPORTANT:** Oozie ignores any set value for `OOZIE_HOME`, Oozie computes its home automatically.
-
-When running Oozie with its embedded Jetty server, the `conf/oozie-env.sh` file can be
-used to configure the following environment variables used by Oozie:
-
-**JETTY_OPTS** : settings for the Embedded Jetty that runs Oozie. Java System properties
-for Oozie should be specified in this variable. No default value.
-
-**OOZIE_CONFIG_FILE** : Oozie configuration file to load from Oozie configuration directory.
-Default value `oozie-site.xml`.
-
-**OOZIE_LOGS** : Oozie logs directory. Default value `logs/` directory in the Oozie installation
-directory.
-
-**OOZIE_LOG4J_FILE** : Oozie Log4J configuration file to load from Oozie configuration directory.
-Default value `oozie-log4j.properties`.
-
-**OOZIE_LOG4J_RELOAD** : Reload interval of the Log4J configuration file, in seconds.
-Default value `10`
-
-**OOZIE_CHECK_OWNER** : If set to `true`, Oozie setup/start/run/stop scripts will check that the
-owner of the Oozie installation directory matches the user invoking the script. The default
-value is undefined and interpreted as a `false`.
-
-**OOZIE_INSTANCE_ID** : The instance id of the Oozie server. When using HA, each server instance should have a unique instance id.
-Default value `${OOZIE_HTTP_HOSTNAME}`
-
-## Oozie Server Setup
-
-The `oozie-setup.sh` script prepares the embedded Jetty server to run Oozie.
-
-The `oozie-setup.sh` script options are:
-
-
-```
-Usage : oozie-setup.sh <Command and OPTIONS>
- sharelib create -fs FS_URI [-locallib SHARED_LIBRARY] [-extralib EXTRA_SHARED_LIBRARY] [-concurrency CONCURRENCY]
- (create sharelib for oozie,
- FS_URI is the fs.default.name
- for hdfs uri; SHARED_LIBRARY, path to the
- Oozie sharelib to install, it can be a tarball
- or an expanded version of it. If omitted,
- the Oozie sharelib tarball from the Oozie
- installation directory will be used.
- EXTRA_SHARED_LIBRARY represents extra sharelib resources.
- This option requires a pair of sharelibname
- and comma-separated list of pathnames in the following format:
- sharelib-name=path-name-1,path-name-2
- In case of more than one sharelib, this option can be specified
- multiple times.
- CONCURRENCY is a number of threads to be used
- for copy operations.
- By default 1 thread will be used)
- (action fails if sharelib is already installed
- in HDFS)
- sharelib upgrade -fs FS_URI [-locallib SHARED_LIBRARY] ([deprecated][use create command to create new version]
- upgrade existing sharelib, fails if there
- is no existing sharelib installed in HDFS)
- db create|upgrade|postupgrade -run [-sqlfile <FILE>] (create, upgrade or postupgrade oozie db with an
- optional sql File)
- export <file> exports the oozie database to the specified
- file in zip format
- import <file> imports the oozie database from the zip file
- created by export
- (without options prints this usage information)
-```
-
-If a directory `libext/` is present in Oozie installation directory, the `oozie-setup.sh` script will
-include all JARs in Jetty's `webapp/WEB_INF/lib/` directory.
-
-If the ExtJS ZIP file is present in the `libext/` directory, it will be added to the Jetty's `webapp/` directory as well.
-The ExtJS library file name be `ext-2.2.zip`.
-
-### Setting Up Oozie with an Alternate Tomcat
-
-Use the `addtowar.sh` script to prepare the Oozie server only if Oozie will run with a different
-servlet container than the embedded Jetty provided with the distribution.
-
-The `addtowar.sh` script adds Hadoop JARs, JDBC JARs and the ExtJS library to the Oozie WAR file.
-
-The `addtowar.sh` script options are:
-
-
-```
- Usage : addtowar <OPTIONS>
- Options: -inputwar INPUT_OOZIE_WAR
- -outputwar OUTPUT_OOZIE_WAR
- [-hadoop HADOOP_VERSION HADOOP_PATH]
- [-extjs EXTJS_PATH]
- [-jars JARS_PATH] (multiple JAR path separated by ':')
- [-secureWeb WEB_XML_PATH] (path to secure web.xml)
-```
-
-The original `oozie.war` file is in the Oozie server installation directory.
-
-After the Hadoop JARs and the ExtJS library has been added to the `oozie.war` file Oozie is ready to run.
-
-Delete any previous deployment of the `oozie.war` from the servlet container (if using Tomcat, delete
-`oozie.war` and `oozie` directory from Tomcat's `webapps/` directory)
-
-Deploy the prepared `oozie.war` file (the one that contains the Hadoop JARs and the ExtJS library) in the
-servlet container (if using Tomcat, copy the prepared `oozie.war` file to Tomcat's `webapps/` directory).
-
-**IMPORTANT:** Only one Oozie instance can be deployed per Tomcat instance.
-
-## Database Configuration
-
-Oozie works with HSQL, Derby, MySQL, Oracle, PostgreSQL or SQL Server databases.
-
-By default, Oozie is configured to use Embedded Derby.
-
-Oozie bundles the JDBC drivers for HSQL, Embedded Derby and PostgreSQL.
-
-HSQL is normally used for test cases as it is an in-memory database and all data is lost every time Oozie is stopped.
-
-If using Derby, MySQL, Oracle, PostgreSQL, or SQL Server, the Oozie database schema must be created using the `ooziedb.sh` command
-line tool.
-
-If using MySQL, Oracle, or SQL Server, the corresponding JDBC driver JAR file must be copied to Oozie's `libext/` directory and
-it must be added to Oozie WAR file using the `bin/addtowar.sh` or the `oozie-setup.sh` scripts using the `-jars` option.
-
-**IMPORTANT:** It is recommended to set the database's timezone to GMT (consult your database's documentation on how to do this).
-Databases don't handle Daylight Saving Time shifts correctly, and may cause problems if you run any Coordinators with actions
-scheduled to materialize during the 1 hour period where we "fall back". For Derby, you can add '-Duser.timezone=GMT'
-to `JETTY_OPTS` in oozie-env.sh to set this. Alternatively, if using MySQL, you can have Oozie use GMT with MySQL without
-setting MySQL's timezone to GMT by adding 'useLegacyDatetimeCode=false&serverTimezone=GMT' arguments to the JDBC
-URL, `oozie.service.JPAService.jdbc.url`. Be advised that changing the timezone on an existing Oozie database while Coordinators
-are already running may cause Coordinators to shift by the offset of their timezone from GMT once after making this change.
-
-The SQL database used by Oozie is configured using the following configuration properties (default values shown):
-
-
-```
- oozie.db.schema.name=oozie
- oozie.service.JPAService.create.db.schema=false
- oozie.service.JPAService.validate.db.connection=false
- oozie.service.JPAService.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
- oozie.service.JPAService.jdbc.url=jdbc:derby:${oozie.data.dir}/${oozie.db.schema.name}-db;create=true
- oozie.service.JPAService.jdbc.username=sa
- oozie.service.JPAService.jdbc.password=
- oozie.service.JPAService.pool.max.active.conn=10
-```
-
-**NOTE:** If the `oozie.db.schema.create` property is set to `true` (default value is `false`) the Oozie tables
-will be created automatically without having to use the `ooziedb` command line tool. Setting this property to
- `true` it is recommended only for development.
-
-**NOTE:** If the `oozie.db.schema.create` property is set to true, the `oozie.service.JPAService.validate.db.connection`
-property value is ignored and Oozie handles it as set to `false`.
-
-Once `oozie-site.xml` has been configured with the database configuration execute the `ooziedb.sh` command line tool to
-create the database:
-
-
-```
-$ bin/ooziedb.sh create -sqlfile oozie.sql -run
-
-Validate DB Connection.
-DONE
-Check DB schema does not exist
-DONE
-Check OOZIE_SYS table does not exist
-DONE
-Create SQL schema
-DONE
-DONE
-Create OOZIE_SYS table
-DONE
-
-Oozie DB has been created for Oozie version '3.2.0'
-
-The SQL commands have been written to: oozie.sql
-
-$
-```
-
-NOTE: If using MySQL, Oracle, or SQL Server, copy the corresponding JDBC driver JAR file to the `libext/` directory before running
-the `ooziedb.sh` command line tool.
-
-NOTE: If instead using the '-run' option, the `-sqlfile <FILE>` option is used, then all the
-database changes will be written to the specified file and the database won't be modified.
-
-If using HSQL there is no need to use the `ooziedb` command line tool as HSQL is an in-memory database. Use the
-following configuration properties in the oozie-site.xml:
-
-
-```
- oozie.db.schema.name=oozie
- oozie.service.JPAService.create.db.schema=true
- oozie.service.JPAService.validate.db.connection=false
- oozie.service.JPAService.jdbc.driver=org.hsqldb.jdbcDriver
- oozie.service.JPAService.jdbc.url=jdbc:hsqldb:mem:${oozie.db.schema.name}
- oozie.service.JPAService.jdbc.username=sa
- oozie.service.JPAService.jdbc.password=
- oozie.service.JPAService.pool.max.active.conn=10
-```
-
-If you are interested in fine tuning how Oozie can retry database operations on failing database connectivity or errors, you can
-set following properties to other values. Here are the default ones:
-
-
-```
- oozie.service.JPAService.retry.initial-wait-time.ms=100
- oozie.service.JPAService.retry.maximum-wait-time.ms=30000
- oozie.service.JPAService.retry.max-retries=10
-```
-
-If you set either `oozie.service.JPAService.retry.max-retries` or `oozie.service.JPAService.retry.maximum-wait-time.ms` to `0`,
-no retry attempts will be made on any database connectivity issues. Exact settings for these properties depend also on how much load
-is on Oozie regarding workflow and coordinator jobs.
-
-The database operation retry functionality kicks in when there is a `javax.persistence.PersistenceException` those root cause is not
-part of the normal everyday operation - filtered against a blacklist consisting of descendants like `NoSuchResultException`,
-`NonUniqueResultException`, and the like. This way Oozie won't retry database operations on errors that are more related to the
-current query, or otherwise part of the everyday life. This way it's ensured that this blacklist is database agnostic.
-
-It has been tested with a MySQL / failing every minute 10 seconds / an Oozie coordinator job of an Oozie workflow consisting of four
-workflow actions (some of them are asynchronous). On this setup Oozie was recovering after each and every database outage.
-
-To set up such a failing MySQL scenario following has to be performed:
-
- * Set `oozie.service.JPAService.connection.data.source` to `org.apache.oozie.util.db.BasicDataSourceWrapper`
- within `oozie-site.xml`
- * Set `oozie.service.JPAService.jdbc.driver` to `org.apache.oozie.util.db.FailingMySQLDriverWrapper` within `oozie-site.xml`
- * Restart Oozie server
- * Submit / start some workflows, coordinators etc.
- * See how Oozie is retrying on injected database errors by looking at the Oozie server logs, grepping `JPAException` instances
- with following message prefix: `Deliberately failing to prepare statement.`
-
-## Database Migration
-
-Oozie provides an easy way to switch between databases without losing any data. Oozie servers should be stopped during the
-database migration process.
-The export of the database can be done using the following command:
-
-```
-$ bin/oozie-setup.sh export /tmp/oozie_db.zip
-1 rows exported from OOZIE_SYS
-50 rows exported from WF_JOBS
-340 rows exported from WF_ACTIONS
-10 rows exported from COORD_JOBS
-70 rows exported from COORD_ACTIONS
-0 rows exported from BUNDLE_JOBS
-0 rows exported from BUNDLE_ACTIONS
-0 rows exported from SLA_REGISTRATION
-0 rows exported from SLA_SUMMARY
-```
-
-The database configuration is read from `oozie-site.xml`. After updating the configuration to point to the new database,
-the tables have to be created with ooziedb.sh in the [Database configuration](AG_Install.html#Database_Configuration)
-section above.
-Once the tables are created, they can be filled with data using the following command:
-
-
-```
-$ bin/oozie-setup.sh import /tmp/oozie_db.zip
-Loading to Oozie database version 3
-50 rows imported to WF_JOBS
-340 rows imported to WF_ACTIONS
-10 rows imported to COORD_JOBS
-70 rows imported to COORD_ACTIONS
-0 rows imported to BUNDLE_JOBS
-0 rows imported to BUNDLE_ACTIONS
-0 rows imported to SLA_REGISTRATION
-0 rows imported to SLA_SUMMARY
-```
-
-NOTE: The database version of the zip must match the version of the Oozie database it's imported to.
-
-After starting the Oozie server, the history and the currently running workflows should be available.
-
-**IMPORTANT:** The tool was primarily developed to make the migration from embedded databases (e.g. Derby) to standalone databases
- (e.g. MySQL, PosgreSQL, Oracle, MS SQL Server), though it will work between any supported databases.
-It is **not** optimized to handle databases over 1 Gb. If the database size is larger, it should be purged before migration.
-
-## Oozie Configuration
-
-By default, Oozie configuration is read from Oozie's `conf/` directory
-
-The Oozie configuration is distributed in 3 different files:
-
- * `oozie-site.xml` : Oozie server configuration
- * `oozie-log4j.properties` : Oozie logging configuration
- * `adminusers.txt` : Oozie admin users list
-
-### Oozie Configuration Properties
-
-All Oozie configuration properties and their default values are defined in the `oozie-default.xml` file.
-
-Oozie resolves configuration property values in the following order:
-
- * If a Java System property is defined, it uses its value
- * Else, if the Oozie configuration file (`oozie-site.xml`) contains the property, it uses its value
- * Else, it uses the default value documented in the `oozie-default.xml` file
-
-**NOTE:** The `oozie-default.xml` file found in Oozie's `conf/` directory is not used by Oozie, it is there
-for reference purposes only.
-
-### Precedence of Configuration Properties
-
-For compatibility reasons across Hadoop / Oozie versions, some configuration properties can be defined using multiple keys
-in the launcher configuration. Beginning with Oozie 5.0.0, some of them can be overridden, some others will be prepended to default
-configuration values.
-
-#### Overriding Configuration Values
-
-Overriding happens for following configuration entries with `oozie.launcher` prefix, by switching `oozie.launcher.override`
-(on by default).
-
-For those, following is the general approach:
-
- * check whether a YARN compatible entry is present. If yes, use it to override default value
- * check whether a MapReduce v2 compatible entry is present. If yes, use it to override default value
- * check whether a MapReduce v1 compatible entry is present. If yes, use it to override default value
- * use default value
-
-Such properties are (legend: YARN / MapReduce v2 / MapReduce v1):
-
- * max attempts of the MapReduce Application Master:
- * N / A
- * `mapreduce.map.maxattempts`
- * `mapred.map.max.attempts`
- * memory amount in MB of the MapReduce Application Master:
- * `yarn.app.mapreduce.am.resource.mb`
- * `mapreduce.map.memory.mb`
- * `mapred.job.map.memory.mb`
- * CPU vcore count of the MapReduce Application Master:
- * `yarn.app.mapreduce.am.resource.cpu-vcores`
- * `mapreduce.map.cpu.vcores`
- * N / A
- * logging level of the MapReduce Application Master:
- * N / A
- * `mapreduce.map.log.level`
- * `mapred.map.child.log.level`
- * MapReduce Application Master JVM options:
- * `yarn.app.mapreduce.am.command-opts`
- * `mapreduce.map.java.opts`
- * `mapred.child.java.opts`
- * MapReduce Application Master environment variable settings:
- * `yarn.app.mapreduce.am.env`
- * `mapreduce.map.env`
- * `mapred.child.env`
- * MapReduce Application Master job priority:
- * N / A
- * `mapreduce.job.priority`
- * `mapred.job.priority`
- * MapReduce Application Master job queue name:
- * N / A
- * `mapreduce.job.queuename`
- * `mapred.job.queue.name`
- * MapReduce View ACL settings:
- * N / A
- * `mapreduce.job.acl-view-job`
- * N / A
- * MapReduce Modify ACL settings:
- * N / A
- * `mapreduce.job.acl-modify-job`
- * N / A
-
-This list can be extended or modified by adding new configuration entries or updating existing values
-beginning with `oozie.launcher.override.` within `oozie-site.xml`. Examples can be found in `oozie-default.xml`.
-
-#### Prepending Configuration Values
-
-Prepending happens for following configuration entries with `oozie.launcher` prefix, by switching `oozie.launcher.prepend`
-(on by default).
-
-For those, following is the general approach:
-
- * check whether a YARN compatible entry is present. If yes, use it to prepend to default value
- * use default value
-
-Such properties are (legend: YARN only):
-
- * MapReduce Application Master JVM options: `yarn.app.mapreduce.am.admin-command-opts`
- * MapReduce Application Master environment settings: `yarn.app.mapreduce.am.admin.user.env`
-
-This list can be extended or modified by adding new configuration entries or updating existing values
-beginning with `oozie.launcher.prepend.` within `oozie-site.xml`. Examples can be found in `oozie-default.xml`.
-
-### Logging Configuration
-
-By default, Oozie log configuration is defined in the `oozie-log4j.properties` configuration file.
-
-If the Oozie log configuration file changes, Oozie reloads the new settings automatically.
-
-By default, Oozie logs to Oozie's `logs/` directory.
-
-Oozie logs in 4 different files:
-
- * oozie.log: web services log streaming works from this log
- * oozie-ops.log: messages for Admin/Operations to monitor
- * oozie-instrumentation.log: instrumentation data, every 60 seconds (configurable)
- * oozie-audit.log: audit messages, workflow jobs changes
-
-The embedded Jetty and embedded Derby log files are also written to Oozie's `logs/` directory.
-
-### Oozie User Authentication Configuration
-
-Oozie supports Kerberos HTTP SPNEGO authentication, pseudo/simple authentication and anonymous access
-for client connections.
-
-Anonymous access (**default**) does not require the user to authenticate and the user ID is obtained from
-the job properties on job submission operations, other operations are anonymous.
-
-Pseudo/simple authentication requires the user to specify the user name on the request, this is done by
-the PseudoAuthenticator class by injecting the `user.name` parameter in the query string of all requests.
-The `user.name` parameter value is taken from the client process Java System property `user.name`.
-
-Kerberos HTTP SPNEGO authentication requires the user to perform a Kerberos HTTP SPNEGO authentication sequence.
-
-If Pseudo/simple or Kerberos HTTP SPNEGO authentication mechanisms are used, Oozie will return the user an
-authentication token HTTP Cookie that can be used in later requests as identity proof.
-
-Oozie uses Apache Hadoop-Auth (Java HTTP SPNEGO) library for authentication.
-This library can be extended to support other authentication mechanisms.
-
-Oozie user authentication is configured using the following configuration properties (default values shown):
-
-
-```
- oozie.authentication.type=simple
- oozie.authentication.token.validity=36000
- oozie.authentication.signature.secret=
- oozie.authentication.cookie.domain=
- oozie.authentication.simple.anonymous.allowed=true
- oozie.authentication.kerberos.principal=HTTP/localhost@${local.realm}
- oozie.authentication.kerberos.keytab=${oozie.service.HadoopAccessorService.keytab.file}
-```
-
-The `type` defines authentication used for Oozie HTTP endpoint, the supported values are:
-simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#.
-
-The `token.validity` indicates how long (in seconds) an authentication token is valid before it has
-to be renewed.
-
-The `signature.secret` is the signature secret for signing the authentication tokens. It is recommended to not set this, in which
-case Oozie will randomly generate one on startup.
-
-The `oozie.authentication.cookie.domain` The domain to use for the HTTP cookie that stores the
-authentication token. In order to authentication to work correctly across all Hadoop nodes web-consoles
-the domain must be correctly set.
-
-The `simple.anonymous.allowed` indicates if anonymous requests are allowed. This setting is meaningful
-only when using 'simple' authentication.
-
-The `kerberos.principal` indicates the Kerberos principal to be used for HTTP endpoint.
-The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
-
-The `kerberos.keytab` indicates the location of the keytab file with the credentials for the principal.
-It should be the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
-
-### Oozie Hadoop Authentication Configuration
-
-Oozie works with Hadoop versions which support Kerberos authentication.
-
-Oozie Hadoop authentication is configured using the following configuration properties (default values shown):
-
-
-```
- oozie.service.HadoopAccessorService.kerberos.enabled=false
- local.realm=LOCALHOST
- oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
- oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}
-```
-
-The above default values are for a Hadoop 0.20 secure distribution (with support for Kerberos authentication).
-
-To enable Kerberos authentication, the following property must be set:
-
-
-```
- oozie.service.HadoopAccessorService.kerberos.enabled=true
-```
-
-When using Kerberos authentication, the following properties must be set to the correct values (default values shown):
-
-
-```
- local.realm=LOCALHOST
- oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
- oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}
-```
-
-**IMPORTANT:** When using Oozie with a Hadoop 20 with Security distribution, the Oozie user in Hadoop must be configured
-as a proxy user.
-
-### User ProxyUser Configuration
-
-Oozie supports impersonation or proxyuser functionality (identical to Hadoop proxyuser capabilities and conceptually
-similar to Unix 'sudo').
-
-Proxyuser enables other systems that are Oozie clients to submit jobs on behalf of other users.
-
-Because proxyuser is a powerful capability, Oozie provides the following restriction capabilities
-(similar to Hadoop):
-
- * Proxyuser is an explicit configuration on per proxyuser user basis.
- * A proxyuser user can be restricted to impersonate other users from a set of hosts.
- * A proxyuser user can be restricted to impersonate users belonging to a set of groups.
-
-There are 2 configuration properties needed to set up a proxyuser:
-
- * oozie.service.ProxyUserService.proxyuser.#USER#.hosts: hosts from where the user #USER# can impersonate other users.
- * oozie.service.ProxyUserService.proxyuser.#USER#.groups: groups the users being impersonated by user #USER# must belong to.
-
-Both properties support the '*' wildcard as value. Although this is recommended only for testing/development.
-
-### User Authorization Configuration
-
-Oozie has a basic authorization model:
-
- * Users have read access to all jobs
- * Users have write access to their own jobs
- * Users have write access to jobs based on an Access Control List (list of users and groups)
- * Users have read access to admin operations
- * Admin users have write access to all jobs
- * Admin users have write access to admin operations
-
-If security is disabled all users are admin users.
-
-Oozie security is set via the following configuration property (default value shown):
-
-
-```
- oozie.service.AuthorizationService.security.enabled=false
-```
-
-NOTE: the old ACL model where a group was provided is still supported if the following property is set
-in `oozie-site.xml`:
-
-
-```
- oozie.service.AuthorizationService.default.group.as.acl=true
-```
-
-#### Defining Admin Users
-
-Admin users are determined from the list of admin groups, specified in
- `oozie.service.AuthorizationService.admin.groups` property. Use commas to separate multiple groups, spaces, tabs
-and ENTER characters are trimmed.
-
-If the above property for admin groups is not set, then defining the admin users can happen in the following manners.
-The list of admin users can be in the `conf/adminusers.txt` file. The syntax of this file is:
-
- * One user name per line
- * Empty lines and lines starting with '#' are ignored
-
-Admin users can also be defined in
-`oozie.serviceAuthorizationService.admin.users` property. Use commas to separate multiple admin users, spaces, tabs
-and ENTER characters are trimmed.
-
-In case there are admin users defined using both methods, the effective list of admin users will be the union
-of the admin users found in the adminusers.txt and those specified with `oozie.serviceAuthorizationService.admin.users`.
-
-#### Defining Access Control Lists
-
-Access Control Lists are defined in the following ways:
-
- * workflow job submission over CLI: configuration property `group.name` of `job.properties`
- * workflow job submission over HTTP: configuration property `group.name` of the XML submitted over HTTP
- * workflow job re-run: configuration property `oozie.job.acl` (preferred) or configuration property `group.name` of
- `job.properties`
- * coordinator job submission over CLI: configuration property `oozie.job.acl` (preferred) or configuration property `group.name`
- of `job.properties`
- * bundle job submission over CLI: configuration property `oozie.job.acl` (preferred) or configuration property `group.name` of
- `job.properties`
-
-For all other workflow, coordinator, or bundle actions the ACL set in beforehand will be used as basis.
-
-Once the ACL for the job is defined, Oozie will check over HDFS whether the user trying to perform a specific action is part of the
-necessary group(s). For implementation details please check out `org.apache.hadoop.security.Groups#getGroups(String user)`.
-
-Note that it's enough that the submitting user be part of at least one group of the ACL. Note also that the ACL can contain user
-names as well. If there is an ACL defined and the submitting user isn't part of any group or user name present in the ACL, an
-`AuthorizationException` is thrown.
-
-**Example: A typical ACL setup**
-
-Detail of `job.properties` on workflow job submission:
-
-```
-user.name=joe
-group.name=marketing,admin,qa,root
-```
-
-HDFS group membership of HDFS user `joe` is `qa`. That is, the check to `org.apache.hadoop.security.Groups#getGroups("joe")` returns
-`qa`. Hence, ACL check will pass inside `AuthorizationService`, because the `user.name` provided belongs to at least of the ACL list
-elements provided as `group.name`.
-
-### Oozie System ID Configuration
-
-Oozie has a system ID that is is used to generate the Oozie temporary runtime directory, the workflow job IDs, and the
-workflow action IDs.
-
-Two Oozie systems running with the same ID will not have any conflict but in case of troubleshooting it will be easier
-to identify resources created/used by the different Oozie systems if they have different system IDs (default value
-shown):
-
-
-```
- oozie.system.id=oozie-${user.name}
-```
-
-### Filesystem Configuration
-
-Oozie lets you to configure the allowed Filesystems by using the following configuration property in oozie-site.xml:
-
-```
- <property>
- <name>oozie.service.HadoopAccessorService.supported.filesystems</name>
- <value>hdfs</value>
- </property>
-```
-
-The above value, `hdfs`, which is the default, means that Oozie will only allow HDFS filesystems to be used. Examples of other
-filesystems that Oozie is compatible with are: hdfs, hftp, webhdfs, and viewfs. Multiple filesystems can be specified as
-comma-separated values. Putting a * will allow any filesystem type, effectively disabling this check.
-
-### HCatalog Configuration
-
-Refer to the [Oozie HCatalog Integration](DG_HCatalogIntegration.html) document for a overview of HCatalog and
-integration of Oozie with HCatalog. This section explains the various settings to be configured in oozie-site.xml on
-the Oozie server to enable Oozie to work with HCatalog.
-
-**Adding HCatalog jars to Oozie war:**
-
- For Oozie server to talk to HCatalog server, HCatalog and hive jars need to be in the server classpath.
-hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath or specified by the
-following configuration property in oozie-site.xml:
-
-```
- <property>
- <name>oozie.service.HCatAccessorService.hcat.configuration</name>
- <value>/local/filesystem/path/to/hive-site.xml</value>
- </property>
-```
-The hive-site.xml can also be placed in a location on HDFS and the above property can have a value
-of `hdfs://HOST:PORT/path/to/hive-site.xml` to point there instead of the local file system.
-
-The oozie-[version]-hcataloglibs.tar.gz in the oozie distribution bundles the required hcatalog and hive jars that
-needs to be placed in the Oozie server classpath. If using a version of HCatalog bundled in
-Oozie hcataloglibs/, copy the corresponding HCatalog jars from hcataloglibs/ to the libext/ directory. If using a
-different version of HCatalog, copy the required HCatalog jars from such version in the libext/ directory.
-This needs to be done before running the `oozie-setup.sh` script so that these jars get added for Oozie.
-
-**Configure HCatalog URI Handling:**
-
-
-```
- <property>
- <name>oozie.service.URIHandlerService.uri.handlers</name>
- <value>org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler</value>
- <description>
- Enlist the different uri handlers supported for data availability checks.
- </description>
- </property>
-```
-
-The above configuration defines the different uri handlers which check for existence of data dependencies defined in a
-Coordinator. The default value is `org.apache.oozie.dependency.FSURIHandler`. FSURIHandler supports uris with
-schemes defined in the configuration `oozie.service.HadoopAccessorService.supported.filesystems` which are hdfs, hftp
-and webhcat by default. HCatURIHandler supports uris with the scheme as hcat.
-
-**Configure HCatalog services:**
-
-
-```
- <property>
- <name>oozie.services.ext</name>
- <value>
- org.apache.oozie.service.JMSAccessorService,
- org.apache.oozie.service.PartitionDependencyManagerService,
- org.apache.oozie.service.HCatAccessorService
- </value>
- <description>
- To add/replace services defined in 'oozie.services' with custom implementations.
- Class names must be separated by commas.
- </description>
- </property>
-```
-
-PartitionDependencyManagerService and HCatAccessorService are required to work with HCatalog and support Coordinators
-having HCatalog uris as data dependency. If the HCatalog server is configured to publish partition availability
-notifications to a JMS compliant messaging provider like ActiveMQ, then JMSAccessorService needs to be added
-to `oozie.services.ext` to handle those notifications.
-
-**Configure JMS Provider JNDI connection mapping for HCatalog:**
-
-
-```
- <property>
- <name>oozie.service.HCatAccessorService.jmsconnections</name>
- <value>
- hcat://hcatserver.colo1.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.colo1.com:61616,
- default=java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://broker.colo.com:61616;connectionFactoryNames#ConnectionFactory
- </value>
- <description>
- Specify the map of endpoints to JMS configuration properties. In general, endpoint
- identifies the HCatalog server URL. "default" is used if no endpoint is mentioned
- in the query. If some JMS property is not defined, the system will use the property
- defined jndi.properties. jndi.properties files is retrieved from the application classpath.
- Mapping rules can also be provided for mapping Hcatalog servers to corresponding JMS providers.
- hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616
- </description>
- </property>
-```
-
- Currently HCatalog does not provide APIs to get the connection details to connect to the JMS Provider it publishes
-notifications to. It only has APIs which provide the topic name in the JMS Provider to which the notifications are
-published for a given database table. So the JMS Provider's connection properties needs to be manually configured
-in Oozie using the above setting. You can either provide a `default` JNDI configuration which will be used as the
-JMS Provider for all HCatalog servers, or can specify a configuration per HCatalog server URL or provide a
-configuration based on a rule matching multiple HCatalog server URLs. For example: With the configuration of
-`hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616`,
-request URL of `hcat://server1.colo1.com:8020 `will map to`tcp://broker.colo1.com:61616`, `hcat://server2.colo2.com:8020`
-will map to`tcp://broker.colo2.com:61616` and so on.
-
-**Configure HCatalog Polling Frequency:**
-
-
-```
- <property>
- <name>oozie.service.coord.push.check.requeue.interval
- </name>
- <value>600000</value>
- <description>Command re-queue interval for push dependencies (in millisecond).
- </description>
- </property>
-```
-
- If there is no JMS Provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined
-in `oozie.service.coord.input.check.requeue.interval`. This config also applies to HDFS polling.
-If there is a JMS provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined
-in `oozie.service.coord.push.check.requeue.interval` as a fallback.
-The defaults for `oozie.service.coord.input.check.requeue.interval` and `oozie.service.coord.push.check.requeue.interval`
-are 1 minute and 10 minutes respectively.
-
-### Notifications Configuration
-
-Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss events. For
-more information on the feature, refer [JMS Notifications](DG_JMSNotifications.html) documentation. Oozie can also send email
-notifications on SLA misses.
-
- * **Message Broker Installation**: <br/>
-For Oozie to send/receive messages, a JMS-compliant broker should be installed. Apache ActiveMQ is a popular JMS-compliant
-broker usable for this purpose. See [here](http://activemq.apache.org/getting-started.html) for instructions on
-installing and running ActiveMQ.
-
- * **Services**: <br/>
-Add/modify `oozie.services.ext` property in `oozie-site.xml` to include the following services.
-
-```
- <property>
- <name>oozie.services.ext</name>
- <value>
- org.apache.oozie.service.JMSAccessorService,
- org.apache.oozie.service.JMSTopicService,
- org.apache.oozie.service.EventHandlerService,
- org.apache.oozie.sla.service.SLAService
- </value>
- </property>
-```
-
- * **Event Handlers**: <br/>
-
-```
- <property>
- <name>oozie.service.EventHandlerService.event.listeners</name>
- <value>
- org.apache.oozie.jms.JMSJobEventListener,
- org.apache.oozie.sla.listener.SLAJobEventListener,
- org.apache.oozie.jms.JMSSLAEventListener,
- org.apache.oozie.sla.listener.SLAEmailEventListener
- </value>
- </property>
-```
-It is also recommended to increase `oozie.service.SchedulerService.threads` to 15 for faster event processing and sending notifications. The services and their functions are as follows: <br/>
- JMSJobEventListener - Sends JMS job notifications <br/>
- JMSSLAEventListener - Sends JMS SLA notifications <br/>
- SLAEmailEventListener - Sends Email SLA notifications <br/>
- SLAJobEventListener - Processes job events and calculates SLA. Does not send any notifications
-
- * **JMS properties**: <br/>
-Add `oozie.jms.producer.connection.properties` property in `oozie-site.xml`. Its value corresponds to an
-identifier (e.g. default) assigned to a semi-colon separated key#value list of properties from your JMS broker's
-`jndi.properties` file. The important properties are `java.naming.factory.initial` and `java.naming.provider.url`.
-
-As an example, if using ActiveMQ in local env, the property can be set to
-
-```
- <property>
- <name>oozie.jms.producer.connection.properties</name>
- <value>
- java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory
- </value>
- </property>
-```
- * **JMS Topic name**: <br/>
-JMS consumers listen on a particular "topic". Hence Oozie needs to define a topic variable with which to publish messages
-about the various jobs.
-
-```
- <property>
- <name>oozie.service.JMSTopicService.topic.name</name>
- <value>
- default=${username}
- </value>
- <description>
- Topic options are ${username}, ${jobId}, or a fixed string which can be specified as default or for a
- particular job type.
- For e.g To have a fixed string topic for workflows, coordinators and bundles,
- specify in the following comma-separated format: {jobtype1}={some_string1}, {jobtype2}={some_string2}
- where job type can be WORKFLOW, COORDINATOR or BUNDLE.
- Following example defines topic for workflow job, workflow action, coordinator job, coordinator action,
- bundle job and bundle action
- WORKFLOW=workflow,
- COORDINATOR=coordinator,
- BUNDLE=bundle
- For jobs with no defined topic, default topic will be ${username}
- </description>
- </property>
-```
-
-Another related property is the topic prefix.
-
-```
- <property>
- <name>oozie.service.JMSTopicService.topic.prefix</name>
- <value></value>
- <description>
- This can be used to append a prefix to the topic in oozie.service.JMSTopicService.topic.name. For eg: oozie.
- </description>
- </property>
-```
-
-
-### Setting Up Oozie with HTTPS (SSL)
-
-**IMPORTANT**:
-The default HTTPS configuration will cause all Oozie URLs to use HTTPS except for the JobTracker callback URLs. This is to simplify
-configuration (no changes needed outside of Oozie), but this is okay because Oozie doesn't inherently trust the callbacks anyway;
-they are used as hints.
-
-The related environment variables are explained at [Environment Setup](AG_Install.html#Environment_Setup).
-
-You can use either a certificate from a Certificate Authority or a Self-Signed Certificate. Using a self-signed certificate
-requires some additional configuration on each Oozie client machine. If possible, a certificate from a Certificate Authority is
-recommended because it's simpler to configure.
-
-There's also some additional considerations when using Oozie HA with HTTPS.
-
-#### To use a Self-Signed Certificate
-There are many ways to create a Self-Signed Certificate, this is just one way. We will be using
-the [keytool](http://docs.oracle.com/javase/6/docs/technotes/tools/solaris/keytool.html) program, which is
-included with your JRE. If it's not on your path, you should be able to find it in $JAVA_HOME/bin.
-
-1. Run the following command (as the Oozie user) to create the keystore file, which will be named `.keystore` and located in the
- Oozie user's home directory.
-
- ```
- keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=hostname" -storepass password -keypass password
- ```
- The `hostname` should be the host name of the Oozie Server or a wildcard on the subdomain it belongs to. Make sure to include
- the "CN=" part. You can change `storepass` and `keypass` values, but they should be the same. If you do want to use something
- other than password, you'll also need to change the value of the `oozie.https.keystore.pass` property in `oozie-site.xml` to
- match; `password` is the default.
-
- For example, if your Oozie server was at oozie.int.example.com, then you would do this:
-
- ```
- keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=oozie.int.example.com" -storepass password -keypass password
- ```
- If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie servers in the HA group can use.
- To do that, you'll need to use a wildcard on the subdomain it belongs to:
-
- ```
- keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=*.int.example.com" -storepass password -keypass password
- ```
- The above would work on any server in the int.example.com domain.
-
-2. Run the following command (as the Oozie user) to export a certificate file from the keystore file:
-
- ```
- keytool -exportcert -alias jetty -file path/to/anywhere/certificate.cert -storepass password
- ```
-
-3. Run the following command (as any user) to create a truststore containing the certificate we just exported:
-
- ```
- keytool -import -alias jetty -file path/to/certificate.cert -keystore /path/to/anywhere/oozie.truststore -storepass password2
- ```
- You'll need the `oozie.truststore` later if you're using the Oozie client (or other Java-based client); otherwise, you can skip
- this step. The `storepass` value here is only used to verify or change the truststore and isn't typically required when only
- reading from it; so it does not have to be given to users only using the client.
-
-#### To use a Certificate from a Certificate Authority
-
-1. You will need to make a request to a Certificate Authority in order to obtain a proper Certificate; please consult a Certificate
- Authority on this procedure. If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie
- servers in the HA group can use. To do that, you'll need to use a wild on the subdomain it belongs to (e.g. "*.int.example.com").
-
-2. Once you have your .cert file, run the following command (as the Oozie user) to create a keystore file from your certificate:
-
- ```
- keytool -import -alias jetty -file path/to/certificate.cert
- ```
- The keystore file will be named `.keystore` and located in the Oozie user's home directory.
-
-#### Configure the Oozie Server to use SSL (HTTPS)
-
-1. Make sure the Oozie server isn't running
-
-2. Configure settings necessary for enabling SSL/TLS support in `oozie-site.xml`.
-
- 2a. Set `oozie.https.enabled` to `true`. To revert back to HTTP, set `oozie.https.enabled` to `false`.
-
- 2b. Set location and password for the keystore and location for truststore by setting `oozie.https.keystore.file`,
- `oozie.https.keystore.pass`, `oozie.https.truststore.file`.
-
- **Note:** `oozie.https.truststore.file` can be overridden by setting `javax.net.ssl.trustStore` system property.
-
- The default HTTPS port Oozie listens on for secure connections is 11443; it can be changed via `oozie.https.port`.
-
- It is possible to specify other HTTPS settings via `oozie-site.xml`:
- - To include / exclude cipher suites, set `oozie.https.include.cipher.suites` / `oozie.https.exclude.cipher.suites`.
- - To include / exclude TLS protocols, set `oozie.https.include.protocols` / `oozie.https.exclude.protocols`.
- **Note:** Exclude is always preferred over include (i.e. if you both include and exclude an entity, it will be excluded).
-
-3. Start the Oozie server
-
- **Note:** If using Oozie HA, make sure that each Oozie server has a copy of the .keystore file.
-
-#### Configure the Oozie Client to connect using SSL (HTTPS)
-
-The first two steps are only necessary if you are using a Self-Signed Certificate; the third is required either way.
-Also, these steps must be done on every machine where you intend to use the Oozie Client.
-
-1. Copy or download the oozie.truststore file onto the client machine
-
-2. When using any Java-based program, you'll need to pass `-Djavax.net.ssl.trustStore` to the JVM. To
- do this for the Oozie client:
-
- ```
- export OOZIE_CLIENT_OPTS='-Djavax.net.ssl.trustStore=/path/to/oozie.truststore'
- ```
-
-3. When using the Oozie Client, you will need to use `https://oozie.server.hostname:11443/oozie` instead of
- `http://oozie.server.hostname:11000/oozie` -- Java will not automatically redirect from the http address to the https address.
-
-#### Connect to the Oozie Web UI using SSL (HTTPS)
-
-1. Use `https://oozie.server.hostname:11443/oozie`
- though most browsers should automatically redirect you if you use `http://oozie.server.hostname:11000/oozie`
-
- **IMPORTANT**: If using a Self-Signed Certificate, your browser will warn you that it can't verify the certificate or something
- similar. You will probably have to add your certificate as an exception.
-
-#### Additional considerations for Oozie HA with SSL
-
-You'll need to configure the load balancer to do SSL pass-through. This will allow the clients talking to Oozie to use the
-SSL certificate provided by the Oozie servers (so the load balancer does not need one). Please consult your load balancer's
-documentation on how to configure this. Make sure to point the load balancer at the `https://HOST:HTTPS_PORT` addresses for your
-Oozie servers. Clients can then connect to the load balancer at `https://LOAD_BALANCER_HOST:PORT`.
-
-**Important:** Callbacks from the ApplicationMaster are done via http or https depending on what you enter for the
-`OOZIE_BASE_URL` property. If you are using a Certificate from a Certificate Authority, you can simply put the https address here.
-If you are using a self-signed certificate, you have to do one of the following options (Option 1 is recommended):
-
-Option 1) You'll need to follow the steps in
-the [Configure the Oozie Client to connect using SSL (HTTPS)](AG_Install.html#Configure_the_Oozie_Client_to_connect_using_SSL_HTTPS)
-section, but on the host of the ApplicationMaster. You can then set `OOZIE_BASE_URL` to the load balancer https address.
-This will allow the ApplicationMaster to contact the Oozie server with https (like the Oozie client, they are also Java
-programs).
-
-Option 2) You'll need setup another load balancer, or another "pool" on the existing load balancer, with the http addresses of the
-Oozie servers. You can then set `OOZIE_BASE_URL` to the load balancer http address. Clients should use the https load balancer
-address. This will allow clients to use https while the ApplicationMaster uses http for callbacks.
-
-### Fine Tuning an Oozie Server
-
-Refer to the [oozie-default.xml](./oozie-default.xml) for details.
-
-### Using Instrumentation instead of Metrics
-
-As of version 4.1.0, Oozie includes a replacement for the Instrumentation based on Codahale's Metrics library. It includes a
-number of improvements over the original Instrumentation included in Oozie. They both report most of the same information, though
-the formatting is slightly different and there's some additional information in the Metrics version; the format of the output to the
-oozie-instrumentation log is also different.
-
-As of version 5.0.0, `MetricsInstrumentationService` is the default one, it's enlisted in `oozie.services`:
-
-```
- <property>
- <name>oozie.services</name>
- <value>
- ...
- org.apache.oozie.service.MetricsInstrumentationService,
- ...
- </value>
- </property>
-```
-
-The deprecated `InstrumentationService` can be enabled by adding `InstrumentationService` reference to the list of
-`oozie.services.ext`:
-
-```
- <property>
- <name>oozie.services.ext</name>
- <value>
- ...
- org.apache.oozie.service.InstrumentationService,
- ...
- </value>
- </property>
-```
-
-By default the `admin/instrumentation` REST endpoint is no longer be available and instead the `admin/metrics` endpoint can
-be used (see the [Web Services API](WebServicesAPI.html#Oozie_Metrics) documentation for more details); the Oozie Web UI also replaces
-the "Instrumentation" tab with a "Metrics" tab.
-
-If the deprecated `InstrumentationService` is used, the `admin/instrumentation` REST endpoint gets enabled, the `admin/metrics`
-REST endpoint is no longer available (see the [Web Services API](WebServicesAPI.html#Oozie_Metrics) documentation for more details);
-the Oozie Web UI also replaces the "Metrics" tab with the "Instrumentation" tab.
-
-We can also publish the instrumentation metrics to the external server graphite or ganglia. For this the following
-properties should be specified in oozie-site.xml :
-
-```
- <property>
- <name>oozie.external_monitoring.enable</name>
- <value>false</value>
- <description>
- If the oozie functional metrics needs to be exposed to the metrics-server backend, set it to true
- If set to true, the following properties has to be specified : oozie.metrics.server.name,
- oozie.metrics.host, oozie.metrics.prefix, oozie.metrics.report.interval.sec, oozie.metrics.port
- </description>
- </property>
-
- <property>
- <name>oozie.external_monitoring.type</name>
- <value>graphite</value>
- <description>
- The name of the server to which we want to send the metrics, would be graphite or ganglia.
- </description>
- </property>
-
- <property>
- <name>oozie.external_monitoring.address</name>
- <value>http://localhost:2020</value>
- </property>
-
- <property>
- <name>oozie.external_monitoring.metricPrefix</name>
- <value>oozie</value>
- </property>
-
- <property>
- <name>oozie.external_monitoring.reporterIntervalSecs</name>
- <value>60</value>
- </property>
-```
-
-We can also publish the instrumentation metrics via JMX interface. For this the following property should be specified
-in oozie-site.xml :
-
-```
- <property>
- <name>oozie.jmx_monitoring.enable</name>
- <value>false</value>
- <description>
- If the oozie functional metrics needs to be exposed via JMX interface, set it to true.
- </description>
- </property>>
-```
-
-<a name="HA"></a>
-### High Availability (HA)
-
-Multiple Oozie Servers can be configured against the same database to provide High Availability (HA) of the Oozie service.
-
-#### Pre-requisites
-
-1. A database that supports multiple concurrent connections. In order to have full HA, the database should also have HA support, or
- it becomes a single point of failure.
-
- **NOTE:** The default derby database does not support this
-
-2. A ZooKeeper ensemble.
-
- Apache ZooKeeper is a distributed, open-source coordination service for distributed applications; the Oozie servers use it for
- coordinating access to the database and communicating with each other. In order to have full HA, there should be at least 3
- ZooKeeper servers.
- More information on ZooKeeper can be found [here](http://zookeeper.apache.org).
-
-3. Multiple Oozie servers.
-
- **IMPORTANT:** While not strictly required for all configuration properties, all of the servers should ideally have exactly the same
- configuration for consistency's sake.
-
-4. A Loadbalancer, Virtual IP, or Round-Robin DNS.
-
- This is used to provide a single entry-point for users and for callbacks from the JobTracker/ResourceManager. The load balancer
- should be configured for round-robin between the Oozie servers to distribute the requests. Users (using either the Oozie client, a
- web browser, or the REST API) should connect through the load balancer. In order to have full HA, the load balancer should also
- have HA support, or it becomes a single point of failure.
-
-#### Installation/Configuration Steps
-
-1. Install identically configured Oozie servers normally. Make sure they are all configured against the same database and make sure
- that you DO NOT start them yet.
-
-2. Add the following services to the extension services configuration property in oozie-site.xml in all Oozie servers. This will
- make Oozie use the ZooKeeper versions of these services instead of the default implementations.
-
-
- ```
- <property>
- <name>oozie.services.ext</name>
- <value>
- org.apache.oozie.service.ZKLocksService,
- org.apache.oozie.service.ZKXLogStreamingService,
- org.apache.oozie.service.ZKJobsConcurrencyService,
- org.apache.oozie.service.ZKUUIDService
- </value>
- </property>
- ```
-
-3. Add the following property to oozie-site.xml in all Oozie servers. It should be a comma-separated list of host:port pairs of the
- ZooKeeper servers. The default value is shown below.
-
-
- ```
- <property>
- <name>oozie.zookeeper.connection.string</name>
- <value>localhost:2181</value>
- </property>
- ```
-
-4. (Optional) Add the following property to oozie-site.xml in all Oozie servers to specify the namespace to use. All of the Oozie
- Servers that are planning on talking to each other should have the same namespace. If there are multiple Oozie setups each doing
- their own HA, they should have their own namespace. The default value is shown below.
-
-
- ```
- <property>
- <name>oozie.zookeeper.namespace</name>
- <value>oozie</value>
- </property>
- ```
-
-5. Change the value of `OOZIE_BASE_URL` in oozie-site.xml to point to the loadbalancer or virtual IP, for example:
-
-
- ```
- <property>
- <name>oozie.base.url</name>
- <value>http://my.loadbalancer.hostname:11000/oozie</value>
- </property>
- ```
-
-6. (Optional) If using a secure cluster, see [Security](AG_Install.html#Security) below on configuring Kerberos with Oozie HA.
-
-7. Start the ZooKeeper servers.
-
-8. Start the Oozie servers.
-
- Note: If one of the Oozie servers becomes unavailable, querying Oozie for the logs from a job in the Web UI, REST API, or client may
- be missing information until that server comes back up.
-
-#### Security
-
-Oozie HA works with the existing Oozie security framework and settings. For HA features (log streaming, share lib, etc) to work
-properly in a secure setup, following property can be set on each server. If `oozie.server.authentication.type` is not set, then
-server-server authentication will fall back on `oozie.authentication.type`.
-
-
-```
-<property>
- <name>oozie.server.authentication.type</name>
- <value>kerberos</value>
-</property>
-```
-
-Below are some additional steps and information specific to Oozie HA:
-
-1. (Optional) To prevent unauthorized users or programs from interacting with or reading the znodes used by Oozie in ZooKeeper,
- you can tell Oozie to use Kerberos-backed ACLs. To enforce this for all of the Oozie-related znodes, simply add the following
- property to oozie-site.xml in all Oozie servers and set it to `true`. The default is `false`.
-
-
- ```
- <property>
- <name>oozie.zookeeper.secure</name>
- <value>true</value>
- </property>
- ```
-
- Note: The Kerberos principals of each of the Oozie servers should have the same primary name (i.e. in `primary/instance@REALM`, each
- server should have the same value for `primary`).
-
- **Important:** Once this property is set to `true`, it will set the ACLs on all existing Oozie-related znodes to only allow Kerberos
- authenticated users with a principal that has the same primary as described above (also for any subsequently created new znodes).
- This means that if you ever want to turn this feature off, you will have to manually connect to ZooKeeper using a Kerberos principal
- with the same primary and either delete all znodes under and including the namespace (i.e. if `oozie.zookeeper.namespace` ` `oozie=
- then that would be `/oozie`); alternatively, instead of deleting them all, you can manually set all of their ACLs to `world:anyone`.
- In either case, make sure that no Oozie servers are running while this is being done.
-
- Also, in your zoo.cfg for ZooKeeper, make sure to set the following properties:
-
- ```
- authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
- kerberos.removeHostFromPrincipal=true
- kerberos.removeRealmFromPrincipal=true
- ```
-
-2. Until Hadoop 2.5.0 and later, there is a known limitation where each Oozie server can only use one HTTP principal. However,
- for Oozie HA, we need to use two HTTP principals: `HTTP/oozie-server-host@realm` and `HTTP/load-balancer-host@realm`. This
- allows access to each Oozie server directly and through the load balancer. While users should always go through the load balancer,
- certain features (e.g. log streaming) require the Oozie servers to talk to each other directly; it can also be helpful for an
- administrator to talk directly to an Oozie server. So, if using a Hadoop version prior to 2.5.0, you will have to choose which
- HTTP principal to use as you cannot use both; it is recommended to choose `HTTP/load-balancer-host@realm` so users can connect
- through the load balancer. This will prevent Oozie servers from talking to each other directly, which will effectively disable
- log streaming.
-
- For Hadoop 2.5.0 and later:
-
- 2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the two HTTP principals mentioned above.
-
-
- 2b. Set `oozie.authentication.kerberos.principal` to * (that is, an asterisks) so it will use both HTTP principals.
-
- For earlier versions of Hadoop:
-
- 2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the load balancer HTTP principal
-
-
- 2b. Set `oozie.authentication.kerberos.principal` to `HTTP/load-balancer-host@realm`.
-
-3. With Hadoop 2.6.0 and later, a rolling random secret that is synchronized across all Oozie servers will be used for signing the
- Oozie auth tokens. This is done automatically when HA is enabled; no additional configuration is needed.
-
- For earlier versions of Hadoop, each server will have a different random secret. This will still work but will likely result in
- additional calls to the KDC to authenticate users to the Oozie server (because the auth tokens will not be accepted by other
- servers, which will cause a fallback to Kerberos).
-
-4. If you'd like to use HTTPS (SSL) with Oozie HA, there's some additional considerations that need to be made.
- See the [Setting Up Oozie with HTTPS (SSL)](AG_Install.html#Setting_Up_Oozie_with_HTTPS_SSL) section for more information.
-
-#### JobId sequence
-Oozie in HA mode, uses ZK to generate job id sequence. Job Ids are of following format.
-`<Id sequence>-<yyMMddHHmmss(server start time)>-<system_id>-<W/C/B>`
-
-Where, `<systemId>` is configured as `oozie.system.id` (default is "oozie-" + "user.name")
-W/C/B is suffix to job id indicating that generated job is a type of workflow or coordinator or bundle.
-
-Maximum allowed character for job id sequence is 40. "Id sequence" is stored in ZK and reset to 0 once maximum job id sequence is
-reached. Maximum job id sequence is configured as `oozie.service.ZKUUIDService.jobid.sequence.max`, default value is 99999999990.
-
-
-```
-<property>
- <name>oozie.service.ZKUUIDService.jobid.sequence.max</name>
- <value>99999999990</value>
-</property>
-```
-
-## Starting and Stopping Oozie
-
-Use the standard commands to start and stop Oozie.
-
-## Oozie Command Line Installation
-
-Copy and expand the `oozie-client` TAR.GZ file bundled with the distribution. Add the `bin/` directory to the `PATH`.
-
-Refer to the [Command Line Interface Utilities](DG_CommandLineTool.html) document for a full reference of the `oozie`
-command line tool.
-
-## Oozie Share Lib
-
-The Oozie sharelib TAR.GZ file bundled with the distribution contains the necessary files to run Oozie map-reduce streaming, pig,
-hive, sqooop, and distcp actions. There is also a sharelib for HCatalog. The sharelib is required for these actions to work; any
-other actions (mapreduce, shell, ssh, and java) do not require the sharelib to be installed.
-
-As of Oozie 4.0, the following property is included. If true, Oozie will create and ship a "launcher jar" to hdfs that contains
-classes necessary for the launcher job. If false, Oozie will not do this, and it is assumed that the necessary classes are in their
-respective sharelib jars or the "oozie" sharelib instead. When false, the sharelib is required for ALL actions; when true, the
-sharelib is only required for actions that need additional jars (the original list from above).
-
-
-```
-<property>
- <name>oozie.action.ship.launcher.jar</name>
- <value>true</value>
-</property>
-```
-
-Using sharelib CLI, sharelib files are copied to new lib_`<timestamped>` directory. At start, server picks the sharelib from latest
-time-stamp directory. While starting, server also purges sharelib directory which are older than sharelib retention days
-(defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).
-
-Sharelib mapping file can be also configured. Configured file is a key value mapping, where key will be the sharelib name for the
-action and value is a comma separated list of DFS or local filesystem directories or jar files. Local filesystem refers to the local
-filesystem of the node where the Oozie launcher is running. This can be configured in oozie-site.xml as :
-
-```
- <!-- OOZIE -->
- <property>
- <name>oozie.service.ShareLibService.mapping.file</name>
- <value></value>
- <description>
- Sharelib mapping files contains list of key=value,
- where key will be the sharelib name for the action and value is a comma separated list of
- DFS or local filesystem directories or jar files.
- Example.
- oozie.pig_10=hdfs:///share/lib/pig/pig-0.10.1/lib/
- oozie.pig=hdfs:///share/lib/pig/pig-0.11.1/lib/
- oozie.distcp=hdfs:///share/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-distcp-2.2.0.jar
- oozie.spark=hdfs:///share/lib/spark/lib/,hdfs:///share/lib/spark/python/lib/pyspark.zip,hdfs:///share/lib/spark/python/lib/py4j-0-9-src.zip
- oozie.hive=file:///usr/local/oozie/share/lib/hive/
- </description>
- </property>
-```
-
-Example mapping file with local filesystem resources:
-
-
-```
- <property>
- <name>oozie.service.ShareLibService.mapping.file</name>
- <value>
- oozie.distcp=file:///usr/local/oozie/share/lib/distcp
- oozie.hcatalog=file:///usr/local/oozie/share/lib/hcatalog
- oozie.hive=file:///usr/local/oozie/share/lib/hive
- oozie.hive2=file:///usr/local/oozie/share/lib/hive2
- oozie.mapreduce-streaming=file:///usr/local/oozie/share/lib/mapreduce-streaming
- oozie.oozie=file://usr/local/oozie/share/lib/oozie
- oozie.pig=file:///usr/local/oozie/share/lib/pig
- oozie.spark=file:///usr/local/oozie/share/lib/spark
- oozie.sqoop=file:///usr/localoozie/share/lib/sqoop
- </value>
- </property>
-```
-
-If you are using local filesystem resources in the mapping file, make sure corresponding jars are already deployed to
-all the nodes where Oozie launcher jobs will be executed, and the files are readable by the launchers. To do this, you
-can extract Oozie sharelib TAR.GZ file in the directory of your choice on the nodes, and set permission of the files.
-
-Oozie sharelib TAR.GZ file bundled with the distribution does not contain pyspark and py4j zip files since they vary
-with Apache Spark version. Therefore, to run pySpark using Spark Action, user need to specify pyspark and py4j zip
-files. These files can be added either to workflow's lib/ directory, to the sharelib or in sharelib mapping file.
-
-
-## Oozie Coordinators/Bundles Processing Timezone
-
-By default Oozie runs coordinator and bundle jobs using `UTC` timezone for datetime values specified in the application
-XML and in the job parameter properties. This includes coordinator applications start and end times of jobs, coordinator
-datasets initial-instance, and bundle applications kickoff times. In addition, coordinator dataset instance URI templates
-will be resolved using datetime values of the Oozie processing timezone.
-
-It is possible to set the Oozie processing timezone to a timezone that is an offset of UTC, alternate timezones must
-expressed in using a GMT offset ( `GMT+/-#### ` ). For example: `GMT+0530` (India timezone).
-
-To change the default `UTC` timezone, use the `oozie.processing.timezone` property in the `oozie-site.xml`. For example:
-
-
-```
-<configuration>
- <property>
- <name>oozie.processing.timezone</name>
- <value>GMT+0530</value>
- </property>
-</configuration>
-```
-
-**IMPORTANT:** If using a processing timezone other than `UTC`, all datetime values in coordinator and bundle jobs must
-be expressed in the corresponding timezone, for example `2012-08-08T12:42+0530`.
-
-**NOTE:** It is strongly encouraged to use `UTC`, the default Oozie processing timezone.
-
-For more details on using an alternate Oozie processing timezone, please refer to the
-[Coordinator Functional Specification, section '4. Datetime'](CoordinatorFunctionalSpec.html#datetime)
-
-<a name="UberJar"></a>
-## MapReduce Workflow Uber Jars
-For Map-Reduce jobs (not including streaming or pipes), additional jar files can also be included via an uber jar. An uber jar is a
-jar file that contains additional jar files within a "lib" folder (see
-[Workflow Functional Specification](WorkflowFunctionalSpec.html#AppDeployment) for more information). Submitting a workflow with an uber jar
-requires at least Hadoop 2.2.0 or 1.2.0. As such, using uber jars in a workflow is disabled by default. To enable this feature, use
-the `oozie.action.mapreduce.uber.jar.enable` property in the `oozie-site.xml` (and make sure to use a supported version of Hadoop).
-
-
-```
-<configuration>
- <property>
- <name>oozie.action.mapreduce.uber.jar.enable</name>
- <value>true</value>
- </property>
-</configuration>
-```
-
-## Advanced/Custom Environment Settings
-
-Oozie can be configured to use Unix standard filesystem hierarchy for its different files
-(configuration, logs, data and temporary files).
-
-These settings must be done in the `bin/oozie-env.sh` script.
-
-This script is sourced before the configuration `oozie-env.sh` and supports additional
-environment variables (shown with their default values):
-
-
-```
-export OOZIE_CONFIG=${OOZIE_HOME}/conf
-export OOZIE_DATA={OOZIE_HOME}/data
-export OOZIE_LOG={OOZIE_HOME}/logs
-export JETTY_OUT=${OOZIE_LOGS}/jetty.out
-export JETTY_PID=/tmp/oozie.pid
-```
-
-Sample values to make Oozie follow Unix standard filesystem hierarchy:
-
-
-```
-export OOZIE_CONFIG=/etc/oozie
-export OOZIE_DATA=/var/lib/oozie
-export OOZIE_LOG=/var/log/oozie
-export JETTY_PID=/tmp/oozie.pid
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/AG_Monitoring.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_Monitoring.twiki b/docs/src/site/twiki/AG_Monitoring.twiki
deleted file mode 100644
index 4877dfc..0000000
--- a/docs/src/site/twiki/AG_Monitoring.twiki
+++ /dev/null
@@ -1,199 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Oozie Monitoring
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Oozie Instrumentation
-
-Oozie code is instrumented in several places to collect runtime metrics. The instrumentation data can be used to
-determine the health of the system, performance of the system, and to tune the system.
-
-This comes in two flavors:
-
- * metrics (by default enabled since 5.0.0)
- * instrumentation (deprecated and by default disabled since 5.0.0)
-
-The instrumentation is accessible via the Admin web-services API (see the [metrics](WebServicesAPI.html#Oozie_Metrics) and
-[instrumentation](WebServicesAPI.html#Oozie_Instrumentation) Web Services API documentations for more details) and is also written on
-regular intervals to an instrumentation log.
-
-Instrumentation data includes variables, samplers, timers and counters.
-
-### Variables
-
- * oozie
- * version: Oozie build version.
-
- * configuration
- * config.dir: directory from where the configuration files are loaded. If null, all configuration files are loaded from the classpath. [Configuration files are described here](AG_Install.html#Oozie_Configuration).
- * config.file: the Oozie custom configuration for the instance.
-
- * jvm
- * free.memory
- * max.memory
- * total.memory
-
- * locks
- * locks: Locks are used by Oozie to synchronize access to workflow and action entries when the database being used does not support 'select for update' queries. (MySQL supports 'select for update').
-
- * logging
- * config.file: Log4j '.properties' configuration file.
- * from.classpath: whether the config file has been read from the classpath or from the config directory.
- * reload.interval: interval at which the config file will be reloaded. 0 if the config file will never be reloaded, when loaded from the classpath is never reloaded.
-
-### Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).
-
-Poll for data over fixed interval and generate an average over the time interval. Unless specified, all samplers in
-Oozie work on a 1 minute interval.
-
- * callablequeue
- * delayed.queue.size: The size of the delayed command queue.
- * queue.size: The size of the command queue.
- * threads.active: The number of threads processing callables.
-
- * jdbc:
- * connections.active: Active Connections over the past minute.
-
- * webservices: Requests to the Oozie HTTP endpoints over the last minute.
- * admin
- * callback
- * job
- * jobs
- * requests
- * version
-
-### Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.
-
- * action.executors - Counters related to actions.
- * [action_type]#action.[operation_performed] (start, end, check, kill)
- * [action_type]#ex.[exception_type] (transient, non-transient, error, failed)
- * e.g.
-```
-ssh#action.end: 306
-ssh#action.start: 316
-```
-
- * callablequeue - count of events in various execution queues.
- * delayed.queued: Number of commands queued with a delay.
- * executed: Number of executions from the queue.
- * failed: Number of queue attempts which failed.
- * queued: Number of queued commands.
-
- * commands: Execution Counts for various commands. This data is generated for all commands.
- * action.end
- * action.notification
- * action.start
- * callback
- * job.info
- * job.notification
- * purge
- * signal
- * start
- * submit
-
- * jobs: Job Statistics
- * start: Number of started jobs.
- * submit: Number of submitted jobs.
- * succeeded: Number of jobs which succeeded.
- * kill: Number of killed jobs.
-
- * authorization
- * failed: Number of failed authorization attempts.
-
- * webservices: Number of request to various web services along with the request type.
- * failed: total number of failed requests.
- * requests: total number of requests.
- * admin
- * admin-GET
- * callback
- * callback-GET
- * jobs
- * jobs-GET
- * jobs-POST
- * version
- * version-GET
-
-### Timers - Maintain information about the time spent in various operations.
-
- * action.executors - Counters related to actions.
- * [action_type]#action.[operation_performed] (start, end, check, kill)
-
- * callablequeue
- * time.in.queue: Time a callable spent in the queue before being processed.
-
- * commands: Generated for all Commands.
- * action.end
- * action.notification
- * action.start
- * callback
- * job.info
- * job.notification
- * purge
- * signal
- * start
- * submit
-
- * db - Timers related to various database operations.
- * create-workflow
- * load-action
- * load-pending-actions
- * load-running-actions
- * load-workflow
- * load-workflows
- * purge-old-workflows
- * save-action
- * update-action
- * update-workflow
-
- * webservices
- * admin
- * admin-GET
- * callback
- * callback-GET
- * jobs
- * jobs-GET
- * jobs-POST
- * version
- * version-GET
-
-## Oozie JVM Thread Dump
-The `admin/jvminfo.jsp` servlet can be used to get some basic jvm stats and thread dump.
-For eg: `http://localhost:11000/oozie/admin/jvminfo.jsp?cpuwatch=1000&threadsort=cpu`. It takes the following optional
-query parameters:
-
- * threadsort - The order in which the threads are sorted for display. Valid values are name, cpu, state. Default is state.
- * cpuwatch - Time interval in milliseconds to monitor cpu usage of threads. Default value is 0.
-
-## Monitoring Database Schema Integrity
-
-Oozie stores all of its state in a database. Hence, ensuring that the database schema is correct is very important to ensuring that
-Oozie is healthy and behaves correctly. To help with this, Oozie includes a `SchemaCheckerService` which periodically runs and
-performs a series of checks on the database schema. More specifically, it checks the following:
-
- * Existence of the required tables
- * Existence of the required columns in each table
- * Each column has the correct type and default value
- * Existence of the required primary keys and indexes
-
-After each run, the `SchemaCheckerService` writes the result of the checks to the Oozie log and to the "schema-checker.status"
-instrumentation variable. If there's a problem, it will be logged at the ERROR level, while correct checks are logged at the DEBUG
-level.
-
-By default, the `SchemaCheckerService` runs every 7 days. This can be configured
-by `oozie.service.SchemaCheckerService.check.interval`
-
-By default, the `SchemaCheckerService` will consider "extra" tables, columns, and indexes to be incorrect. Advanced users who have
-added additional tables, columns, and indexes can tell Oozie to ignore these by
-setting `oozie.service.SchemaCheckerService.ignore.extras` to `false`.
-
-The `SchemaCheckerService` currently only supports MySQL, PostgreSQL, and Oracle databases. SQL Server and Derby are currently not
-supported.
-
-When Oozie HA is enabled, only one of the Oozie servers will perform the checks.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/AG_OozieUpgrade.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_OozieUpgrade.twiki b/docs/src/site/twiki/AG_OozieUpgrade.twiki
deleted file mode 100644
index ec24011..0000000
--- a/docs/src/site/twiki/AG_OozieUpgrade.twiki
+++ /dev/null
@@ -1,86 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Oozie Upgrade
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Preparation
-
-Make sure there are not Workflows in RUNNING or SUSPENDED status, otherwise the database upgrade will fail.
-
-Shutdown Oozie and backup the Oozie database.
-
-Copy the oozie-site.xml from your current setup.
-
-## Oozie Server Upgrade
-
-Expand the new Oozie tarball in a new location.
-
-Edit the new `oozie-site.xml` setting all custom properties values from the old `oozie-site.xml`
-
-IMPORTANT: From Oozie 2.x to Oozie 3.x the names of the database configuration properties have
-changed. Their prefix has changed from `oozie.service.StoreService.*` to `oozie.service.JPAService.*`.
-Make sure you are using the new prefix.
-
-After upgrading the Oozie server, the `oozie-setup.sh` MUST be rerun before starting the
-upgraded Oozie server.
-
-Oozie database migration is required when there Oozie database schema changes, like
-upgrading from Oozie 2.x to Oozie 3.x.
-
-Configure the oozie-site.xml with the correct database configuration properties as
-explained in the 'Database Configuration' section in [Oozie Install](AG_Install.html).
-
-Once `oozie-site.xml` has been configured with the database configuration execute the `ooziedb.sh`
-command line tool to upgrade the database:
-
-
-```
-$ bin/ooziedb.sh upgrade -run
-
-Validate DB Connection.
-DONE
-Check DB schema exists
-DONE
-Check OOZIE_SYS table does not exist
-DONE
-Verify there are not active Workflow Jobs
-DONE
-Create SQL schema
-DONE
-DONE
-Create OOZIE_SYS table
-DONE
-Upgrade COORD_JOBS new columns default values.
-DONE
-Upgrade COORD_JOBS & COORD_ACTIONS status values.
-DONE
-Table 'WF_ACTIONS' column 'execution_path', length changed to 1024
-DONE
-
-Oozie DB has been upgraded to Oozie version '3.2.0'
-
-The SQL commands have been written to: /tmp/ooziedb-5737263881793872034.sql
-
-$
-```
-
-The new version of the Oozie server is ready to be started.
-
-NOTE: If using MySQL or Oracle, copy the corresponding JDBC driver JAR file to the `libext/` directory before running
-the `ooziedb.sh` command line tool.
-
-NOTE: If instead using the '-run' option, the `-sqlfile <FILE>` option is used, then all the
-database changes will be written to the specified file and the database won't be modified.
-
-## Oozie Client Upgrade
-
-While older Oozie clients work with newer Oozie server, to have access to all the
-functionality of the Oozie server the same version of Oozie client should be installed
-and used by users.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
[14/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_SparkActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_SparkActionExtension.md b/docs/src/site/markdown/DG_SparkActionExtension.md
new file mode 100644
index 0000000..5a56cca
--- /dev/null
+++ b/docs/src/site/markdown/DG_SparkActionExtension.md
@@ -0,0 +1,436 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Spark Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Spark Action
+
+The `spark` action runs a Spark job.
+
+The workflow job will wait until the Spark job completes before
+continuing to the next action.
+
+To run the Spark job, you have to configure the `spark` action with
+the `resource-manager`, `name-node`, Spark `master` elements as
+well as the necessary elements, arguments and configuration.
+
+Spark options can be specified in an element called `spark-opts`.
+
+A `spark` action can be configured to create or delete HDFS directories
+before starting the Spark job.
+
+Oozie EL expressions can be used in the inline configuration. Property
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <spark xmlns="uri:oozie:spark-action:1.0">
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[SPARK SETTINGS FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <master>[SPARK MASTER URL]</master>
+ <mode>[SPARK MODE]</mode>
+ <name>[SPARK JOB NAME]</name>
+ <class>[SPARK MAIN CLASS]</class>
+ <jar>[SPARK DEPENDENCIES JAR / PYTHON FILE]</jar>
+ <spark-opts>[SPARK-OPTIONS]</spark-opts>
+ <arg>[ARG-VALUE]</arg>
+ ...
+ <arg>[ARG-VALUE]</arg>
+ ...
+ </spark>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
+
+The `job-xml` element, if present, specifies a file containing configuration
+for the Spark job. Multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
+
+The `configuration` element, if present, contains configuration
+properties that are passed to the Spark job.
+
+The `master` element indicates the url of the Spark Master. Ex: `spark://host:port`, `mesos://host:port`, yarn-cluster, yarn-client,
+or local.
+
+The `mode` element if present indicates the mode of spark, where to run spark driver program. Ex: client,cluster. This is typically
+not required because you can specify it as part of `master` (i.e. master`yarn, mode`client is equivalent to master=yarn-client).
+A local `master` always runs in client mode.
+
+Depending on the `master` (and `mode`) entered, the Spark job will run differently as follows:
+
+ * local mode: everything runs here in the Launcher Job.
+ * yarn-client mode: the driver runs here in the Launcher Job and the executor in Yarn.
+ * yarn-cluster mode: the driver and executor run in Yarn.
+
+The `name` element indicates the name of the spark application.
+
+The `class` element if present, indicates the spark's application main class.
+
+The `jar` element indicates a comma separated list of jars or python files.
+
+The `spark-opts` element, if present, contains a list of Spark options that can be passed to Spark. Spark configuration
+options can be passed by specifying '--conf key=value' or other Spark CLI options.
+Values containing whitespaces can be enclosed by double quotes.
+
+Some examples of the `spark-opts` element:
+
+ * '--conf key=value'
+ * '--conf key1=value1 value2'
+ * '--conf key1="value1 value2"'
+ * '--conf key1=value1 key2="value2 value3"'
+ * '--conf key=value --verbose --properties-file user.properties'
+
+There are several ways to define properties that will be passed to Spark. They are processed in the following order:
+
+ * propagated from `oozie.service.SparkConfigurationService.spark.configurations`
+ * read from a localized `spark-defaults.conf` file
+ * read from a file defined in `spark-opts` via the `--properties-file`
+ * properties defined in `spark-opts` element
+
+(The latter takes precedence over the former.)
+The server propagated properties, the `spark-defaults.conf` and the user-defined properties file are merged together into a
+single properties file as Spark handles only one file in its `--properties-file` option.
+
+The `arg` element if present, contains arguments that can be passed to spark application.
+
+In case some property values are present both in `spark-defaults.conf` and as property key/value pairs generated by Oozie, the user
+configured values from `spark-defaults.conf` are prepended to the ones generated by Oozie, as part of the Spark arguments list.
+
+Following properties to prepend to Spark arguments:
+
+ * `spark.executor.extraClassPath`
+ * `spark.driver.extraClassPath`
+ * `spark.executor.extraJavaOptions`
+ * `spark.driver.extraJavaOptions`
+
+All the above elements can be parameterized (templatized) using EL
+expressions.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="myfirstsparkjob">
+ <spark xmlns="uri:oozie:spark-action:1.0">
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <master>local[*]</master>
+ <mode>client</mode>
+ <name>Spark Example</name>
+ <class>org.apache.spark.examples.mllib.JavaALS</class>
+ <jar>/lib/spark-examples_2.10-1.1.0.jar</jar>
+ <spark-opts>--executor-memory 20G --num-executors 50
+ --conf spark.executor.extraJavaOptions="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"</spark-opts>
+ <arg>inputpath=hdfs://localhost/input/file.txt</arg>
+ <arg>value=2</arg>
+ </spark>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+### Spark Action Logging
+
+Spark action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Spark.
+
+From Oozie web-console, from the Spark action pop up using the 'Console URL' link, it is possible
+to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
+
+### Spark on YARN
+
+To ensure that your Spark job shows up in the Spark History Server, make sure to specify these three Spark configuration properties
+either in `spark-opts` with `--conf` or from `oozie.service.SparkConfigurationService.spark.configurations` in oozie-site.xml.
+
+1. spark.yarn.historyServer.address=SPH-HOST:18088
+
+2. spark.eventLog.dir=`hdfs://NN:8020/user/spark/applicationHistory`
+
+3. spark.eventLog.enabled=true
+
+### PySpark with Spark Action
+
+To submit PySpark scripts with Spark Action, pyspark dependencies must be available in sharelib or in workflow's lib/ directory.
+For more information, please refer to [installation document.](AG_Install.html#Oozie_Share_Lib)
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ....
+ <action name="myfirstpysparkjob">
+ <spark xmlns="uri:oozie:spark-action:1.0">
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <master>yarn-cluster</master>
+ <name>Spark Example</name>
+ <jar>pi.py</jar>
+ <spark-opts>--executor-memory 20G --num-executors 50
+ --conf spark.executor.extraJavaOptions="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"</spark-opts>
+ <arg>100</arg>
+ </spark>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `jar` element indicates python file. Refer to the file by it's localized name, because only local files are allowed
+in PySpark. The py file should be in the lib/ folder next to the workflow.xml or added using the `file` element so that
+it's localized to the working directory with just its name.
+
+### Using Symlink in \<jar\>
+
+A symlink must be specified using [file](WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job) element. Then, you can use
+the symlink name in `jar` element.
+
+**Example:**
+
+Specifying relative path for symlink:
+
+Make sure that the file is within the application directory i.e. `oozie.wf.application.path` .
+
+```
+ <spark xmlns="uri:oozie:spark-action:1.0">
+ ...
+ <jar>py-spark-example-symlink.py</jar>
+ ...
+ ...
+ <file>py-spark.py#py-spark-example-symlink.py</file>
+ ...
+ </spark>
+```
+
+Specifying full path for symlink:
+
+```
+ <spark xmlns="uri:oozie:spark-action:1.0">
+ ...
+ <jar>spark-example-symlink.jar</jar>
+ ...
+ ...
+ <file>hdfs://localhost:8020/user/testjars/all-oozie-examples.jar#spark-example-symlink.jar</file>
+ ...
+ </spark>
+```
+
+
+
+## Appendix, Spark XML-Schema
+
+### AE.A Appendix A, Spark XML-Schema
+
+#### Spark Action Schema Version 1.0
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:spark="uri:oozie:spark-action:1.0" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:spark-action:1.0">
+.
+ <xs:include schemaLocation="oozie-common-1.0.xsd"/>
+.
+ <xs:element name="spark" type="spark:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:choice>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="launcher" type="spark:LAUNCHER" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Spark Action Schema Version 0.2
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:spark="uri:oozie:spark-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:spark-action:0.2">
+
+ <xs:element name="spark" type="spark:ACTION"/>
+
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="spark:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="spark:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+</xs:schema>
+```
+
+#### Spark Action Schema Version 0.1
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:spark="uri:oozie:spark-action:0.1" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:spark-action:0.1">
+
+ <xs:element name="spark" type="spark:ACTION"/>
+
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="spark:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="spark:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+</xs:schema>
+```
+[::Go back to Oozie Documentation Index::](index.html)
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_SqoopActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_SqoopActionExtension.md b/docs/src/site/markdown/DG_SqoopActionExtension.md
new file mode 100644
index 0000000..b186c5a
--- /dev/null
+++ b/docs/src/site/markdown/DG_SqoopActionExtension.md
@@ -0,0 +1,348 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Sqoop Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Sqoop Action
+
+**IMPORTANT:** The Sqoop action requires Apache Hadoop 1.x or 2.x.
+
+The `sqoop` action runs a Sqoop job.
+
+The workflow job will wait until the Sqoop job completes before
+continuing to the next action.
+
+To run the Sqoop job, you have to configure the `sqoop` action with the `resource-manager`, `name-node` and Sqoop `command`
+or `arg` elements as well as configuration.
+
+A `sqoop` action can be configured to create or delete HDFS directories
+before starting the Sqoop job.
+
+Sqoop configuration can be specified with a file, using the `job-xml`
+element, and inline, using the `configuration` elements.
+
+Oozie EL expressions can be used in the inline configuration. Property
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
+
+Note that YARN `yarn.resourcemanager.address` / `resource-manager` and HDFS `fs.default.name` / `name-node` properties must not
+be present in the inline configuration.
+
+As with Hadoop `map-reduce` jobs, it is possible to add files and
+archives in order to make them available to the Sqoop job. Refer to the
+[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
+section for more information about this feature.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <sqoop xmlns="uri:oozie:sqoop-action:1.0">
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <command>[SQOOP-COMMAND]</command>
+ <arg>[SQOOP-ARGUMENT]</arg>
+ ...
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ </sqoop>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
+
+The `job-xml` element, if present, specifies a file containing configuration
+for the Sqoop job. As of schema 0.3, multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
+
+The `configuration` element, if present, contains configuration
+properties that are passed to the Sqoop job.
+
+**Sqoop command**
+
+The Sqoop command can be specified either using the `command` element or multiple `arg`
+elements.
+
+When using the `command` element, Oozie will split the command on every space
+into multiple arguments.
+
+When using the `arg` elements, Oozie will pass each argument value as an argument to Sqoop.
+
+The `arg` variant should be used when there are spaces within a single argument.
+
+Consult the Sqoop documentation for a complete list of valid Sqoop commands.
+
+All the above elements can be parameterized (templatized) using EL
+expressions.
+
+**Examples:**
+
+Using the `command` element:
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="myfirsthivejob">
+ <sqoop xmlns="uri:oozie:sqoop-action:1.0">
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <command>import --connect jdbc:hsqldb:file:db.hsqldb --table TT --target-dir hdfs://localhost:8020/user/tucu/foo -m 1</command>
+ </sqoop>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The same Sqoop action using `arg` elements:
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="myfirstsqoopjob">
+ <sqoop xmlns="uri:oozie:sqoop-action:1.0">
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <arg>import</arg>
+ <arg>--connect</arg>
+ <arg>jdbc:hsqldb:file:db.hsqldb</arg>
+ <arg>--table</arg>
+ <arg>TT</arg>
+ <arg>--target-dir</arg>
+ <arg>hdfs://localhost:8020/user/tucu/foo</arg>
+ <arg>-m</arg>
+ <arg>1</arg>
+ </sqoop>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+NOTE: The `arg` elements syntax, while more verbose, allows to have spaces in a single argument, something useful when
+using free from queries.
+
+### Sqoop Action Counters
+
+The counters of the map-reduce job run by the Sqoop action are available to be used in the workflow via the
+[hadoop:counters() EL function](WorkflowFunctionalSpec.html#HadoopCountersEL).
+
+If the Sqoop action run an import all command, the `hadoop:counters()` EL will return the aggregated counters
+of all map-reduce jobs run by the Sqoop import all command.
+
+### Sqoop Action Logging
+
+Sqoop action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Sqoop.
+
+From Oozie web-console, from the Sqoop action pop up using the 'Console URL' link, it is possible
+to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
+
+The logging level of the Sqoop action can set in the Sqoop action configuration using the
+property `oozie.sqoop.log.level`. The default value is `INFO`.
+
+## Appendix, Sqoop XML-Schema
+
+### AE.A Appendix A, Sqoop XML-Schema
+
+#### Sqoop Action Schema Version 1.0
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:sqoop="uri:oozie:sqoop-action:1.0"
+ elementFormDefault="qualified"
+ targetNamespace="uri:oozie:sqoop-action:1.0">
+.
+ <xs:include schemaLocation="oozie-common-1.0.xsd"/>
+.
+ <xs:element name="sqoop" type="sqoop:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:choice>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="launcher" type="sqoop:LAUNCHER" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:choice>
+ <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/>
+ </xs:choice>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Sqoop Action Schema Version 0.3
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:sqoop="uri:oozie:sqoop-action:0.3" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:sqoop-action:0.3">
+
+ <xs:element name="sqoop" type="sqoop:ACTION"/>
+
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:choice>
+ <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/>
+ </xs:choice>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="sqoop:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="sqoop:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+</xs:schema>
+```
+
+#### Sqoop Action Schema Version 0.2
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:sqoop="uri:oozie:sqoop-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:sqoop-action:0.2">
+
+ <xs:element name="sqoop" type="sqoop:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:choice>
+ <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/>
+ </xs:choice>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="sqoop:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="sqoop:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_SshActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_SshActionExtension.md b/docs/src/site/markdown/DG_SshActionExtension.md
new file mode 100644
index 0000000..e53e1c3
--- /dev/null
+++ b/docs/src/site/markdown/DG_SshActionExtension.md
@@ -0,0 +1,161 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Ssh Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Ssh Action
+
+The `ssh` action starts a shell command on a remote machine as a remote secure shell in background. The workflow job
+will wait until the remote shell command completes before continuing to the next action.
+
+The shell command must be present in the remote machine and it must be available for execution via the command path.
+
+The shell command is executed in the home directory of the specified user in the remote host.
+
+The output (STDOUT) of the ssh job can be made available to the workflow job after the ssh job ends. This information
+could be used from within decision nodes. If the output of the ssh job is made available to the workflow job the shell
+command must follow the following requirements:
+
+ * The format of the output must be a valid Java Properties file.
+ * The size of the output must not exceed 2KB.
+
+Note: Ssh Action will fail if any output is written to standard error / output upon login (e.g. .bashrc of the remote
+user contains ls -a).
+
+Note: Ssh Action will fail if oozie fails to ssh connect to host for action status check
+(e.g., the host is under heavy load, or network is bad) after a configurable number (3 by default) of retries.
+The first retry will wait a configurable period of time ( 3 seconds by default) before check.
+The following retries will wait 2 times of previous wait time.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <ssh xmlns="uri:oozie:ssh-action:0.1">
+ <host>[USER]@[HOST]</host>
+ <command>[SHELL]</command>
+ <args>[ARGUMENTS]</args>
+ ...
+ <capture-output/>
+ </ssh>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `host` indicates the user and host where the shell will be executed.
+
+**IMPORTANT:** The `oozie.action.ssh.allow.user.at.host` property, in the `oozie-site.xml` configuration, indicates if
+an alternate user than the one submitting the job can be used for the ssh invocation. By default this property is set
+to `true`.
+
+The `command` element indicates the shell command to execute.
+
+The `args` element, if present, contains parameters to be passed to the shell command. If more than one `args` element
+is present they are concatenated in order. When an `args` element contains a space, even when quoted, it will be considered as
+separate arguments (i.e. "Hello World" becomes "Hello" and "World"). Starting with ssh schema 0.2, you can use the `arg` element
+(note that this is different than the `args` element) to specify arguments that have a space in them (i.e. "Hello World" is
+preserved as "Hello World"). You can use either `args` elements, `arg` elements, or neither; but not both in the same action.
+
+If the `capture-output` element is present, it indicates Oozie to capture output of the STDOUT of the ssh command
+execution. The ssh command output must be in Java Properties file format and it must not exceed 2KB. From within the
+workflow definition, the output of an ssh action node is accessible via the =String action:output(String node,
+String key)= function (Refer to section '4.2.6 Action EL Functions').
+
+The configuration of the `ssh` action can be parameterized (templatized) using EL expressions.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="myssjob">
+ <ssh xmlns="uri:oozie:ssh-action:0.1">
+ <host>foo@bar.com<host>
+ <command>uploaddata</command>
+ <args>jdbc:derby://bar.com:1527/myDB</args>
+ <args>hdfs://foobar.com:8020/usr/tucu/myData</args>
+ </ssh>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+In the above example, the `uploaddata` shell command is executed with two arguments, `jdbc:derby://foo.com:1527/myDB`
+and `hdfs://foobar.com:8020/usr/tucu/myData`.
+
+The `uploaddata` shell must be available in the remote host and available in the command path.
+
+The output of the command will be ignored because the `capture-output` element is not present.
+
+## Appendix, Ssh XML-Schema
+
+### AE.A Appendix A, Ssh XML-Schema
+
+#### Ssh Action Schema Version 0.2
+
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:ssh="uri:oozie:ssh-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:ssh-action:0.2">
+.
+ <xs:element name="ssh" type="ssh:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="host" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:choice>
+ <xs:element name="args" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:choice>
+ <xs:element name="capture-output" type="ssh:FLAG" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="FLAG"/>
+.
+</xs:schema>
+```
+
+#### Ssh Action Schema Version 0.1
+
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:ssh="uri:oozie:ssh-action:0.1" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:ssh-action:0.1">
+.
+ <xs:element name="ssh" type="ssh:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="host" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="args" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="capture-output" type="ssh:FLAG" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="FLAG"/>
+.
+</xs:schema>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_WorkflowReRun.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_WorkflowReRun.md b/docs/src/site/markdown/DG_WorkflowReRun.md
new file mode 100644
index 0000000..c128681
--- /dev/null
+++ b/docs/src/site/markdown/DG_WorkflowReRun.md
@@ -0,0 +1,42 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Workflow ReRrun
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+## Configs
+
+ * oozie.wf.application.path
+ * Only one of following two configurations is mandatory. Both should not be defined at the same time
+ * oozie.wf.rerun.skip.nodes
+ * oozie.wf.rerun.failnodes
+ * Skip nodes are comma separated list of action names. They can be any action nodes including decision node.
+ * The valid value of `oozie.wf.rerun.failnodes` is true or false.
+ * If secured hadoop version is used, the following two properties needs to be specified as well
+ * mapreduce.jobtracker.kerberos.principal
+ * dfs.namenode.kerberos.principal.
+ * Configurations can be passed as -D param.
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -rerun 14-20090525161321-oozie-joe -Doozie.wf.rerun.skip.nodes=<>
+```
+
+## Pre-Conditions
+
+ * Workflow with id wfId should exist.
+ * Workflow with id wfId should be in SUCCEEDED/KILLED/FAILED.
+ * If specified , nodes in the config oozie.wf.rerun.skip.nodes must be completed successfully.
+
+## ReRun
+
+ * Reloads the configs.
+ * If no configuration is passed, existing coordinator/workflow configuration will be used. If configuration is passed then, it will be merged with existing workflow configuration. Input configuration will take the precedence.
+ * Currently there is no way to remove an existing configuration but only override by passing a different value in the input configuration.
+ * Creates a new Workflow Instance with the same wfId.
+ * Deletes the actions that are not skipped from the DB and copies data from old Workflow Instance to new one for skipped actions.
+ * Action handler will skip the nodes given in the config with the same exit transition as before.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/ENG_MiniOozie.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/ENG_MiniOozie.md b/docs/src/site/markdown/ENG_MiniOozie.md
new file mode 100644
index 0000000..e793676
--- /dev/null
+++ b/docs/src/site/markdown/ENG_MiniOozie.md
@@ -0,0 +1,83 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Running MiniOozie Tests
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## System Requirements
+
+ * Unix box (tested on Mac OS X and Linux)
+ * Java JDK 1.8+
+ * Eclipse (tested on 3.5 and 3.6)
+ * [Maven 3.0.1+](http://maven.apache.org/)
+
+The Maven command (mvn) must be in the command path.
+
+## Installing Oozie Jars To Maven Cache
+
+Oozie source tree is at Apache SVN or Apache GIT. MiniOozie sample project is under Oozie source tree.
+
+The following command downloads Oozie trunk to local:
+
+
+```
+$ svn co https://svn.apache.org/repos/asf/incubator/oozie/trunk
+```
+
+OR
+
+
+```
+$ git clone git://github.com/apache/oozie.git
+```
+
+To run MiniOozie tests, the required jars like oozie-core, oozie-client, oozie-core-tests need to be
+available in remote maven repositories or local maven repository. The local maven cache for the above
+jars can be created and installed using the command:
+
+
+```
+$ mvn clean install -DskipTests -DtestJarSimple
+```
+
+The following properties should be specified to install correct jars for MiniOozie:
+
+ * -DskipTests : ignore executing Oozie unittests
+ * -DtestJarSimple= : build only required test classes to oozie-core-tests
+
+MiniOozie is a folder named 'minitest' under Oozie source tree. Two sample tests are included in the project.
+The following command to execute tests under MiniOozie:
+
+
+```
+$ cd minitest
+$ mvn clean test
+```
+
+## Create Tests Using MiniOozie
+
+MiniOozie is a JUnit test class to test Oozie applications such as workflow and coordinator. The test case
+needs to extend from MiniOozieTestCase and does the same as the example class 'WorkflowTest.java' to create Oozie
+workflow application properties and workflow XML. The example file is under Oozie source tree:
+
+ * `minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java`
+
+## IDE Setup
+
+Eclipse and IntelliJ can use directly MiniOozie Maven project files. MiniOozie project can be imported to
+Eclipse and IntelliJ as independent project.
+
+The test directories under MiniOozie are:
+
+ * `minitest/src/test/java` : as test-source directory
+ * `minitest/src/test/resources` : as test-resource directory
+
+
+Also asynchronous actions like FS action can be used / tested using `LocalOozie` / `OozieClient` API.
+Please see `fs-decision.xml` workflow example.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
[09/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki b/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
deleted file mode 100644
index d31d1aa..0000000
--- a/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
+++ /dev/null
@@ -1,4890 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Coordinator Specification
-
-The goal of this document is to define a coordinator engine system specialized in submitting workflows based on time and data triggers.
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Changelog
-
-**03/JUL/2013**
-
- * Appendix A, Added new coordinator schema 0.4, sla schema 0.2 and changed schemas ordering to newest first
-
-**07/JAN/2013**
-
- * 6.8 Added section on new EL functions for datasets defined with HCatalog
-
-**26/JUL/2012**
-
- * Appendix A, updated XML schema 0.4 to include `parameters` element
- * 6.5 Updated to mention about `parameters` element as of schema 0.4
-
-**23/NOV/2011:**
-
- * Update execution order typo
-
-**05/MAY/2011:**
-
- * Update coordinator schema 0.2
-
-**09/MAR/2011:**
-
- * Update coordinator status
-
-**02/DEC/2010:**
-
- * Update coordinator done-flag
-
-**26/AUG/2010:**
-
- * Update coordinator rerun
-
-**09/JUN/2010:**
-
- * Clean up unsupported functions
-
-**02/JUN/2010:**
-
- * Update all EL functions in CoordFunctionalSpec with "coord:" prefix
-
-**02/OCT/2009:**
-
- * Added Appendix A, Oozie Coordinator XML-Schema
- * Change #5.3., Datasets definition supports 'include' element
-
-**29/SEP/2009:**
-
- * Change #4.4.1, added `${coord:endOfDays(int n)}` EL function
- * Change #4.4.2, added `${coord:endOfMonths(int n)}` EL function
-
-**11/SEP/2009:**
-
- * Change #6.6.4. `${coord:tzOffset()}` EL function now returns offset in minutes. Added more explanation on behavior
- * Removed 'oozie' URL from action workflow invocation, per arch review feedback coord&wf run on the same instance
-
-**07/SEP/2009:**
-
- * Full rewrite of sections #4 and #7
- * Added sections #6.1.7, #6.6.2, #6.6.3 & #6.6.4
- * Rewording through the spec definitions
- * Updated all examples and syntax to latest changes
-
-**03/SEP/2009:**
-
- * Change #2. Definitions. Some rewording in the definitions
- * Change #6.6.4. Replaced `${coord:next(int n)}` with `${coord:version(int n)}` EL Function
- * Added #6.6.5. Dataset Instance Resolution for Instances Before the Initial Instance
-
-## 1. Coordinator Overview
-
-Users typically run map-reduce, hadoop-streaming, hdfs and/or Pig jobs on the grid. Multiple of these jobs can be combined to form a workflow job. [Oozie, Hadoop Workflow System](https://issues.apache.org/jira/browse/HADOOP-5303) defines a workflow system that runs such jobs.
-
-Commonly, workflow jobs are run based on regular time intervals and/or data availability. And, in some cases, they can be triggered by an external event.
-
-Expressing the condition(s) that trigger a workflow job can be modeled as a predicate that has to be satisfied. The workflow job is started after the predicate is satisfied. A predicate can reference to data, time and/or external events. In the future, the model can be extended to support additional event types.
-
-It is also necessary to connect workflow jobs that run regularly, but at different time intervals. The outputs of multiple subsequent runs of a workflow become the input to the next workflow. For example, the outputs of last 4 runs of a workflow that runs every 15 minutes become the input of another workflow that runs every 60 minutes. Chaining together these workflows result it is referred as a data application pipeline.
-
-The Oozie **Coordinator** system allows the user to define and execute recurrent and interdependent workflow jobs (data application pipelines).
-
-Real world data application pipelines have to account for reprocessing, late processing, catchup, partial processing, monitoring, notification and SLAs.
-
-This document defines the functional specification for the Oozie Coordinator system.
-
-## 2. Definitions
-
-**Actual time:** The actual time indicates the time when something actually happens.
-
-**Nominal time:** The nominal time specifies the time when something should happen. In theory the nominal time and the actual time should match, however, in practice due to delays the actual time may occur later than the nominal time.
-
-**Dataset:** Collection of data referred to by a logical name. A dataset normally has several instances of data and each
-one of them can be referred individually. Each dataset instance is represented by a unique set of URIs.
-
-**Synchronous Dataset:** Synchronous datasets instances are generated at fixed time intervals and there is a dataset
-instance associated with each time interval. Synchronous dataset instances are identified by their nominal time.
-For example, in the case of a HDFS based dataset, the nominal time would be somewhere in the file path of the
-dataset instance: `hdfs://foo:8020/usr/logs/2009/04/15/23/30`. In the case of HCatalog table partitions, the nominal time
-would be part of some partition values: `hcat://bar:8020/mydb/mytable/year=2009;month=04;dt=15;region=us`.
-
-**Coordinator Action:** A coordinator action is a workflow job that is started when a set of conditions are met (input dataset instances are available).
-
-**Coordinator Application:** A coordinator application defines the conditions under which coordinator actions should be created (the frequency) and when the actions can be started. The coordinator application also defines a start and an end time. Normally, coordinator applications are parameterized. A Coordinator application is written in XML.
-
-**Coordinator Job:** A coordinator job is an executable instance of a coordination definition. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
-
-**Data pipeline:** A data pipeline is a connected set of coordinator applications that consume and produce interdependent datasets.
-
-**Coordinator Definition Language:** The language used to describe datasets and coordinator applications.
-
-**Coordinator Engine:** A system that executes coordinator jobs.
-
-## 3. Expression Language for Parameterization
-
-Coordinator application definitions can be parameterized with variables, built-in constants and built-in functions.
-
-At execution time all the parameters are resolved into concrete values.
-
-The parameterization of workflow definitions it done using JSP Expression Language syntax from the [JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to support variables as parameters but also functions and complex expressions.
-
-EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.
-
-Refer to section #6.5 'Parameterization of Coordinator Applications' for more details.
-
-## 4. Datetime, Frequency and Time-Period Representation
-
-Oozie processes coordinator jobs in a fixed timezone with no DST (typically `UTC`), this timezone is referred as 'Oozie
-processing timezone'.
-
-The Oozie processing timezone is used to resolve coordinator jobs start/end times, job pause times and the initial-instance
-of datasets. Also, all coordinator dataset instance URI templates are resolved to a datetime in the Oozie processing
-time-zone.
-
-All the datetimes used in coordinator applications and job parameters to coordinator applications must be specified
-in the Oozie processing timezone. If Oozie processing timezone is `UTC`, the qualifier is **Z**. If Oozie processing
-time zone is other than `UTC`, the qualifier must be the GMT offset, `(+/-)####`.
-
-For example, a datetime in `UTC` is `2012-08-12T00:00Z`, the same datetime in `GMT+5:30` is `2012-08-12T05:30+0530`.
-
-For simplicity, the rest of this specification uses `UTC` datetimes.
-
-<a name="datetime"></a>
-### 4.1. Datetime
-
-If the Oozie processing timezone is `UTC`, all datetime values are always in
-[UTC](http://en.wikipedia.org/wiki/Coordinated_Universal_Time) down to a minute precision, 'YYYY-MM-DDTHH:mmZ'.
-
-For example `2009-08-10T13:10Z` is August 10th 2009 at 13:10 UTC.
-
-If the Oozie processing timezone is a GMT offset `GMT(+/-)####`, all datetime values are always in
-[ISO 8601](http://en.wikipedia.org/wiki/ISO_8601) in the corresponding GMT offset down to a minute precision,
-'YYYY-MM-DDTHH:mmGMT(+/-)####'.
-
-For example `2009-08-10T13:10+0530` is August 10th 2009 at 13:10 GMT+0530, India timezone.
-
-#### 4.1.1 End of the day in Datetime Values
-
-It is valid to express the end of day as a '24:00' hour (i.e. `2009-08-10T24:00Z`).
-
-However, for all calculations and display, Oozie resolves such dates as the zero hour of the following day
-(i.e. `2009-08-11T00:00Z`).
-
-### 4.2. Timezone Representation
-
-There is no widely accepted standard to identify timezones.
-
-Oozie Coordinator will understand the following timezone identifiers:
-
- * Generic NON-DST timezone identifier: `GMT[+/-]##:##` (i.e.: GMT+05:30)
- * UTC timezone identifier: `UTC` (i.e.: 2009-06-06T00:00Z)
- * ZoneInfo identifiers, with DST support, understood by Java JDK (about 600 IDs) (i.e.: America/Los_Angeles)
-
-Due to DST shift from PST to PDT, it is preferred that GMT, UTC or Region/City timezone notation is used in
-favor of direct three-letter ID (PST, PDT, BST, etc.). For example, America/Los_Angeles switches from PST to PDT
-at a DST shift. If used directly, PST will not handle DST shift when time is switched to PDT.
-
-Oozie Coordinator must provide a tool for developers to list all supported timezone identifiers.
-
-### 4.3. Timezones and Daylight-Saving
-
-While Oozie coordinator engine works in a fixed timezone with no DST (typically `UTC`), it provides DST support for coordinator applications.
-
-The baseline datetime for datasets and coordinator applications are expressed in UTC. The baseline datetime is the time of the first occurrence.
-
-Datasets and coordinator applications also contain a timezone indicator.
-
-The use of UTC as baseline enables a simple way of mix and matching datasets and coordinator applications that use a different timezone by just adding the timezone offset.
-
-The timezone indicator enables Oozie coordinator engine to properly compute frequencies that are daylight-saving sensitive. For example: a daily frequency can be 23, 24 or 25 hours for timezones that observe daylight-saving. Weekly and monthly frequencies are also affected by this as the number of hours in the day may change.
-
-Section #7 'Handling Timezones and Daylight Saving Time' explains how coordinator applications can be written to handle timezones and daylight-saving-time properly.
-
-### 4.4. Frequency and Time-Period Representation
-
-Frequency is used to capture the periodic intervals at which datasets that are produced, and coordinator applications are scheduled to run.
-
-This time periods representation is also used to specify non-recurrent time-periods, for example a timeout interval.
-
-For datasets and coordinator applications the frequency time-period is applied `N` times to the baseline datetime to compute recurrent times.
-
-Frequency is always expressed in minutes.
-
-Because the number of minutes in day may vary for timezones that observe daylight saving time, constants cannot be use to express frequencies greater than a day for datasets and coordinator applications for such timezones. For such uses cases, Oozie coordinator provides 2 EL functions, `${coord:days(int n)}` and `${coord:months(int n)}`.
-
-Frequencies can be expressed using EL constants and EL functions that evaluate to an positive integer number.
-
-Coordinator Frequencies can also be expressed using cron syntax.
-
-**<font color="#008000"> Examples: </font>**
-
-| **EL Constant** | **Value** | **Example** |
-| --- | --- | --- |
-| `${coord:minutes(int n)}` | _n_ | `${coord:minutes(45)}` --> `45` |
-| `${coord:hours(int n)}` | _n * 60_ | `${coord:hours(3)}` --> `180` |
-| `${coord:days(int n)}` | _variable_ | `${coord:days(2)}` --> minutes in 2 full days from the current date |
-| `${coord:months(int n)}` | _variable_ | `${coord:months(1)}` --> minutes in a 1 full month from the current date |
-| `${cron syntax}` | _variable_ | `${0,10 15 * * 2-6}` --> a job that runs every weekday at 3:00pm and 3:10pm UTC time|
-
-Note that, though `${coord:days(int n)}` and `${coord:months(int n)}` EL functions are used to calculate minutes precisely including
-variations due to daylight saving time for Frequency representation, when specified for coordinator timeout interval, one day is
-calculated as 24 hours and one month is calculated as 30 days for simplicity.
-
-#### 4.4.1. The coord:days(int n) and coord:endOfDays(int n) EL functions
-
-The `${coord:days(int n)}` and `${coord:endOfDays(int n)}` EL functions should be used to handle day based frequencies.
-
-Constant values should not be used to indicate a day based frequency (every 1 day, every 1 week, etc) because the number of hours in
-every day is not always the same for timezones that observe daylight-saving time.
-
-It is a good practice to use always these EL functions instead of using a constant expression (i.e. `24 * 60`) even if the timezone
-for which the application is being written for does not support daylight saving time. This makes application foolproof to country
-legislation changes and also makes applications portable across timezones.
-
-##### 4.4.1.1. The coord:days(int n) EL function
-
-The `${coord:days(int n)}` EL function returns the number of minutes for 'n' complete days starting with the day of the specified nominal time for which the computation is being done.
-
-The `${coord:days(int n)}` EL function includes **all** the minutes of the current day, regardless of the time of the day of the current nominal time.
-
-**<font color="#008000"> Examples: </font>**
-
-| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
-| --- | --- | --- | --- | --- | --- |
-| `2009-01-01T08:00Z` | `UTC` | `${coord:days(1)}` | 1440 | `2009-01-01T08:00Z` | total minutes on 2009JAN01 UTC time |
-| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1440 | `2009-01-01T08:00Z` | total minutes in 2009JAN01 PST8PDT time |
-| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:days(2)}` | 2880 | `2009-01-01T08:00Z` | total minutes in 2009JAN01 and 2009JAN02 PST8PDT time |
-| |||||
-| `2009-03-08T08:00Z` | `UTC` | `${coord:days(1)}` | 1440 | `2009-03-08T08:00Z` | total minutes on 2009MAR08 UTC time |
-| `2009-03-08T08:00Z` | `Europe/London` | `${coord:days(1)}` | 1440 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 BST1BDT time |
-| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1380 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
-| `2009-03-08T08:00Z` | `UTC` | `${coord:days(2)}` | 2880 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 and 2009MAR09 UTC time |
-| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:days(2)}` | 2820 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 and 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
-| `2009-03-09T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1440 | `2009-03-09T07:00Z` | total minutes in 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST ON, frequency tick is earlier in UTC) |
-
-For all these examples, the first occurrence of the frequency will be at `08:00Z` (UTC time).
-
-##### 4.4.1.2. The coord:endOfDays(int n) EL function
-
-The `${coord:endOfDays(int n)}` EL function is identical to the `${coord:days(int n)}` except that it shifts the first occurrence to the end of the day for the specified timezone before computing the interval in minutes.
-
-**<font color="#008000"> Examples: </font>**
-
-| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
-| --- | --- | --- | --- | --- | --- |
-| `2009-01-01T08:00Z` | `UTC` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T00:00Z` | first occurrence in 2009JAN02 00:00 UTC time, <br/> first occurrence shifted to the end of the UTC day |
-| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
-| `2009-01-01T08:01Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
-| `2009-01-01T18:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
-| |||||
-| `2009-03-07T09:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1380 | `2009-03-08T08:00Z` | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
-| `2009-03-08T07:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-03-08T08:00Z` | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
-| `2009-03-09T07:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-03-10T07:00Z` | first occurrence in 2009MAR10 07:00 UTC time <br/> (2009MAR08 is DST switch in the US), <br/> first occurrence shifted to the end of the PST8PDT day |
-
-
-```
-<coordinator-app name="hello-coord" frequency="${coord:days(1)}"
- start="2009-01-02T08:00Z" end="2009-01-04T08:00Z" timezone="America/Los_Angeles"
- xmlns="uri:oozie:coordinator:0.5">
- <controls>
- <timeout>10</timeout>
- <concurrency>${concurrency_level}</concurrency>
- <execution>${execution_order}</execution>
- <throttle>${materialization_throttle}</throttle>
- </controls>
-
- <datasets>
- <dataset name="din" frequency="${coord:endOfDays(1)}"
- initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
- <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
- </dataset>
- <dataset name="dout" frequency="${coord:minutes(30)}"
- initial-instance="2009-01-02T08:00Z" timezone="UTC">
- <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
- </dataset>
- </datasets>
-
- <input-events>
- <data-in name="input" dataset="din">
- <instance>${coord:current(0)}</instance>
- </data-in>
- </input-events>
-
- <output-events>
- <data-out name="output" dataset="dout">
- <instance>${coord:current(1)}</instance>
- </data-out>
- </output-events>
-
- <action>
- <workflow>
- <app-path>${wf_app_path}</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>${coord:dataIn('input')}</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>${coord:dataOut('output')}</value>
- </property>
- </configuration>
- </workflow>
- </action>
- </coordinator-app>
-```
-
-#### 4.4.2. The coord:months(int n) and coord:endOfMonths(int n) EL functions
-
-The `${coord:months(int n)}` and `${coord:endOfMonths(int n)}` EL functions should be used to handle month based frequencies.
-
-Constant values cannot be used to indicate a month based frequency because the number of days in a month changes month to month and on leap years; plus the number of hours in every day of the month are not always the same for timezones that observe daylight-saving time.
-
-##### 4.4.2.1. The coord:months(int n) EL function
-
-The `${coord:months(int n)}` EL function returns the number of minutes for 'n' complete months starting with the month of the current nominal time for which the computation is being done.
-
-The `${coord:months(int n)}` EL function includes **all** the minutes of the current month, regardless of the day of the month of the current nominal time.
-
-**<font color="#008000"> Examples: </font>**
-
-| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
-| --- | --- | --- | --- | --- | --- |
-| `2009-01-01T08:00Z` | `UTC` | `${coord:months(1)}` | 44640 | `2009-01-01T08:00Z` |total minutes for 2009JAN UTC time |
-| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:months(1)}` | 44640 | `2009-01-01T08:00Z` | total minutes in 2009JAN PST8PDT time |
-| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:months(2)}` | 84960 | `2009-01-01T08:00Z` | total minutes in 2009JAN and 2009FEB PST8PDT time |
-| |||||
-| `2009-03-08T08:00Z` | `UTC` | `${coord:months(1)}` | 44640 | `2009-03-08T08:00Z` | total minutes on 2009MAR UTC time |
-| `2009-03-08T08:00Z` | `Europe/London` | `${coord:months(1)}` | 44580 | `2009-03-08T08:00Z` | total minutes in 2009MAR BST1BDT time <br/> (2009MAR29 is DST switch in Europe) |
-| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:months(1)}` | 44580 | `2009-03-08T08:00Z` | total minutes in 2009MAR PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
-| `2009-03-08T08:00Z` | `UTC` | `${coord:months(2)}` | 87840 | `2009-03-08T08:00Z` | total minutes in 2009MAR and 2009APR UTC time |
-| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:months(2)}` | 87780 | `2009-03-08T08:00Z` | total minutes in 2009MAR and 2009APR PST8PDT time <br/> (2009MAR08 is DST switch in US) |
-
-##### 4.4.2.2. The coord:endOfMonths(int n) EL function
-
-The `${coord:endOfMonths(int n)}` EL function is identical to the `${coord:months(int n)}` except that it shifts the first occurrence to the end of the month for the specified timezone before computing the interval in minutes.
-
-**<font color="#008000"> Examples: </font>**
-
-| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
-| --- | --- | --- | --- | --- | --- |
-| `2009-01-01T00:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
-| `2009-01-01T08:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
-| `2009-01-31T08:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
-| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T08:00Z` | first occurrence in 2009FEB 08:00 UTC time |
-| `2009-02-02T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 44580 | `2009-03-01T08:00Z` | first occurrence in 2009MAR 08:00 UTC time |
-| `2009-02-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 44580 | `2009-03-01T08:00Z` | first occurrence in 2009MAR 08:00 UTC time |
-
-
-```
-<coordinator-app name="hello-coord" frequency="${coord:months(1)}"
- start="2009-01-02T08:00Z" end="2009-04-02T08:00Z" timezone="America/Los_Angeles"
- xmlns="uri:oozie:coordinator:0.5">
- <controls>
- <timeout>10</timeout>
- <concurrency>${concurrency_level}</concurrency>
- <execution>${execution_order}</execution>
- <throttle>${materialization_throttle}</throttle>
- </controls>
-
- <datasets>
- <dataset name="din" frequency="${coord:endOfMonths(1)}"
- initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
- <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
- </dataset>
- <dataset name="dout" frequency="${coord:minutes(30)}"
- initial-instance="2009-01-02T08:00Z" timezone="UTC">
- <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
- </dataset>
- </datasets>
-
- <input-events>
- <data-in name="input" dataset="din">
- <instance>${coord:current(0)}</instance>
- </data-in>
- </input-events>
-
- <output-events>
- <data-out name="output" dataset="dout">
- <instance>${coord:current(1)}</instance>
- </data-out>
- </output-events>
-
- <action>
- <workflow>
- <app-path>${wf_app_path}</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>${coord:dataIn('input')}</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>${coord:dataOut('output')}</value>
- </property>
- </configuration>
- </workflow>
- </action>
- </coordinator-app>
-```
-
-#### 4.4.3. The coord:endOfWeeks(int n) EL function
-
-The `${coord:endOfWeeks(int n)}` EL function shifts the first occurrence to the start of the week for the specified
-timezone before computing the interval in minutes. The start of the week depends on the Java's implementation of
-[Calendar.getFirstDayOfWeek()](https://docs.oracle.com/javase/8/docs/api/java/util/Calendar.html#getFirstDayOfWeek--)
- i.e. first day of the week is SUNDAY in the U.S., MONDAY in France.
-
-**<font color="#008000"> Examples: </font>**
-
-| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
-| --- | --- | --- | --- | --- | --- |
-| `2017-01-04T00:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T00:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
-| `2017-01-04T08:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
-| `2017-01-06T08:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
-| `2017-01-04T08:00Z` | `America/Los_Angeles` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence in 2017JAN08 08:00 UTC time |
-| `2017-01-06T08:00Z` | `America/Los_Angeles` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence in 2017JAN08 08:00 UTC time |
-
-
-```
-<coordinator-app name="hello-coord" frequency="${coord:endOfWeeks(1)}"
- start="2017-01-04T08:00Z" end="2017-12-31T08:00Z" timezone="America/Los_Angeles"
- xmlns="uri:oozie:coordinator:0.5">
- <controls>
- <timeout>10</timeout>
- <concurrency>${concurrency_level}</concurrency>
- <execution>${execution_order}</execution>
- <throttle>${materialization_throttle}</throttle>
- </controls>
-
- <datasets>
- <dataset name="din" frequency="${coord:endOfWeeks(1)}"
- initial-instance="2017-01-01T08:00Z" timezone="America/Los_Angeles">
- <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}</uri-template>
- </dataset>
- <dataset name="dout" frequency="${coord:endOfWeeks(1)}"
- initial-instance="2017-01-01T08:00Z" timezone="UTC">
- <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}</uri-template>
- </dataset>
- </datasets>
-
- <input-events>
- <data-in name="input" dataset="din">
- <instance>${coord:current(0)}</instance>
- </data-in>
- </input-events>
-
- <output-events>
- <data-out name="output" dataset="dout">
- <instance>${coord:current(1)}</instance>
- </data-out>
- </output-events>
-
- <action>
- <workflow>
- <app-path>${wf_app_path}</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>${coord:dataIn('input')}</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>${coord:dataOut('output')}</value>
- </property>
- </configuration>
- </workflow>
- </action>
- </coordinator-app>
-```
-
-#### 4.4.4. Cron syntax in coordinator frequency
-
-Oozie has historically allowed only very basic forms of scheduling: You could choose
-to run jobs separated by a certain number of minutes, hours, days or weeks. That's
-all. This works fine for processes that need to run continuously all year like building
-a search index to power an online website.
-
-However, there are a lot of cases that don't fit this model. For example, maybe you
-want to export data to a reporting system used during the day by business analysts.
-It would be wasteful to run the jobs when no analyst is going to take advantage of
-the new information, such as overnight. You might want a policy that says "only run
-these jobs on weekdays between 6AM and 8PM". Previous versions of Oozie didn't support
-this kind of complex scheduling policy without requiring multiple identical coordinators.
-Cron-scheduling improves the user experience in this area, allowing for a lot more flexibility.
-
-Cron is a standard time-based job scheduling mechanism in unix-like operating system. It is used extensively by system
-administrators to setup jobs and maintain software environment. Cron syntax generally consists of five fields, minutes,
-hours, date of month, month, and day of week respectively although multiple variations do exist.
-
-
-```
-<coordinator-app name="cron-coord" frequency="0/10 1/2 ** ** *" start="${start}" end="${end}" timezone="UTC"
- xmlns="uri:oozie:coordinator:0.2">
- <action>
- <workflow>
- <app-path>${workflowAppUri}</app-path>
- <configuration>
- <property>
- <name>jobTracker</name>
- <value>${jobTracker}</value>
- </property>
- <property>
- <name>nameNode</name>
- <value>${nameNode}</value>
- </property>
- <property>
- <name>queueName</name>
- <value>${queueName}</value>
- </property>
- </configuration>
- </workflow>
- </action>
-</coordinator-app>
-```
-
-Cron expressions are comprised of 5 required fields. The fields respectively are described as follows:
-
-| **Field name** | **Allowed Values** | **Allowed Special Characters** |
-| --- | --- | --- |
-| `Minutes` | `0-59` | , - * / |
-| `Hours` | `0-23` | , - * / |
-| `Day-of-month` | `1-31` | , - * ? / L W |
-| `Month` | `1-12 or JAN-DEC` | , - * / |
-| `Day-of-Week` | `1-7 or SUN-SAT` | , - * ? / L #|
-
-The '**' character is used to specify all values. For example, "**" in the minute field means "every minute".
-
-The '?' character is allowed for the day-of-month and day-of-week fields. It is used to specify 'no specific value'.
-This is useful when you need to specify something in one of the two fields, but not the other.
-
-The '-' character is used to specify ranges For example "10-12" in the hour field means "the hours 10, 11 and 12".
-
-The ',' character is used to specify additional values. For example "MON,WED,FRI" in the day-of-week field means
-"the days Monday, Wednesday, and Friday".
-
-The '/' character is used to specify increments. For example "0/15" in the minutes field means "the minutes 0, 15, 30, and 45".
-And "5/15" in the minutes field means "the minutes 5, 20, 35, and 50". Specifying '*' before the '/' is equivalent to
-specifying 0 is the value to start with.
-Essentially, for each field in the expression, there is a set of numbers that can be turned on or off.
-For minutes, the numbers range from 0 to 59. For hours 0 to 23, for days of the month 0 to 31, and for months 1 to 12.
-The "/" character simply helps you turn on every "nth" value in the given set. Thus "7/6" in the month field only turns on
-month "7", it does NOT mean every 6th month, please note that subtlety.
-
-The 'L' character is allowed for the day-of-month and day-of-week fields. This character is short-hand for "last",
-but it has different meaning in each of the two fields.
-For example, the value "L" in the day-of-month field means "the last day of the month" - day 31 for January, day 28 for
-February on non-leap years.
-If used in the day-of-week field by itself, it simply means "7" or "SAT".
-But if used in the day-of-week field after another value, it means "the last xxx day of the month" - for example
-"6L" means "the last Friday of the month".
-You can also specify an offset from the last day of the month, such as "L-3" which would mean the third-to-last day of the
-calendar month.
-When using the 'L' option, it is important not to specify lists, or ranges of values, as you'll get confusing/unexpected results.
-
-The 'W' character is allowed for the day-of-month field. This character is used to specify the weekday (Monday-Friday)
-nearest the given day.
-As an example, if you were to specify "15W" as the value for the day-of-month field, the meaning is:
-"the nearest weekday to the 15th of the month". So if the 15th is a Saturday, the trigger will fire on Friday the 14th.
-If the 15th is a Sunday, the trigger will fire on Monday the 16th. If the 15th is a Tuesday, then it will fire on Tuesday the 15th.
-However if you specify "1W" as the value for day-of-month, and the 1st is a Saturday, the trigger will fire on Monday the 3rd,
-as it will not 'jump' over the boundary of a month's days.
-The 'W' character can only be specified when the day-of-month is a single day, not a range or list of days.
-
-The 'L' and 'W' characters can also be combined for the day-of-month expression to yield 'LW', which translates to
-"last weekday of the month".
-
-The '#' character is allowed for the day-of-week field. This character is used to specify "the nth" XXX day of the month.
-For example, the value of "6#3" in the day-of-week field means the third Friday of the month (day 6 = Friday and "#3" =
-the 3rd one in the month).
-Other examples: "2#1" = the first Monday of the month and "4#5" = the fifth Wednesday of the month.
-Note that if you specify "#5" and there is not 5 of the given day-of-week in the month, then no firing will occur that month.
-If the '#' character is used, there can only be one expression in the day-of-week field ("3#1,6#3" is not valid,
-since there are two expressions).
-
-The legal characters and the names of months and days of the week are not case sensitive.
-
-If a user specifies an invalid cron syntax to run something on Feb, 30th for example: "0 10 30 2 *", the coordinator job
-will not be created and an invalid coordinator frequency parse exception will be thrown.
-
-If a user has a coordinator job that materializes no action during run time, for example: frequency of "0 10 ** ** *" with
-start time of 2013-10-18T21:00Z and end time of 2013-10-18T22:00Z, the coordinator job submission will be rejected and
-an invalid coordinator attribute exception will be thrown.
-
-**<font color="#008000"> Examples: </font>**
-
-| **Cron Expression** | **Meaning** |
-| --- | --- |
-| 10 9 ** ** * | Runs everyday at 9:10am |
-| 10,30,45 9 ** ** * | Runs everyday at 9:10am, 9:30am, and 9:45am |
-| `0 * 30 JAN 2-6` | Runs at 0 minute of every hour on weekdays and 30th of January |
-| `0/20 9-17 ** ** 2-5` | Runs every Mon, Tue, Wed, and Thurs at minutes 0, 20, 40 from 9am to 5pm |
-| 1 2 L-3 ** ** | Runs every third-to-last day of month at 2:01am |
-| `1 2 6W 3 ?` | Runs on the nearest weekday to March, 6th every year at 2:01am |
-| `1 2 * 3 3#2` | Runs every second Tuesday of March at 2:01am every year |
-| `0 10,13 ** ** MON-FRI` | Runs every weekday at 10am and 1pm |
-
-
-NOTES:
-
- Cron expression and syntax in Oozie are inspired by Quartz:http://quartz-scheduler.org/api/2.0.0/org/quartz/CronExpression.html.
- However, there is a major difference between Quartz cron and Oozie cron in which Oozie cron doesn't have "Seconds" field
- since everything in Oozie functions at the minute granularity at most. Everything related to Oozie cron syntax should be based
- on the documentation in the Oozie documentation.
-
- Cron expression uses oozie server processing timezone. Since default oozie processing timezone is UTC, if you want to
- run a job on every weekday at 10am in Tokyo, Japan(UTC + 9), your cron expression should be "0 1 * * 2-6" instead of
- the "0 10 * * 2-6" which you might expect.
-
- Overflowing ranges is supported but strongly discouraged - that is, having a larger number on the left hand side than the right.
- You might do 22-2 to catch 10 o'clock at night until 2 o'clock in the morning, or you might have NOV-FEB.
- It is very important to note that overuse of overflowing ranges creates ranges that don't make sense and
- no effort has been made to determine which interpretation CronExpression chooses.
- An example would be "0 14-6 ? * FRI-MON".
-
-## 5. Dataset
-
-A dataset is a collection of data referred to by a logical name.
-
-A dataset instance is a particular occurrence of a dataset and it is represented by a unique set of URIs. A dataset instance can be individually referred. Dataset instances for datasets containing ranges are identified by a set of unique URIs, otherwise a dataset instance is identified by a single unique URI.
-
-Datasets are typically defined in some central place for a business domain and can be accessed by the coordinator. Because of this, they can be defined once and used many times.
-
-A dataset is a synchronous (produced at regular time intervals, it has an expected frequency) input.
-
-A dataset instance is considered to be immutable while it is being consumed by coordinator jobs.
-
-### 5.1. Synchronous Datasets
-
-Instances of synchronous datasets are produced at regular time intervals, at an expected frequency. They are also referred to as "clocked datasets".
-
-Synchronous dataset instances are identified by their nominal creation time. The nominal creation time is normally specified in the dataset instance URI.
-
-A synchronous dataset definition contains the following information:
-
- * **<font color="#0000ff"> name: </font>** The dataset name. It must be a valid Java identifier.
- * **<font color="#0000ff"> frequency: </font>*** It represents the rate, in minutes at which data is _periodically_ created. The granularity is in minutes and can be expressed using EL expressions, for example: ${5 ** HOUR}.
- * **<font color="#0000ff"> initial-instance: </font>** The UTC datetime of the initial instance of the dataset. The initial-instance also provides the baseline datetime to compute instances of the dataset using multiples of the frequency.
- * **<font color="#0000ff"> timezone:</font>** The timezone of the dataset.
- * **<font color="#0000ff"> uri-template:</font>** The URI template that identifies the dataset and can be resolved into concrete URIs to identify a particular dataset instance. The URI template is constructed using:
- * **<font color="#0000ff"> constants </font>** See the allowable EL Time Constants below. Ex: ${YEAR}/${MONTH}.
- * **<font color="#0000ff"> variables </font>** Variables must be resolved at the time a coordinator job is submitted to the coordinator engine. They are normally provided a job parameters (configuration properties). Ex: ${market}/${language}
- * **<font color="#0000ff"> done-flag:</font>** This flag denotes when a dataset instance is ready to be consumed.
- * If the done-flag is omitted the coordinator will wait for the presence of a _SUCCESS file in the directory (Note: MapReduce jobs create this on successful completion automatically).
- * If the done-flag is present but empty, then the existence of the directory itself indicates that the dataset is ready.
- * If the done-flag is present but non-empty, Oozie will check for the presence of the named file within the directory, and will be considered ready (done) when the file exists.
-
-The following EL constants can be used within synchronous dataset URI templates:
-
-| **EL Constant** | **Resulting Format** | **Comments** |
-| --- | --- | --- |
-| `YEAR` | _YYYY_ | 4 digits representing the year |
-| `MONTH` | _MM_ | 2 digits representing the month of the year, January = 1 |
-| `DAY` | _DD_ | 2 digits representing the day of the month |
-| `HOUR` | _HH_ | 2 digits representing the hour of the day, in 24 hour format, 0 - 23 |
-| `MINUTE` | _mm_ | 2 digits representing the minute of the hour, 0 - 59 |
-
-**<font color="#800080">Syntax: </font>**
-
-
-```
- <dataset name="[NAME]" frequency="[FREQUENCY]"
- initial-instance="[DATETIME]" timezone="[TIMEZONE]">
- <uri-template>[URI TEMPLATE]</uri-template>
- <done-flag>[FILE NAME]</done-flag>
- </dataset>
-```
-
-IMPORTANT: The values of the EL constants in the dataset URIs (in HDFS) are expected in UTC. Oozie Coordinator takes care of the timezone conversion when performing calculations.
-
-**<font color="#008000"> Examples: </font>**
-
-1. **A dataset produced once every day at 00:15 PST8PDT and done-flag is set to empty:**
-
-
- ```
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/app/logs/${market}/${YEAR}${MONTH}/${DAY}/data
- </uri-template>
- <done-flag></done-flag>
- </dataset>
- ```
-
-
- The dataset would resolve to the following URIs and Coordinator looks for the existence of the directory itself:
-
-
- ```
- [market] will be replaced with user given property.
-
- hdfs://foo:8020/usr/app/[market]/2009/02/15/data
- hdfs://foo:8020/usr/app/[market]/2009/02/16/data
- hdfs://foo:8020/usr/app/[market]/2009/02/17/data
- ...
- ```
-
-
-2. **A dataset available on the 10th of each month and done-flag is default '_SUCCESS':**
-
-
- ```
- <dataset name="stats" frequency="${coord:months(1)}"
- initial-instance="2009-01-10T10:00Z" timezone="America/Los_Angeles">
- <uri-template>hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data</uri-template>
- </dataset>
- ```
-
- The dataset would resolve to the following URIs:
-
-
- ```
- hdfs://foo:8020/usr/app/stats/2009/01/data
- hdfs://foo:8020/usr/app/stats/2009/02/data
- hdfs://foo:8020/usr/app/stats/2009/03/data
- ...
- ```
-
- The dataset instances are not ready until '_SUCCESS' exists in each path:
-
-
- ```
- hdfs://foo:8020/usr/app/stats/2009/01/data/_SUCCESS
- hdfs://foo:8020/usr/app/stats/2009/02/data/_SUCCESS
- hdfs://foo:8020/usr/app/stats/2009/03/data/_SUCCESS
- ...
- ```
-
-
-3. **A dataset available at the end of every quarter and done-flag is 'trigger.dat':**
-
-
- ```
- <dataset name="stats" frequency="${coord:months(3)}"
- initial-instance="2009-01-31T20:00Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data
- </uri-template>
- <done-flag>trigger.dat</done-flag>
- </dataset>
- ```
-
- The dataset would resolve to the following URIs:
-
-
- ```
- hdfs://foo:8020/usr/app/stats/2009/01/data
- hdfs://foo:8020/usr/app/stats/2009/04/data
- hdfs://foo:8020/usr/app/stats/2009/07/data
- ...
- ```
-
- The dataset instances are not ready until 'trigger.dat' exists in each path:
-
-
- ```
- hdfs://foo:8020/usr/app/stats/2009/01/data/trigger.dat
- hdfs://foo:8020/usr/app/stats/2009/04/data/trigger.dat
- hdfs://foo:8020/usr/app/stats/2009/07/data/trigger.dat
- ...
- ```
-
-
-4. **Normally the URI template of a dataset has a precision similar to the frequency:**
-
-
- ```
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/data
- </uri-template>
- </dataset>
- ```
-
- The dataset would resolve to the following URIs:
-
-
- ```
- hdfs://foo:8020/usr/app/logs/2009/01/01/data
- hdfs://foo:8020/usr/app/logs/2009/01/02/data
- hdfs://foo:8020/usr/app/logs/2009/01/03/data
- ...
- ```
-
-5. **However, if the URI template has a finer precision than the dataset frequency:**
-
-
- ```
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}/data
- </uri-template>
- </dataset>
- ```
-
- The dataset resolves to the following URIs with fixed values for the finer precision template variables:
-
-
- ```
- hdfs://foo:8020/usr/app/logs/2009/01/01/10/30/data
- hdfs://foo:8020/usr/app/logs/2009/01/02/10/30/data
- hdfs://foo:8020/usr/app/logs/2009/01/03/10/30/data
- ...
- ```
-
-### 5.2. Dataset URI-Template types
-
-Each dataset URI could be a HDFS path URI denoting a HDFS directory: `hdfs://foo:8020/usr/logs/20090415` or a
-HCatalog partition URI identifying a set of table partitions: `hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US`.
-
-HCatalog enables table and storage management for Pig, Hive and MapReduce. The format to specify a HCatalog table partition URI is
-`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...`
-
-For example,
-
-```
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
- <uri-template>
- hcat://myhcatmetastore:9080/database1/table1/myfirstpartitionkey=myfirstvalue;mysecondpartitionkey=mysecondvalue
- </uri-template>
- <done-flag></done-flag>
- </dataset>
-```
-
-### 5.3. Asynchronous Datasets
- * TBD
-
-### 5.4. Dataset Definitions
-
-Dataset definitions are grouped in XML files.
-**IMPORTANT:** Please note that if an XML namespace version is specified for the coordinator-app element in the coordinator.xml file, no namespace needs to be defined separately for the datasets element (even if the dataset is defined in a separate file). Specifying it at multiple places might result in xml errors while submitting the coordinator job.
-
-**<font color="#800080">Syntax: </font>**
-
-
-```
- <!-- Synchronous datasets -->
-<datasets>
- <include>[SHARED_DATASETS]</include>
- ...
- <dataset name="[NAME]" frequency="[FREQUENCY]"
- initial-instance="[DATETIME]" timezone="[TIMEZONE]">
- <uri-template>[URI TEMPLATE]</uri-template>
- </dataset>
- ...
-</datasets>
-```
-
-**<font color="#008000"> Example: </font>**
-
-
-```
-<datasets>
-.
- <include>hdfs://foo:8020/app/dataset-definitions/globallogs.xml</include>
-.
- <dataset name="logs" frequency="${coord:hours(12)}"
- initial-instance="2009-02-15T08:15Z" timezone="Americas/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/app/logs/${market}/${YEAR}${MONTH}/${DAY}/${HOUR}/${MINUTE}/data
- </uri-template>
- </dataset>
-.
- <dataset name="stats" frequency="${coord:months(1)}"
- initial-instance="2009-01-10T10:00Z" timezone="Americas/Los_Angeles">
- <uri-template>hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data</uri-template>
- </dataset>
-.
-</datasets>
-```
-
-## 6. Coordinator Application
-
-### 6.1. Concepts
-
-#### 6.1.1. Coordinator Application
-
-A coordinator application is a program that triggers actions (commonly workflow jobs) when a set of conditions are met. Conditions can be a time frequency, the availability of new dataset instances or other external events.
-
-Types of coordinator applications:
-
- * **Synchronous:** Its coordinator actions are created at specified time intervals.
-
-Coordinator applications are normally parameterized.
-
-#### 6.1.2. Coordinator Job
-
-To create a coordinator job, a job configuration that resolves all coordinator application parameters must be provided to the coordinator engine.
-
-A coordinator job is a running instance of a coordinator application running from a start time to an end time. The start
-time must be earlier than the end time.
-
-At any time, a coordinator job is in one of the following status: **PREP, RUNNING, RUNNINGWITHERROR, PREPSUSPENDED, SUSPENDED, SUSPENDEDWITHERROR, PREPPAUSED, PAUSED, PAUSEDWITHERROR, SUCCEEDED, DONEWITHERROR, KILLED, FAILED**.
-
-Valid coordinator job status transitions are:
-
- * **PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED**
- * **RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED**
- * **RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | KILLED | FAILED**
- * **PREPSUSPENDED --> PREP | KILLED**
- * **SUSPENDED --> RUNNING | KILLED**
- * **SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED**
- * **PREPPAUSED --> PREP | KILLED**
- * **PAUSED --> SUSPENDED | RUNNING | KILLED**
- * **PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED**
- * **FAILED | KILLED --> IGNORED**
- * **IGNORED --> RUNNING**
-
-When a coordinator job is submitted, oozie parses the coordinator job XML. Oozie then creates a record for the coordinator with status **PREP** and returns a unique ID. The coordinator is also started immediately if pause time is not set.
-
-When a user requests to suspend a coordinator job that is in **PREP** state, oozie puts the job in status **PREPSUSPENDED**. Similarly, when pause time reaches for a coordinator job with **PREP** status, oozie puts the job in status **PREPPAUSED**.
-
-Conversely, when a user requests to resume a **PREPSUSPENDED** coordinator job, oozie puts the job in status **PREP**. And when pause time is reset for a coordinator job and job status is **PREPPAUSED**, oozie puts the job in status **PREP**.
-
-When a coordinator job starts, oozie puts the job in status **RUNNING** and start materializing workflow jobs based on job frequency. If any workflow job goes to **FAILED/KILLED/TIMEDOUT** state, the coordinator job is put in **RUNNINGWITHERROR**
-
-When a user requests to kill a coordinator job, oozie puts the job in status **KILLED** and it sends kill to all submitted workflow jobs.
-
-When a user requests to suspend a coordinator job that is in **RUNNING** status, oozie puts the job in status **SUSPENDED** and it suspends all submitted workflow jobs. Similarly, when a user requests to suspend a coordinator job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **SUSPENDEDWITHERROR** and it suspends all submitted workflow jobs.
-
-When pause time reaches for a coordinator job that is in **RUNNING** status, oozie puts the job in status **PAUSED**. Similarly, when pause time reaches for a coordinator job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **PAUSEDWITHERROR**.
-
-Conversely, when a user requests to resume a **SUSPENDED** coordinator job, oozie puts the job in status **RUNNING**. Also, when a user requests to resume a **SUSPENDEDWITHERROR** coordinator job, oozie puts the job in status **RUNNINGWITHERROR**. And when pause time is reset for a coordinator job and job status is **PAUSED**, oozie puts the job in status **RUNNING**. Also, when the pause time is reset for a coordinator job and job status is **PAUSEDWITHERROR**, oozie puts the job in status **RUNNINGWITHERROR**
-
-A coordinator job creates workflow jobs (commonly coordinator actions) only for the duration of the coordinator job and only if the coordinator job is in **RUNNING** status. If the coordinator job has been suspended, when resumed it will create all the coordinator actions that should have been created during the time it was suspended, actions will not be lost, they will delayed.
-
-When the coordinator job materialization finishes and all workflow jobs finish, oozie updates the coordinator status accordingly.
-For example, if all workflows are **SUCCEEDED**, oozie puts the coordinator job into **SUCCEEDED** status.
-If all workflows are **FAILED**, oozie puts the coordinator job into **FAILED** status. If all workflows are **KILLED**, the coordinator
-job status changes to KILLED. However, if any workflow job finishes with not **SUCCEEDED** and combination of **KILLED**, **FAILED** or
-**TIMEOUT**, oozie puts the coordinator job into **DONEWITHERROR**. If all coordinator actions are **TIMEDOUT**, oozie puts the
-coordinator job into **DONEWITHERROR**.
-
-A coordinator job in **FAILED** or **KILLED** status can be changed to **IGNORED** status. A coordinator job in **IGNORED** status can be changed to
- **RUNNING** status.
-
-#### 6.1.3. Coordinator Action
-
-A coordinator job creates and executes coordinator actions.
-
-A coordinator action is normally a workflow job that consumes and produces dataset instances.
-
-Once an coordinator action is created (this is also referred as the action being materialized), the coordinator action will be in waiting until all required inputs for execution are satisfied or until the waiting times out.
-
-##### 6.1.3.1. Coordinator Action Creation (Materialization)
-
-A coordinator job has one driver event that determines the creation (materialization) of its coordinator actions (typically a workflow job).
-
- * For synchronous coordinator jobs the driver event is the frequency of the coordinator job.
-
-##### 6.1.3.2. Coordinator Action Status
-
-Once a coordinator action has been created (materialized) the coordinator action qualifies for execution. At this point, the action status is **WAITING**.
-
-A coordinator action in **WAITING** status must wait until all its input events are available before is ready for execution. When a coordinator action is ready for execution its status is **READY**.
-
-A coordinator action in **WAITING** status may timeout before it becomes ready for execution. Then the action status is **TIMEDOUT**.
-
-A coordinator action may remain in **READY** status for a while, without starting execution, due to the concurrency execution policies of the coordinator job.
-
-A coordinator action in **READY** or **WAITING** status changes to **SKIPPED** status if the execution strategy is LAST_ONLY and the
-current time is past the next action's nominal time. See section 6.3 for more details.
-
-A coordinator action in **READY** or **WAITING** status changes to **SKIPPED** status if the execution strategy is NONE and the
-current time is past the action's nominal time + 1 minute. See section 6.3 for more details.
-
-A coordinator action in **READY** status changes to **SUBMITTED** status if total current **RUNNING** and **SUBMITTED** actions are less than concurrency execution limit.
-
-A coordinator action in **SUBMITTED** status changes to **RUNNING** status when the workflow engine start execution of the coordinator action.
-
-A coordinator action is in **RUNNING** status until the associated workflow job completes its execution. Depending on the workflow job completion status, the coordinator action will be in **SUCCEEDED**, **KILLED** or **FAILED** status.
-
-A coordinator action in **WAITING**, **READY**, **SUBMITTED** or **RUNNING** status can be killed, changing to **KILLED** status.
-
-A coordinator action in **SUBMITTED** or **RUNNING** status can also fail, changing to **FAILED** status.
-
-A coordinator action in **FAILED**, **KILLED**, or **TIMEDOUT** status can be changed to **IGNORED** status. A coordinator action in **IGNORED** status can be
- rerun, changing to **WAITING** status.
-
-Valid coordinator action status transitions are:
-
- * **WAITING --> READY | TIMEDOUT | SKIPPED | KILLED**
- * **READY --> SUBMITTED | SKIPPED | KILLED**
- * **SUBMITTED --> RUNNING | KILLED | FAILED**
- * **RUNNING --> SUCCEEDED | KILLED | FAILED**
- * **FAILED | KILLED | TIMEDOUT --> IGNORED**
- * **IGNORED --> WAITING**
-
-#### 6.1.4. Input Events
-
-The Input events of a coordinator application specify the input conditions that are required in order to execute a coordinator action.
-
-In the current specification input events are restricted to dataset instances availability.
-
-All the datasets instances defined as input events must be available for the coordinator action to be ready for execution ( **READY** status).
-
-Input events are normally parameterized. For example, the last 24 hourly instances of the 'searchlogs' dataset.
-
-Input events can be refer to multiple instances of multiple datasets. For example, the last 24 hourly instances of the 'searchlogs' dataset and the last weekly instance of the 'celebrityRumours' dataset.
-
-#### 6.1.5. Output Events
-
-A coordinator action can produce one or more dataset(s) instances as output.
-
-Dataset instances produced as output by one coordinator actions may be consumed as input by another coordinator action(s) of other coordinator job(s).
-
-The chaining of coordinator jobs via the datasets they produce and consume is referred as a **data pipeline.**
-
-In the current specification coordinator job output events are restricted to dataset instances.
-
-#### 6.1.6. Coordinator Action Execution Policies
-
-The execution policies for the actions of a coordinator job can be defined in the coordinator application.
-
- * Timeout: A coordinator job can specify the timeout for its coordinator actions, this is, how long the coordinator action will be in *WAITING* or *READY* status before giving up on its execution.
- * Concurrency: A coordinator job can specify the concurrency for its coordinator actions, this is, how many coordinator actions are allowed to run concurrently ( **RUNNING** status) before the coordinator engine starts throttling them.
- * Execution strategy: A coordinator job can specify the execution strategy of its coordinator actions when there is backlog of coordinator actions in the coordinator engine. The different execution strategies are 'oldest first', 'newest first', 'none' and 'last one only'. A backlog normally happens because of delayed input data, concurrency control or because manual re-runs of coordinator jobs.
- * Throttle: A coordinator job can specify the materialization or creation throttle value for its coordinator actions, this is, how many maximum coordinator actions are allowed to be in WAITING state concurrently.
-
-#### 6.1.7. Data Pipeline Application
-
-Commonly, multiple workflow applications are chained together to form a more complex application.
-
-Workflow applications are run on regular basis, each of one of them at their own frequency. The data consumed and produced by these workflow applications is relative to the nominal time of workflow job that is processing the data. This is a **coordinator application**.
-
-The output of multiple workflow jobs of a single workflow application is then consumed by a single workflow job of another workflow application, this is done on regular basis as well. These workflow jobs are triggered by recurrent actions of coordinator jobs. This is a set of **coordinator jobs** that inter-depend on each other via the data they produce and consume.
-
-This set of interdependent **coordinator applications** is referred as a **data pipeline application**.
-
-### 6.2. Synchronous Coordinator Application Example
-
- * The `checkouts` synchronous dataset is created every 15 minutes by an online checkout store.
- * The `hourlyRevenue` synchronous dataset is created every hour and contains the hourly revenue.
- * The `dailyRevenue` synchronous dataset is created every day and contains the daily revenue.
- * The `monthlyRevenue` synchronous dataset is created every month and contains the monthly revenue.
-
- * The `revenueCalculator-wf` workflow consumes checkout data and produces as output the corresponding revenue.
- * The `rollUpRevenue-wf` workflow consumes revenue data and produces a consolidated output.
-
- * The `hourlyRevenue-coord` coordinator job triggers, every hour, a `revenueCalculator-wf` workflow. It specifies as input the last 4 `checkouts` dataset instances and it specifies as output a new instance of the `hourlyRevenue` dataset.
- * The `dailyRollUpRevenue-coord` coordinator job triggers, every day, a `rollUpRevenue-wf` workflow. It specifies as input the last 24 `hourlyRevenue` dataset instances and it specifies as output a new instance of the `dailyRevenue` dataset.
- * The `monthlyRollUpRevenue-coord` coordinator job triggers, once a month, a `rollUpRevenue-wf` workflow. It specifies as input all the `dailyRevenue` dataset instance of the month and it specifies as output a new instance of the `monthlyRevenue` dataset.
-
-This example contains describes all the components that conform a data pipeline: datasets, coordinator jobs and coordinator actions (workflows).
-
-The coordinator actions (the workflows) are completely agnostic of datasets and their frequencies, they just use them as input and output data (i.e. HDFS files or directories). Furthermore, as the example shows, the same workflow can be used to process similar datasets of different frequencies.
-
-The frequency of the `hourlyRevenue-coord` coordinator job is 1 hour, this means that every hour a coordinator action is created. A coordinator action will be executed only when the 4 `checkouts` dataset instances for the corresponding last hour are available, until then the coordinator action will remain as created (materialized), in **WAITING** status. Once the 4 dataset instances for the corresponding last hour are available, the coordinator action will be executed and it will start a `revenueCalculator-wf` workflow job.
-
-### 6.3. Synchronous Coordinator Application Definition
-
-A synchronous coordinator definition is a is defined by a name, start time and end time, the frequency of creation of its coordinator actions, the input events, the output events and action control information:
-
- * **<font color="#0000ff"> start: </font>** The start datetime for the job. Starting at this time actions will be materialized. Refer to section #3 'Datetime Representation' for syntax details.
- * **<font color="#0000ff"> end: </font>** The end datetime for the job. When actions will stop being materialized. Refer to section #3 'Datetime Representation' for syntax details.
- * **<font color="#0000ff"> timezone:</font>** The timezone of the coordinator application.
- * **<font color="#0000ff"> frequency: </font>** The frequency, in minutes, to materialize actions. Refer to section #4 'Time Interval Representation' for syntax details.
- * Control information:
- * **<font color="#0000ff"> timeout: </font>** The maximum time, in minutes, that a materialized action will be waiting for the additional conditions to be satisfied before being discarded. A timeout of `0` indicates that at the time of materialization all the other conditions must be satisfied, else the action will be discarded. A timeout of `0` indicates that if all the input events are not satisfied at the time of action materialization, the action should timeout immediately. A timeout of `-1` indicates no timeout, the materialized action will wait forever for the other conditions to be satisfied. The default value is `-1`. The timeout can only cause a `WAITING` action to transition to `TIMEDOUT`; once the data dependency is satisified, a `WAITING` action transitions to `READY`, and the timeout no longer has any affect, even if the action hasn't transitioned to `SUBMITTED` or `RUNNING` when it expires.
- * **<font color="#0000ff"> concurrency: </font>** The maximum number of actions for this job that can be running at the same time. This value allows to materialize and submit multiple instances of the coordinator app, and allows operations to catchup on delayed processing. The default value is `1`.
- * **<font color="#0000ff"> execution: </font>** Specifies the execution order if multiple instances of the coordinator job have satisfied their execution criteria. Valid values are:
- ** `FIFO` (oldest first) **default*.
- * `LIFO` (newest first).
- * `LAST_ONLY` (see explanation below).
- * `NONE` (see explanation below).
- * **<font color="#0000ff"> throttle: </font>** The maximum coordinator actions are allowed to be in WAITING state concurrently. The default value is `12`.
- * **<font color="#0000ff"> datasets: </font>** The datasets coordinator application uses.
- * **<font color="#0000ff"> input-events: </font>** The coordinator job input events.
- * **<font color="#0000ff"> data-in: </font>** It defines one job input condition that resolves to one or more instances of a dataset.
- * **<font color="#0000ff"> name: </font>** input condition name.
- * **<font color="#0000ff"> dataset: </font>** dataset name.
- * **<font color="#0000ff"> instance: </font>** refers to a single dataset instance (the time for a synchronous dataset).
- * **<font color="#0000ff"> start-instance: </font>** refers to the beginning of an instance range (the time for a synchronous dataset).
- * **<font color="#0000ff"> end-instance: </font>** refers to the end of an instance range (the time for a synchronous dataset).
- * **<font color="#0000ff"> output-events: </font>** The coordinator job output events.
- * **<font color="#0000ff"> data-out: </font>** It defines one job output that resolves to a dataset instance.
- * **<font color="#0000ff"> name: </font>** output name.
- * **<font color="#0000ff"> dataset: </font>** dataset name.
- * **<font color="#0000ff"> instance: </font>** dataset instance that will be generated by coordinator action.
- * **<font color="#0000ff"> nocleanup: </font>** disable cleanup of the output dataset in rerun if true, even when nocleanup option is not used in CLI command.
- * **<font color="#0000ff"> action: </font>** The coordinator action to execute.
- * **<font color="#0000ff"> workflow: </font>** The workflow job invocation. Workflow job properties can refer to the defined data-in and data-out elements.
-
-**LAST_ONLY:** While `FIFO` and `LIFO` simply specify the order in which READY actions should be executed, `LAST_ONLY` can actually
-cause some actions to be SKIPPED and is a little harder to understand. When `LAST_ONLY` is set, an action that is `WAITING`
-or `READY` will be `SKIPPED` when the current time is past the next action's nominal time. For example, suppose action 1 and 2
-are both `READY`, the current time is 5:00pm, and action 2's nominal time is 5:10pm. In 10 minutes from now, at 5:10pm, action 1
-will become SKIPPED, assuming it doesn't transition to `SUBMITTED` (or a terminal state) before then. This sounds similar to the
-timeout control, but there are some important differences:
-
- * The timeout time is configurable while the `LAST_ONLY` time is effectively the frequency.
- * Reaching the timeout causes an action to transition to `TIMEDOUT`, which will cause the Coordinator Job to become `RUNNINGWITHERROR` and eventually `DONEWITHERROR`. With `LAST_ONLY`, an action becomes `SKIPPED` and the Coordinator Job remains `RUNNING` and eventually `DONE`.
- * The timeout is looking satisfying the data dependency, while `LAST_ONLY` is looking at the action itself. This means that the timeout can only cause a transition from `WAITING`, while `LAST_ONLY` can cause a transition from `WAITING` or `READY`.
-
-`LAST_ONLY` is useful if you want a recurring job, but do not actually care about the individual instances and just
-always want the latest action. For example, if you have a coordinator running every 10 minutes and take Oozie down for 1 hour, when
-Oozie comes back, there would normally be 6 actions `WAITING` or `READY` to run. However, with `LAST_ONLY`, only the current one
-will go to `SUBMITTED` and then `RUNNING`; the others will go to `SKIPPED`.
-
-**NONE:** Similar to `LAST_ONLY` except instead of looking at the next action's nominal time, it looks
-at `oozie.coord.execution.none.tolerance` in oozie-site.xml (default is 1 minute). When `NONE` is set, an action that is `WAITING`
-or `READY` will be `SKIPPED` when the current time is more than the configured number of minutes (tolerance) past that action's
-nominal time. For example, suppose action 1 and 2 are both `READY`, the current time is 5:20pm, and both actions' nominal times are
-before 5:19pm. Both actions will become `SKIPPED`, assuming they don't transition to `SUBMITTED` (or a terminal state) before then.
-
-**<font color="#800080">Syntax: </font>**
-
-
-```
- <coordinator-app name="[NAME]" frequency="[FREQUENCY]"
- start="[DATETIME]" end="[DATETIME]" timezone="[TIMEZONE]"
- xmlns="uri:oozie:coordinator:0.1">
- <controls>
- <timeout>[TIME_PERIOD]</timeout>
- <concurrency>[CONCURRENCY]</concurrency>
- <execution>[EXECUTION_STRATEGY]</execution>
- </controls>
-.
- <datasets>
- <include>[SHARED_DATASETS]</include>
- ...
-.
- <!-- Synchronous datasets -->
- <dataset name="[NAME]" frequency="[FREQUENCY]"
- initial-instance="[DATETIME]" timezone="[TIMEZONE]">
- <uri-template>[URI_TEMPLATE]</uri-template>
- </dataset>
- ...
-.
- </datasets>
-.
- <input-events>
- <data-in name="[NAME]" dataset="[DATASET]">
- <instance>[INSTANCE]</instance>
- ...
- </data-in>
- ...
- <data-in name="[NAME]" dataset="[DATASET]">
- <start-instance>[INSTANCE]</start-instance>
- <end-instance>[INSTANCE]</end-instance>
- </data-in>
- ...
- </input-events>
- <output-events>
- <data-out name="[NAME]" dataset="[DATASET]">
- <instance>[INSTANCE]</instance>
- </data-out>
- ...
- </output-events>
- <action>
- <workflow>
- <app-path>[WF-APPLICATION-PATH]</app-path>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- </workflow>
- </action>
- </coordinator-app>
-```
-
-**<font color="#008000"> Examples: </font>**
-
-**1. A Coordinator Job that creates an executes a single coordinator action:**
-
-The following example describes a synchronous coordinator application that runs once a day for 1 day at the end of the day. It consumes an instance of a daily 'logs' dataset and produces an instance of a daily 'siteAccessStats' dataset.
-
-**Coordinator application definition:**
-
-
-```
- <coordinator-app name="hello-coord" frequency="${coord:days(1)}"
- start="2009-01-02T08:00Z" end="2009-01-02T08:00Z"
- timezone="America/Los_Angeles"
- xmlns="uri:oozie:coordinator:0.1">
- <datasets>
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
- <uri-template>hdfs://bar:8020/app/logs/${YEAR}${MONTH}/${DAY}/data</uri-template>
- </dataset>
- <dataset name="siteAccessStats" frequency="${coord:days(1)}"
- initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
- <uri-template>hdfs://bar:8020/app/stats/${YEAR}/${MONTH}/${DAY}/data</uri-template>
- </dataset>
- </datasets>
- <input-events>
- <data-in name="input" dataset="logs">
- <instance>2009-01-02T08:00Z</instance>
- </data-in>
- </input-events>
- <output-events>
- <data-out name="output" dataset="siteAccessStats">
- <instance>2009-01-02T08:00Z</instance>
- </data-out>
- </output-events>
- <action>
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>${coord:dataIn('input')}</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>${coord:dataOut('output')}</value>
- </property>
- </configuration>
- </workflow>
- </action>
- </coordinator-app>
-```
-
-There are 2 synchronous datasets with a daily frequency and they are expected at the end of each PST8PDT day.
-
-This coordinator job runs for 1 day on January 1st 2009 at 24:00 PST8PDT.
-
-The workflow job invocation for the single coordinator action would resolve to:
-
-
-```
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>hdfs://bar:8020/app/logs/200901/02/data</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>hdfs://bar:8020/app/stats/2009/01/02/data</value>
- </property>
- </configuration>
- </workflow>
-```
-
-IMPORTANT: Note Oozie works in UTC datetimes, all URI templates resolve to UTC datetime values. Because of the timezone difference between UTC and PST8PDT, the URIs resolves to `2009-01-02T08:00Z` (UTC) which is equivalent to 2009-01-01T24:00PST8PDT= (PST).
-
-There is single input event, which resolves to January 1st PST8PDT instance of the 'logs' dataset. There is single output event, which resolves to January 1st PST8PDT instance of the 'siteAccessStats' dataset.
-
-The `${coord:dataIn(String name)}` and `${coord:dataOut(String name)}` EL functions resolve to the dataset instance URIs of the corresponding dataset instances. These EL functions are properly defined in a subsequent section.
-
-Because the `${coord:dataIn(String name)}` and `${coord:dataOut(String name)}` EL functions resolve to URIs, which are HDFS URIs, the workflow job itself does not deal with dataset instances, just HDFS URIs.
-
-**2. A Coordinator Job that executes its coordinator action multiple times:**
-
-A more realistic version of the previous example would be a coordinator job that runs for a year creating a daily action an consuming the daily 'logs' dataset instance and producing the daily 'siteAccessStats' dataset instance.
-
-The coordinator application is identical, except for the frequency, 'end' date and parameterization in the input and output events sections:
-
-
-```
- <coordinator-app name="hello-coord" frequency="${coord:days(1)}"
- start="2009-01-02T08:00Z" end="2010-01-02T08:00Z"
- timezone="America/Los_Angeles"
- xmlns="uri:oozie:coordinator:0.1">
- <datasets>
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
- <uri-template>hdfs://bar:8020/app/logs/${YEAR}${MONTH}/${DAY}/data</uri-template>
- </dataset>
- <dataset name="siteAccessStats" frequency="${coord:days(1)}"
- initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
- <uri-template>hdfs://bar:8020/app/stats/${YEAR}/${MONTH}/${DAY}/data</uri-template>
- </dataset>
- </datasets>
- <input-events>
- <data-in name="input" dataset="logs">
- <instance>${coord:current(0)}</instance>
- </data-in>
- </input-events>
- <output-events>
- <data-out name="output" dataset="siteAccessStats">
- <instance>${coord:current(0)}</instance>
- </data-out>
- </output-events>
- <action>
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>${coord:dataIn('input')}</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>${coord:dataOut('output')}</value>
- </property>
- </configuration>
- </workflow>
- </action>
- </coordinator-app>
-```
-
-The `${coord:current(int offset)}` EL function resolves to coordinator action creation time, that would be the current day at the time the coordinator action is created: `2009-01-02T08:00 ... 2010-01-01T08:00`. This EL function is properly defined in a subsequent section.
-
-There is single input event, which resolves to the current day instance of the 'logs' dataset.
-
-There is single output event, which resolves to the current day instance of the 'siteAccessStats' dataset.
-
-The workflow job invocation for the first coordinator action would resolve to:
-
-
-```
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>hdfs://bar:8020/app/logs/200901/02/data</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>hdfs://bar:8020/app/stats/2009/01/02/data</value>
- </property>
- </configuration>
- </workflow>
-```
-
-For the second coordinator action it would resolve to:
-
-
-```
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>hdfs://bar:8020/app/logs/200901/03/data</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>hdfs://bar:8020/app/stats/2009/01/03/data</value>
- </property>
- </configuration>
- </workflow>
-```
-
-And so on.
-
-**3. A Coordinator Job that executes its coordinator action multiple times and as input takes multiple dataset instances:**
-
-The following example is a variation of the example #2 where the synchronous coordinator application runs weekly. It consumes the of the last 7 instances of a daily 'logs' dataset and produces an instance of a weekly 'weeklySiteAccessStats' dataset.
-
-'logs' is a synchronous dataset with a daily frequency and it is expected at the end of each day (24:00).
-
-'weeklystats' is a synchronous dataset with a weekly frequency and it is expected at the end (24:00) of every 7th day.
-
-The coordinator application frequency is weekly and it starts on the 7th day of the year:
-
-
-```
- <coordinator-app name="hello2-coord" frequency="${coord:days(7)}"
- start="2009-01-07T24:00Z" end="2009-12-12T24:00Z"
- timezone="UTC"
- xmlns="uri:oozie:coordinator:0.1">
- <datasets>
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-01-01T24:00Z" timezone="UTC">
- <uri-template>hdfs://bar:8020/app/logs/${YEAR}${MONTH}/${DAY}</uri-template>
- </dataset>
- <dataset name="weeklySiteAccessStats" frequency="${coord:days(7)}"
- initial-instance="2009-01-07T24:00Z" timezone="UTC">
- <uri-template>hdfs://bar:8020/app/weeklystats/${YEAR}/${MONTH}/${DAY}</uri-template>
- </dataset>
- </datasets>
- <input-events>
- <data-in name="input" dataset="logs">
- <start-instance>${coord:current(-6)}</start-instance>
- <end-instance>${coord:current(0)}</end-instance>
- </data-in>
- </input-events>
- <output-events>
- <data-out name="output" dataset="siteAccessStats">
- <instance>${coord:current(0)}</instance>
- </data-out>
- </output-events>
- <action>
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>${coord:dataIn('input')}</value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>${coord:dataOut('output')}</value>
- </property>
- </configuration>
- </workflow>
- </action>
- </coordinator-app>
-```
-
-The `${coord:current(int offset)}` EL function resolves to coordinator action creation time minus the specified offset multiplied by the dataset frequency. This EL function is properly defined in a subsequent section.
-
-The input event, instead resolving to a single 'logs' dataset instance, it refers to a range of 7 dataset instances - the instance for 6 days ago, 5 days ago, ... and today's instance.
-
-The output event resolves to the current day instance of the 'weeklySiteAccessStats' dataset. As the coordinator job will create a coordinator action every 7 days, dataset instances for the 'weeklySiteAccessStats' dataset will be created every 7 days.
-
-The workflow job invocation for the first coordinator action would resolve to:
-
-
-```
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>
- hdfs://bar:8020/app/logs/200901/01,hdfs://bar:8020/app/logs/200901/02,
- hdfs://bar:8020/app/logs/200901/03,hdfs://bar:8020/app/logs/200901/05,
- hdfs://bar:8020/app/logs/200901/05,hdfs://bar:8020/app/logs/200901/06,
- hdfs://bar:8020/app/logs/200901/07
- </value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>hdfs://bar:8020/app/stats/2009/01/07</value>
- </property>
- </configuration>
- </workflow>
-```
-
-For the second coordinator action it would resolve to:
-
-
-```
- <workflow>
- <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
- <configuration>
- <property>
- <name>wfInput</name>
- <value>
- hdfs://bar:8020/app/logs/200901/08,hdfs://bar:8020/app/logs/200901/09,
- hdfs://bar:8020/app/logs/200901/10,hdfs://bar:8020/app/logs/200901/11,
- hdfs://bar:8020/app/logs/200901/12,hdfs://bar:8020/app/logs/200901/13,
- hdfs://bar:8020/app/logs/200901/16
- </value>
- </property>
- <property>
- <name>wfOutput</name>
- <value>hdfs://bar:8020/app/stats/2009/01/16</value>
- </property>
- </configuration>
- </workflow>
-```
-
-And so on.
-
-### 6.4. Asynchronous Coordinator Application Definition
- * TBD
-
-### 6.5. Parameterization of Coordinator Applications
-
-When a coordinator job is submitted to Oozie, the submitter may specify as many coordinator job configuration properties as required (similar to Hadoop JobConf properties).
-
-Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as `${NAME}` variables within the coordinator application definition.
-
-Configuration Properties that are not a valid Java identifier, for example `job.tracker`, are available via the `${coord:conf(String name)}` function. Valid Java identifier properties are available via this function as well.
-
-Using properties that are valid Java identifiers result in a more readable and compact definition.
-
-Dataset definitions can be also parameterized, the parameters are resolved using the configuration properties of Job configuration used to submit the coordinator job.
-
-If a configuration property used in the definitions is not provided with the job configuration used to submit a coordinator job, the value of the parameter will be undefined and the job submission will fail.
-
-**<font color="#008000"> Example: </font>**
-
-Coordinator application definition:
-
-
-```
- <coordinator-app name="app-coord" frequency="${coord:days(1)}"
- start="${jobStart}" end="${jobEnd}" timezone="${timezone}"
- xmlns="uri:oozie:coordinator:0.1">
- <datasets>
- <dataset name="logs" frequency="${coord:hours(1)}"
- initial-instance="${logsInitialInstance}" timezone="${timezone}">
- <uri-template>
- hdfs://bar:8020/app/logs/${market}/${language}/${YEAR}${MONTH}/${DAY}/${HOUR}
- </uri-template>
- </dataset>
- </datasets>
- <input-events>
- <data-in name="input" dataset="logs">
- <start-instance>${coord:current(-23)}</start-instance>
- <end-instance>${coord:current(0)}</end-instance>
- </data-in>
- </input-events>
- <action>
- <workflow>
- ...
- </workflow>
- </action>
- </coordinator-app>
-```
-
-In the above example there are 6 configuration parameters (variables) that have to be provided when submitting a job:
-
- * `jobStart` : start datetime for the job, in UTC
- * `jobEnd` : end datetime for the job, in UTC
- * `logsInitialInstance` : expected time of the first logs instance, in UTC
- * `timezone` : timezone for the job and the dataset
- * `market` : market to compute by this job, used in the uri-template
- * `language` : language to compute by this job, used in the uri-template
-
-IMPORTANT: Note that this example is not completely correct as it always consumes the last 24 instances of the 'logs' dataset. It is assumed that all days have 24 hours. For timezones that observe daylight saving this application will not work as expected as it will consume the wrong number of dataset instances in DST switch days. To be able to handle these scenarios, the `${coord:hoursInDays(int n)}` and `${coord:daysInMonths(int n)}` EL functions must be used (refer to section #6.6.2 and #6.6.3).
-
-If the above 6 properties are not specified, the job will fail.
-
-As of schema 0.4, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
-properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
-
-**Example:**
-
-The previous parameterized coordinator application definition with formal parameters:
-
-
-```
- <coordinator-app name="app-coord" frequency="${coord:days(1)}"
- start="${jobStart}" end="${jobEnd}" timezone="${timezone}"
- xmlns="uri:oozie:coordinator:0.1">
- <parameters>
- <property>
- <name>jobStart</name>
- </property>
- <property>
- <name>jobEnd</name>
- <value>2012-12-01T22:00Z</value>
- </property>
- </parameters>
- <datasets>
- <dataset name="logs" frequency="${coord:hours(1)}"
- initial-instance="${logsInitialInstance}" timezone="${timezone}">
- <uri-template>
- hdfs://bar:8020/app/logs/${market}/${language}/${YEAR}${MONTH}/${DAY}/${HOUR}
- </uri-template>
- </dataset>
- </datasets>
- <input-events>
- <data-in name="input" dataset="logs">
- <start-instance>${coord:current(-23)}</start-instance>
- <end-instance>${coord:current(0)}</end-instance>
- </data-in>
- </input-events>
- <action>
- <workflow>
- ...
- </workflow>
- </action>
- </coordinator-app>
-```
-
-In the above example, if `jobStart` is not specified, Oozie will print an error message instead of submitting the job. If
-`jobEnd` is not specified, Oozie will use the default value, `2012-12-01T22:00Z`.
-
-### 6.6. Parameterization of Dataset Instances in Input and Output Events
-
-A coordinator application job typically launches several coordinator actions during its lifetime. A coordinator action typically uses its creation (materialization) time to resolve the specific datasets instances required for its input and output events.
-
-The following EL functions are the means for binding the coordinator action creation time to the datasets instances of its input and output events.
-
-#### 6.6.1. coord:current(int n) EL Function for Synchronous Datasets
-
-`${coord:current(int n)}` represents the n<sup>th</sup> dataset instance for a **synchronous** dataset, relative to the coordinator action creation (materialization) time. The coordinator action creation (materialization) time is computed based on the coordinator job start time and its frequency. The n<sup>th</sup> dataset instance is computed based on the dataset's initial-instance datetime, its frequency and the (current) coordinator action creation (materialization) time.
-
-`n` can be a negative integer, zero or a positive integer.
-
-`${coord:current(int n)}` returns the nominal datetime for n<sup>th</sup> dataset instance relative to the coordinator action creation (materialization) time.
-
-`${coord:current(int n)}` performs the following calculation:
-
-
-```
-DS_II : dataset initial-instance (datetime)
-DS_FREQ: dataset frequency (minutes)
-CA_NT: coordinator action creation (materialization) nominal time
-
-coord:current(int n) = DS_II + DS_FREQ * ( (CA_NT - DS_II) div DS_FREQ + n)
-```
-
-NOTE: The formula above is not 100% correct, because DST changes the calculation has to account for hour shifts. Oozie Coordinator must make the correct calculation accounting for DST hour shifts.
-
-When a positive integer is used with the `${coord:current(int n)}`, it refers to
<TRUNCATED>
[15/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_HiveActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_HiveActionExtension.md b/docs/src/site/markdown/DG_HiveActionExtension.md
new file mode 100644
index 0000000..99a73c6
--- /dev/null
+++ b/docs/src/site/markdown/DG_HiveActionExtension.md
@@ -0,0 +1,476 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Hive Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+<a name="HiveAction"></a>
+## Hive Action
+
+The `hive` action runs a Hive job.
+
+The workflow job will wait until the Hive job completes before
+continuing to the next action.
+
+To run the Hive job, you have to configure the `hive` action with the `resource-manager`, `name-node` and Hive `script`
+(or Hive `query`) elements as well as the necessary parameters and configuration.
+
+A `hive` action can be configured to create or delete HDFS directories
+before starting the Hive job.
+
+Hive configuration can be specified with a file, using the `job-xml`
+element, and inline, using the `configuration` elements.
+
+Oozie EL expressions can be used in the inline configuration. Property
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
+
+Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties
+must not be present in the inline configuration.
+
+As with Hadoop `map-reduce` jobs, it is possible to add files and
+archives in order to make them available to the Hive job. Refer to the
+[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
+section for more information about this feature.
+
+Oozie Hive action supports Hive scripts with parameter variables, their
+syntax is `${VARIABLES}`.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <hive xmlns="uri:oozie:hive-action:1.0">
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[HIVE SETTINGS FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <script>[HIVE-SCRIPT]</script>
+ <param>[PARAM-VALUE]</param>
+ ...
+ <param>[PARAM-VALUE]</param>
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ </hive>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
+
+The `job-xml` element, if present, specifies a file containing configuration
+for the Hive job. As of schema 0.3, multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
+
+The `configuration` element, if present, contains configuration
+properties that are passed to the Hive job.
+
+The `script` element must contain the path of the Hive script to
+execute. The Hive script can be templatized with variables of the form
+`${VARIABLE}`. The values of these variables can then be specified
+using the `params` element.
+
+The `query` element available from uri:oozie:hive-action:0.6, can be used instead of the
+`script` element. It allows for embedding queries within the `worklfow.xml` directly.
+Similar to the `script` element, it also allows for the templatization of variables in the
+form `${VARIABLE}`.
+
+The `params` element, if present, contains parameters to be passed to
+the Hive script.
+
+All the above elements can be parameterized (templatized) using EL
+expressions.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="myfirsthivejob">
+ <hive xmlns="uri:oozie:hive-action:1.0">
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <script>myscript.q</script>
+ <param>InputDir=/home/tucu/input-data</param>
+ <param>OutputDir=${jobOutput}</param>
+ </hive>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+### Hive Default and Site Configuration Files
+
+Hive (as of Hive 0.8) ignores a `hive-default.xml` file. As a result, Oozie (as of Oozie 3.4) ignores the `oozie.hive.defaults`
+property that was previously required by earlier versions of Oozie for the Hive action.
+
+### Hive Action Logging
+
+Hive action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Hive.
+
+From Oozie web-console, from the Hive action pop up using the 'Console URL' link, it is possible
+to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
+
+The logging level of the Hive action can set in the Hive action configuration using the
+property `oozie.hive.log.level`. The default value is `INFO`.
+
+## Appendix, Hive XML-Schema
+
+### AE.A Appendix A, Hive XML-Schema
+
+#### Hive Action Schema Version 1.0
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive="uri:oozie:hive-action:1.0"
+ elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive-action:1.0">
+.
+ <xs:include schemaLocation="oozie-common-1.0.xsd"/>
+.
+ <xs:element name="hive" type="hive:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:choice>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:choice minOccurs="1" maxOccurs="1">
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Hive Action Schema Version 0.6
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive="uri:oozie:hive-action:0.6" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive-action:0.6">
+.
+ <xs:element name="hive" type="hive:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:choice minOccurs="1" maxOccurs="1">
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+#### Hive Action Schema Version 0.5
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive="uri:oozie:hive-action:0.5" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive-action:0.5">
+.
+ <xs:element name="hive" type="hive:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Hive Action Schema Version 0.4
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive="uri:oozie:hive-action:0.4" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive-action:0.4">
+.
+ <xs:element name="hive" type="hive:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Hive Action Schema Version 0.3
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive="uri:oozie:hive-action:0.3" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive-action:0.3">
+.
+ <xs:element name="hive" type="hive:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Hive Action Schema Version 0.2
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive="uri:oozie:hive-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive-action:0.2">
+.
+ <xs:element name="hive" type="hive:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_JMSNotifications.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_JMSNotifications.md b/docs/src/site/markdown/DG_JMSNotifications.md
new file mode 100644
index 0000000..e8f8a76
--- /dev/null
+++ b/docs/src/site/markdown/DG_JMSNotifications.md
@@ -0,0 +1,236 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# JMS Notifications
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Overview
+Since Oozie 4.0, Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss
+events. This provides an alternative to polling Oozie for Job or SLA related information and getting events as they
+happen without any delay. Clients can be written to consume these notifications and integrate with different monitoring
+and alerting systems.
+
+## Oozie Server Configuration
+Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) section of [Oozie Install](AG_Install.html)
+documentation for the Oozie server side configuration required to support publishing notifications to a JMS Provider.
+The JNDI properties for the JMS provider, the topics to publish to and the notification types to publish (Job and/or SLA)
+need to be configured.
+
+## Consuming Notifications
+
+### Notification types
+Job and SLA notifications are published to the configured JMS Provider on the configured topics.
+
+Job status change notifications include job start, success, failure, suspended, etc. Currently only workflow job and
+coordinator action status change notifications are published.
+
+SLA notifications include START_MET, END_MET, DURATION_MET, START_MISS, END_MISS, DURATION_MISS events and are published
+for a workflow job, workflow action or coordinator action for which SLA information is configured in the job xml. Refer
+to [SLA Configuration](DG_SLAMonitoring.html#Configuring_SLA_in_Applications) for information on configuring SLA for a workflow or
+coordinator.
+
+### JMS Topic
+Consumers interested in notification on events will require to know the JNDI properties to connect to the JMS provider.
+They will also need to know the JMS topic on which notifications for a particular job are published.
+
+Oozie Client provides the following APIs :
+
+```
+public JMSConnectionInfo getJMSConnectionInfo()
+public String getJMSTopicName(String jobId)
+```
+
+The JMSConnectionInfo exposes 3 methods:
+
+
+```
+Properties getJNDIProperties();
+String getTopicPattern(AppType appType);
+String getTopicPrefix();
+```
+
+The topic is obtained by concatenating topic prefix and the substituted value for topic pattern. The topic pattern
+can be a constant value like workflow or coordinator which the administrator has configured or a variable (either ${username}
+or ${jobId}). If ${username}, it has to be substituted with the name of the user who has submitted the job; and if ${jobId} it has
+to be substituted with the job Id. Administrators can chose to publish messages to topics containing user names to avoid having one
+topic containing all messages and all users having to apply selectors to filter the message they are interested in.
+
+The getJMSTopicName API can be used if the job id is already known and will give the exact topic name to which the
+notifications for that job are published.
+
+
+### JMS Message Format
+JMS messages published are `javax.jms.TextMessage`. The body contains JSON and the header contains multiple properties
+that can be used as selectors. The header properties are not repeated in the body of the message to keep the messages
+small.
+
+<b>Message Header:</b> <br/>
+The different header properties are:
+
+ * msgType - Value can be JOB or SLA.
+ * user - The user who submitted the job.
+ * appName - Application name of the job.
+ * appType - Type of the job. One of WORKFLOW_JOB, WORKFLOW_ACTION, COORDINATOR_ACTION
+ * slaStatus - Applicable only to SLA messages. Value is one of NOT_STARTED, IN_PROCESS, MET, MISS
+ * eventStatus - It takes one of the following values- START_MET, START_MISS, DURATION_MET, DURATION_MISS, END_MET, END_MISS for SLA notifications. It can take any of the following values- WAITING, STARTED, SUCCESS, SUSPEND, FAILURE for job notifications. Note that event status is different from the Job status. It is included in the header to provide better filtering. Below is the mapping of event status to the actual status of workflow job or coordinator action.
+
+WAITING = When a Coordinator Action is waiting for input dependency
+
+STARTED = When the Workflow Job or Coordinator Action is in RUNNING state
+
+SUCCESS = When the Workflow Job or Coordinator Action is in SUCCEEDED state
+
+SUSPEND = When the Workflow Job or Coordinator Action is in SUSPENDED state
+
+FAILURE = When the Workflow Job or Coordinator Action is in terminal state other than SUCCEEDED.
+
+<b>Message Body for Job Notifications:</b> <br/>
+ Sample JSON response for different job and sla events as below.
+
+
+```
+Workflow Job in RUNNING state:
+{"status":"RUNNING","id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000}
+```
+
+
+```
+Workflow Job in FAILED state:
+{"status":"FAILED","errorCode":"EL_ERROR","errorMessage":"variable [dummyvalue] cannot be resolved",
+"id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000,"endTime":1366672183543}
+```
+
+
+```
+Workflow Job in SUCCEEDED state:
+{"status":"SUCCEEDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
+"parentId":"0000025-130618221729631-oozie-oozi-C@1","endTime":1366676224154}
+```
+
+
+```
+Workflow Job in SUSPENDED state:
+{"status":"SUSPENDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
+"parentId":"0000025-130618221729631-oozie-oozi-C@1"}
+```
+
+
+```
+Coordinator Action in WAITING state:
+{"status":"WAITING","nominalTime":1310342400000,"missingDependency":"hdfs://gsbl90107.blue.com:8020/user/john/dir1/file1",
+"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
+```
+
+
+```
+Coordinator Action in RUNNING state:
+{"status":"RUNNING","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
+"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
+```
+
+
+```
+Coordinator Action in SUCCEEDED state:
+{"status":"SUCCEEDED","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
+"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677082799}
+```
+
+
+```
+Coordinator Action in FAILED state:
+{"status":"FAILED","errorCode":"E0101","errorMessage":"dummyError","nominalTime":1310342400000,
+"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,
+"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677140818}
+```
+
+<b>Message Body for SLA Notifications:</b> <br/>
+
+
+```
+Workflow Job in sla END_MISS state:
+{"id":"0000000-000000000000001-oozie-wrkf-C@1","parentId":"0000000-000000000000001-oozie-wrkf-C",
+"expectedStartTime":1356998400000,"notificationMessage":"notification of start miss","actualStartTime":1357002000000,
+"expectedDuration":-1, "actualDuration":3600,"expectedEndTime":1356998400000,"actualEndTime":1357002000000}
+```
+
+### JMS Client
+
+Oozie provides a helper class JMSMessagingUtils for consumers to deserialize the JMS messages back to Java objects.
+The below method getEventMessage() expects a sub type of EventMessage.
+There are different implementations of EventMessage - WorkflowJobMessage, CoordinatorActionMessage and SLAMessage.
+
+
+```
+<T extends EventMessage> T JMSMessagingUtils.getEventMessage(Message jmsMessage)
+```
+#### Example
+Below is a sample code to consume notifications.
+
+First, create the Oozie client and retrieve the JNDI properties to make a connection to the JMS server.
+
+```
+ OozieClient oc = new OozieClient("http://localhost:11000/oozie");
+ JMSConnectionInfo jmsInfo = oc.getJMSConnectionInfo();
+ Properties jndiProperties = jmsInfo.getJNDIProperties();
+ Context jndiContext = new InitialContext(jndiProperties);
+ String connectionFactoryName = (String) jndiContext.getEnvironment().get("connectionFactoryNames");
+ ConnectionFactory connectionFactory = (ConnectionFactory) jndiContext.lookup(connectionFactoryName);
+ Connection connection = connectionFactory.createConnection();
+ Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
+ String topicPrefix = jmsInfo.getTopicPrefix();
+ String topicPattern = jmsInfo.getTopicPattern(AppType.WORKFLOW_JOB);
+ // Following code checks if the topic pattern is
+ //'username', then the topic name is set to the actual user submitting the job
+ String topicName = null;
+ if (topicPattern.equals("${username}")) {
+ topicName = "john";
+ // Following code checks if the topic pattern is
+ //'jobId', then the topic name is set to the job id
+ } else if (topicPattern.equals("${jobId}")) {
+ topicName = "0000004-140328125200198-oozie-oozi-W";
+ }
+ Destination topic = session.createTopic(topicPrefix+topicName);
+ MessageConsumer consumer = session.createConsumer(topic);
+ consumer.setMessageListener(this);
+ connection.start();
+```
+
+To start receiving messages, the JMS [MessageListener](http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html)
+interface needs to be implemented. Also, its onMessage() method needs to be implemented.
+This method will be called whenever a message is available on the JMS bus.
+
+
+```
+ public void onMessage(Message message) {
+ if (message.getStringProperty(JMSHeaderConstants.MESSAGE_TYPE).equals(MessageType.SLA.name())){
+ SLAMessage slaMessage = JMSMessagingUtils.getEventMessage(message);
+ // Further processing
+ }
+ else if (message.getStringProperty(JMSHeaderConstants.APP_TYPE).equals(AppType.WORKFLOW_JOB.name())){
+ WorkflowJobMessage wfJobMessage = JMSMessagingUtils.getEventMessage(message);
+ // Further processing
+ }
+ }
+```
+
+#### Applying Selectors
+
+Below is a sample ActiveMQ text message header properties section.
+
+```
+ActiveMQTextMessage
+{properties = {appName = map-reduce-wf, msgType=JOB, appType=WORKFLOW_JOB, user=john, msgFormat=json, eventStatus=STARTED} ...}
+```
+
+On the header properties, consumers can apply JMS selectors to filter messages from JMS provider.
+They are listed at [JMSHeaderConstants](../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html)
+
+Sample use of selector to filter events related to Job which have failed and has a particular app-name
+
+
+```
+String selector=JMSHeaderConstants.EVENT_STATUS + "='FAILURE' AND " + JMSHeaderConstants.APP_NAME + "='app-name'";
+MessageConsumer consumer = session.createConsumer(topic, selector);
+```
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_Overview.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_Overview.md b/docs/src/site/markdown/DG_Overview.md
new file mode 100644
index 0000000..6a2b9d2
--- /dev/null
+++ b/docs/src/site/markdown/DG_Overview.md
@@ -0,0 +1,86 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Workflow Overview
+
+Oozie is a server based _Workflow Engine_ specialized in running workflow jobs with actions that run Hadoop Map/Reduce
+and Pig jobs.
+
+Oozie is a Java Web-Application that runs in a Java servlet-container.
+
+For the purposes of Oozie, a workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in
+a control dependency DAG (Directed Acyclic Graph). "control dependency" from one action to another means that the second
+action can't run until the first action has completed.
+
+Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to
+[JBOSS JBPM](http://www.jboss.org/jbossjbpm/) jPDL).
+
+Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems
+callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.
+
+Oozie uses a custom SecurityManager inside it's launcher to catch exit() calls from the user code. Make sure to delegate checkExit()
+calls to Oozie's SecurityManager if the user code uses its own SecurityManager. The Launcher also grants java.security.AllPermission
+by default to the user code.
+
+Oozie workflows contain control flow nodes and action nodes.
+
+Control flow nodes define the beginning and the end of a workflow ( `start`, `end` and `fail` nodes) and provide a
+mechanism to control the workflow execution path ( `decision`, `fork` and `join` nodes).
+
+Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie
+provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and
+Oozie sub-workflow. Oozie can be extended to support additional type of actions.
+
+Oozie workflows can be parameterized (using variables like `${inputDir}` within the workflow definition). When
+submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different
+output directories) several identical workflow jobs can concurrently.
+
+## WordCount Workflow Example
+
+**Workflow Diagram:**
+
+<img src="./DG_Overview.png"/>
+
+**hPDL Workflow Definition:**
+
+
+```
+<workflow-app name='wordcount-wf' xmlns="uri:oozie:workflow:0.1">
+ <start to='wordcount'/>
+ <action name='wordcount'>
+ <map-reduce>
+ <job-tracker>${jobTracker}</job-tracker>
+ <name-node>${nameNode}</name-node>
+ <configuration>
+ <property>
+ <name>mapred.mapper.class</name>
+ <value>org.myorg.WordCount.Map</value>
+ </property>
+ <property>
+ <name>mapred.reducer.class</name>
+ <value>org.myorg.WordCount.Reduce</value>
+ </property>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>${inputDir}</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>${outputDir}</value>
+ </property>
+ </configuration>
+ </map-reduce>
+ <ok to='end'/>
+ <error to='end'/>
+ </action>
+ <kill name='kill'>
+ <message>Something went wrong: ${wf:errorCode('wordcount')}</message>
+ </kill/>
+ <end name='end'/>
+</workflow-app>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_SLAMonitoring.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_SLAMonitoring.md b/docs/src/site/markdown/DG_SLAMonitoring.md
new file mode 100644
index 0000000..0831b93
--- /dev/null
+++ b/docs/src/site/markdown/DG_SLAMonitoring.md
@@ -0,0 +1,459 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie SLA Monitoring
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Overview
+
+Critical jobs can have certain SLA requirements associated with them. This SLA can be in terms of time
+i.e. a maximum allowed time limit associated with when the job should start, by when should it end,
+and its duration of run. Oozie workflows and coordinators allow defining such SLA limits in the application definition xml.
+
+With the addition of SLA Monitoring, Oozie can now actively monitor the state of these SLA-sensitive jobs
+and send out notifications for SLA mets and misses.
+
+In versions earlier than 4.x, this was a passive feature where users needed to query the Oozie client SLA API
+to fetch the records regarding job status changes, and use their own custom calculation engine to compute
+whether SLA was met or missed, based on initial definition of time limits.
+
+Oozie now also has a SLA tab in the Oozie UI, where users can query for SLA information and have a summarized view
+of how their jobs fared against their SLAs.
+
+
+## Oozie Server Configuration
+
+Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) for configuring Oozie server to track
+SLA for jobs and send notifications.
+
+## SLA Tracking
+
+Oozie allows tracking SLA for meeting the following criteria:
+
+ * Start time
+ * End time
+ * Job Duration
+
+### Event Status
+Corresponding to each of these 3 criteria, your jobs are processed for whether Met or Miss i.e.
+
+ * START_MET, START_MISS
+ * END_MET, END_MISS
+ * DURATION_MET, DURATION_MISS
+
+### SLA Status
+Expected end-time is the most important criterion for majority of users while deciding overall SLA Met or Miss.
+Hence the _"SLA_Status"_ for a job will transition through these four stages
+
+ * Not_Started <-- Job not yet begun
+ * In_Process <-- Job started and is running, and SLAs are being tracked
+ * Met <-- caused by an END_MET
+ * Miss <-- caused by an END_MISS
+
+In addition to overshooting expected end-time, and END_MISS (and so an eventual SLA MISS) also occurs when the
+job does not end successfully e.g. goes to error state - Failed/Killed/Error/Timedout.
+
+## Configuring SLA in Applications
+
+To make your jobs trackable for SLA, you simply need to add the `<sla:info>` tag to your workflow application definition.
+If you were already using the existing SLA schema
+in your workflows (Schema xmlns:sla="uri:oozie:sla:0.1"), you don't need to
+do anything extra to receive SLA notifications via JMS messages. This new SLA monitoring framework is backward-compatible -
+no need to change application XML for now and you can continue to fetch old records via the [command line API](DG_CommandLineTool.html#SLAOperations).
+However, usage of old schema
+and API is deprecated and we strongly recommend using new schema.
+
+ * New SLA schema
+is 'uri:oozie:sla:0.2'
+ * In order to use new SLA schema,
+ you will need to upgrade your workflow/coordinator schema
+to 0.5 i.e. 'uri:oozie:workflow:0.5'
+
+### SLA Definition in Workflow
+Example:
+
+```
+<workflow-app name="test-wf-job-sla"
+ xmlns="uri:oozie:workflow:0.5"
+ xmlns:sla="uri:oozie:sla:0.2">
+ <start to="grouper"/>
+ <action name="grouper">
+ <map-reduce>
+ <job-tracker>jt</job-tracker>
+ <name-node>nn</name-node>
+ <configuration>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>input</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>output</value>
+ </property>
+ </configuration>
+ </map-reduce>
+ <ok to="end"/>
+ <error to="end"/>
+ </action>
+ <end name="end"/>
+ <sla:info>
+ <sla:nominal-time>${nominal_time}</sla:nominal-time>
+ <sla:should-start>${10 * MINUTES}</sla:should-start>
+ <sla:should-end>${30 * MINUTES}</sla:should-end>
+ <sla:max-duration>${30 * MINUTES}</sla:max-duration>
+ <sla:alert-events>start_miss,end_miss,duration_miss</sla:alert-events>
+ <sla:alert-contact>joe@example.com</sla:alert-contact>
+ </sla:info>
+</workflow-app>
+```
+
+For the list of tags usable under `<sla:info>`, refer to [Schemas Appendix](WorkflowFunctionalSpec.html#SLASchema).
+This new schema
+is much more compact and meaningful, getting rid of redundant and unused tags.
+
+ * `nominal-time`: As the name suggests, this is the time relative to which your jobs' SLAs will be calculated. Generally since Oozie workflows are aligned with synchronous data dependencies, this nominal time can be parameterized to be passed the value of your coordinator nominal time. Nominal time is also required in case of independent workflows and you can specify the time in which you expect the workflow to be run if you don't have a synchronous dataset associated with it.
+ * `should-start`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *start running* to meet SLA. This is optional.
+ * `should-end`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *finish* to meet SLA.
+ * `max-duration`: This is the maximum amount of time (along with time-unit - MINUTES, HOURS, DAYS) your job is expected to run. This is optional.
+ * `alert-events`: Specify the types of events for which **Email** alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not necessary. However, note that this setting is only for alerts via *email* alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured.
+ * `alert-contact`: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts.
+
+NOTE: All tags can be parameterized as a EL function or a fixed value.
+
+Same schema
+can be applied to and embedded under Workflow-Action as well as Coordinator-Action XML.
+
+### SLA Definition in Workflow Action
+
+
+```
+<workflow-app name="test-wf-action-sla" xmlns="uri:oozie:workflow:0.5" xmlns:sla="uri:oozie:sla:0.2">
+ <start to="grouper"/>
+ <action name="grouper">
+ ...
+ <ok to="end"/>
+ <error to="end"/>
+ <sla:info>
+ <sla:nominal-time>${nominal_time}</sla:nominal-time>
+ <sla:should-start>${10 * MINUTES}</sla:should-start>
+ ...
+ </sla:info>
+ </action>
+ <end name="end"/>
+</workflow-app>
+```
+
+### SLA Definition in Coordinator Action
+
+```
+<coordinator-app name="test-coord-sla" frequency="${coord:days(1)}" freq_timeunit="DAY"
+ end_of_duration="NONE" start="2013-06-20T08:01Z" end="2013-12-01T08:01Z"
+ timezone="America/Los_Angeles" xmlns="uri:oozie:coordinator:0.4" xmlns:sla="uri:oozie:sla:0.2">
+ <action>
+ <workflow>
+ <app-path>${wfAppPath}</app-path>
+ </workflow>
+ <sla:info>
+ <sla:nominal-time>${nominal_time}</sla:nominal-time>
+ ...
+ </sla:info>
+ </action>
+</coordinator-app>
+```
+
+## Accessing SLA Information
+
+SLA information is accessible via the following ways:
+
+ * Through the SLA tab of the Oozie Web UI.
+ * JMS messages sent to a configured JMS provider for instantaneous tracking.
+ * RESTful API to query for SLA summary.
+ * As an `Instrumentation.Counter` entry that is accessible via RESTful API and reflects to the number of all SLA tracked external
+ entities. Name of this counter is `sla-calculator.sla-map`.
+
+For JMS Notifications, you have to have a message broker in place, on which Oozie publishes messages and you can
+hook on a subscriber to receive those messages. For more info on setting up and consuming JMS messages, refer
+[JMS Notifications](DG_JMSNotifications.html) documentation.
+
+In the REST API, the following filters can be applied while fetching SLA information:
+
+ * app_name - Application name
+ * id - id of the workflow job, workflow action or coordinator action
+ * parent_id - Parent id of the workflow job, workflow action or coordinator action
+ * nominal_start and nominal_end - Start and End range for nominal time of the workflow or coordinator.
+ * bundle - Bundle Job ID or Bundle App Name. Fetches SLA information for actions of all coordinators in that bundle.
+ * event_status - event status such as START_MET/START_MISS/DURATION_MET/DURATION_MISS/END_MET/END_MISS
+ * sla_status - sla status such as NOT_STARTED/IN_PROCESS/MET/MISS
+
+multiple event_status and sla_status can be specified with comma separation. When multiple statuses are specified, they are considered as OR.
+For example, event_status=START_MET;END_MISS list the coordinator actions where event status is either START_MET OR END_MISS.
+
+When timezone query parameter is specified, the expected and actual start/end time returned is formatted. If not specified,
+the number of milliseconds that have elapsed since January 1, 1970 00:00:00.000 GMT is returned.
+
+The examples below demonstrate the use of REST API and explains the JSON response.
+
+### Scenario 1: Workflow Job Start_Miss
+**Request:**
+
+```
+GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=nominal_start=2013-06-18T00:01Z;nominal_end=2013-06-23T00:01Z;app_name=my-sla-app
+```
+
+**JSON Response**
+
+```
+{
+
+ id : "000056-1238791320234-oozie-joe-W"
+ parentId : "000001-1238791320234-oozie-joe-C@8"
+ appType : "WORKFLOW_JOB"
+ msgType : "SLA"
+ appName : "my-sla-app"
+ slaStatus : "IN_PROCESS"
+ jobStatus : "RUNNING"
+ user: "joe"
+ nominalTime: "2013-16-22T05:00Z"
+ expectedStartTime: "2013-16-22T05:10Z" <-- (should start by this time)
+ actualStartTime: "2013-16-22T05:30Z" <-- (20 min late relative to expected start)
+ expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time)
+ actualEndTime: null
+ expectedDuration: 900000 <-- (expected duration in milliseconds)
+ actualDuration: 120000 <-- (actual duration in milliseconds)
+ notificationMessage: "My Job has encountered an SLA event!"
+ upstreamApps: "dependent-app-1, dependent-app-2"
+
+}
+```
+
+### Scenario 2: Workflow Action End_Miss
+**Request:**
+
+```
+GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=parent_id=000056-1238791320234-oozie-joe-W
+```
+
+**JSON Response**
+
+```
+{
+
+ id : "000056-1238791320234-oozie-joe-W@map-reduce-action"
+ parentId : "000056-1238791320234-oozie-joe-W"
+ appType : "WORKFLOW_ACTION"
+ msgType : "SLA"
+ appName : "map-reduce-action"
+ slaStatus : "MISS"
+ jobStatus : "SUCCEEDED"
+ user: "joe"
+ nominalTime: "2013-16-22T05:00Z"
+ expectedStartTime: "2013-16-22T05:10Z"
+ actualStartTime: "2013-16-22T05:05Z"
+ expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time)
+ actualEndTime: "2013-16-22T06:00Z" <-- (20 min late relative to expected end)
+ expectedDuration: 3600000 <-- (expected duration in milliseconds)
+ actualDuration: 3300000 <-- (actual duration in milliseconds)
+ notificationMessage: "My Job has encountered an SLA event!"
+ upstreamApps: "dependent-app-1, dependent-app-2"
+
+}
+```
+
+### Scenario 3: Coordinator Action Duration_Miss
+**Request:**
+
+```
+GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=id=000001-1238791320234-oozie-joe-C
+```
+
+**JSON Response**
+
+```
+{
+
+ id : "000001-1238791320234-oozie-joe-C@2"
+ parentId : "000001-1238791320234-oozie-joe-C"
+ appType : "COORDINATOR_ACTION"
+ msgType : "SLA"
+ appName : "my-coord-app"
+ slaStatus : "MET"
+ jobStatus : "SUCCEEDED"
+ user: "joe"
+ nominalTime: "2013-16-22T05:00Z"
+ expectedStartTime: "2013-16-22T05:10Z"
+ actualStartTime: "2013-16-22T05:05Z"
+ expectedEndTime: "2013-16-22T05:40Z"
+ actualEndTime: "2013-16-22T05:30Z"
+ expectedDuration: 900000 <-- (expected duration in milliseconds)
+ actualDuration: 1500000 <- (actual duration in milliseconds)
+ notificationMessage: "My Job has encountered an SLA event!"
+ upstreamApps: "dependent-app-1, dependent-app-2"
+
+}
+```
+
+Scenario #3 is particularly interesting because it is an overall "MET" because it met its expected End-time,
+but it is "Duration_Miss" because the actual run (between actual start and actual end) exceeded expected duration.
+
+### Scenario 4: All Coordinator actions in a Bundle
+**Request:**
+
+```
+GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=bundle=1234567-150130225116604-oozie-B;event_status=END_MISS
+```
+
+**JSON Response**
+
+```
+{
+ id : "000001-1238791320234-oozie-joe-C@1"
+ parentId : "000001-1238791320234-oozie-joe-C"
+ appType : "COORDINATOR_ACTION"
+ msgType : "SLA"
+ appName : "my-coord-app"
+ slaStatus : "MET"
+ eventStatus : "START_MET,DURATION_MISS,END_MISS"
+ user: "joe"
+ nominalTime: "2014-01-10T12:00Z"
+ expectedStartTime: "2014-01-10T12:00Z"
+ actualStartTime: "2014-01-10T11:59Z"
+ startDelay: -1
+ expectedEndTime: "2014-01-10T13:00Z"
+ actualEndTime: "2014-01-10T13:05Z"
+ endDelay: 5
+ expectedDuration: 3600000 <-- (expected duration in milliseconds)
+ actualDuration: 3960000 <-- (actual duration in milliseconds)
+ durationDelay: 6 <-- (duration delay in minutes)
+}
+{
+ id : "000001-1238791320234-oozie-joe-C@2"
+ parentId : "000001-1238791320234-oozie-joe-C"
+ appType : "COORDINATOR_ACTION"
+ msgType : "SLA"
+ appName : "my-coord-app"
+ slaStatus : "MET"
+ eventStatus : "START_MISS,DURATION_MET,END_MISS"
+ user: "joe"
+ nominalTime: "2014-01-11T12:00Z"
+ expectedStartTime: "2014-01-11T12:00Z"
+ actualStartTime: "2014-01-11T12:05Z"
+ startDelay: 5
+ expectedEndTime: "2014-01-11T13:00Z"
+ actualEndTime: "2014-01-11T13:01Z"
+ endDelay: 1
+ expectedDuration: 3600000 <-- (expected duration in milliseconds)
+ actualDuration: 3360000 <-- (actual duration in milliseconds)
+ durationDelay: -4 <-- (duration delay in minutes)
+}
+```
+
+Scenario #4 (All Coordinator actions in a Bundle) is to get SLA information of all coordinator actions under bundle job in one call.
+startDelay/durationDelay/endDelay values returned indicate how much delay compared to expected time (positive values in case of MISS, and negative values in case of MET).
+
+### Sample Email Alert
+
+```
+Subject: OOZIE - SLA END_MISS (AppName=wf-sla-job, JobID=0000004-130610225200680-oozie-oozi-W)
+
+
+Status:
+ SLA Status - END_MISS
+ Job Status - RUNNING
+ Notification Message - Missed SLA for Data Pipeline job
+Job Details:
+ App Name - wf-sla-job
+ App Type - WORKFLOW_JOB
+ User - strat_ci
+ Job ID - 0000004-130610225200680-oozie-oozi-W
+ Job URL - http://host.domain.com:4080/oozie//?job=0000004-130610225200680-oozie-oozi-W
+ Parent Job ID - N/A
+ Parent Job URL - N/A
+ Upstream Apps - wf-sla-up-app
+SLA Details:
+ Nominal Time - Mon Jun 10 23:33:00 UTC 2013
+ Expected Start Time - Mon Jun 10 23:35:00 UTC 2013
+ Actual Start Time - Mon Jun 10 23:34:04 UTC 2013
+ Expected End Time - Mon Jun 10 23:38:00 UTC 2013
+ Expected Duration (in mins) - 5
+ Actual Duration (in mins) - -1
+```
+
+### Changing job SLA definition and alerting
+Following are ways to enable/disable SLA alerts for coordinator actions.
+
+#### 1. Specify in Bundle XML during submission.
+Following properties can be specified in bundle xml as properties for coordinator.
+
+`oozie.sla.disable.alerts.older.than` this property can be specified in hours, the SLA notification for
+coord actions will be disabled whose nominal is time older then this value. Default is 48 hours.
+
+```
+<property>
+ <name>oozie.sla.disable.alerts.older.than</name>
+ <value>12</value>
+</property>
+```
+
+`oozie.sla.disable.alerts` List of coord actions to be disabled. Value can be specified as list of coord actions or date range.
+
+```
+<property>
+ <name>oozie.sla.disable.alerts</name>
+ <value>1,3-4,7-10</value>
+</property>
+```
+Will disable alert for coord actions 1,3,5,7,8,9,10
+
+`oozie.sla.enable.alerts` List of coord actions to be enabled. Value can be specified as list of coord actions or date range.
+
+```
+<property>
+ <name>oozie.sla.enable.alerts</name>
+ <value>2009-01-01T01:00Z::2009-05-31T23:59Z</value>
+</property>
+```
+This will enable SLA alert for coord actions whose nominal time is in between (inclusive) 2009-01-01T01:00Z and 2009-05-31T23:59Z.
+
+ALL keyword can be specified to specify all actions. Below property will disable SLA notifications for all coord actions.
+
+```
+<property>
+ <name>oozie.sla.disable.alerts</name>
+ <value>ALL</value>
+</property>
+```
+
+#### 2. Specify during Coordinator job submission or update
+Above properties can be specified in job.properties in
+[Coord job update command](DG_CommandLineTool.html#Updating_coordinator_definition_and_properties),
+in [Coord job submit command](DG_CommandLineTool.html#Submitting_a_Workflow_Coordinator_or_Bundle_Job)
+or in [Coord job run command](DG_CommandLineTool.html#Running_a_Workflow_Coordinator_or_Bundle_Job)
+
+#### 3. Change using command line
+Refer [Changing job SLA definition and alerting](DG_CommandLineTool.html#Changing_job_SLA_definition_and_alerting) for commandline usage.
+
+#### 4. Change using REST API
+Refer the REST API [Changing job SLA definition and alerting](WebServicesAPI.html#Changing_job_SLA_definition_and_alerting).
+
+## In-memory SLA entries and database content
+
+There are special circumstances when the in-memory `SLACalcStatus` entries can exist without the workflow or coordinator job or
+action instances in database. For example:
+
+ * SLA tracked database content may already have been deleted, and `SLA_SUMMARY` entry is not present anymore in database
+ * SLA tracked database content and `SLA_SUMMARY` entry aren't yet present in database
+
+By the time `SLAService` scheduled job will be running, SLA map contents are checked. When the `SLA_SUMMARY` entry for the in-memory
+SLA entry is missing, a counter is increased. When this counter reaches the server-wide preconfigured value
+`oozie.sla.service.SLAService.maximum.retry.count` (by default `3`), in-memory SLA entry will get purged.
+
+## Known issues
+There are two known issues when you define SLA for a workflow action.
+ * If there are decision nodes and SLA is defined for a workflow action not in the execution path because of the decision node, you will still get an SLA_MISS notification.
+ * If you have dangling action nodes in your workflow definition and SLA is defined for it, you will still get an SLA_MISS notification.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_ShellActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_ShellActionExtension.md b/docs/src/site/markdown/DG_ShellActionExtension.md
new file mode 100644
index 0000000..eff4b08
--- /dev/null
+++ b/docs/src/site/markdown/DG_ShellActionExtension.md
@@ -0,0 +1,494 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Shell Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+<a name="ShellAction"></a>
+## Shell Action
+
+The `shell` action runs a Shell command.
+
+The workflow job will wait until the Shell command completes before
+continuing to the next action.
+
+To run the Shell job, you have to configure the `shell` action with the
+`job-tracker`, `name-node` and Shell `exec` elements as
+well as the necessary arguments and configuration.
+
+A `shell` action can be configured to create or delete HDFS directories
+before starting the Shell job.
+
+Shell _launcher_ configuration can be specified with a file, using the `job-xml`
+element, and inline, using the `configuration` elements.
+
+Oozie EL expressions can be used in the inline configuration. Property
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
+
+Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties
+must not be present in the inline configuration.
+
+As with Hadoop `map-reduce` jobs, it is possible to add files and
+archives in order to make them available to the Shell job. Refer to the
+[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
+section for more information about this feature.
+
+The output (STDOUT) of the Shell job can be made available to the workflow job after the Shell job ends. This information
+could be used from within decision nodes. If the output of the Shell job is made available to the workflow job the shell
+command must follow the following requirements:
+
+ * The format of the output must be a valid Java Properties file.
+ * The size of the output must not exceed 2KB.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <shell xmlns="uri:oozie:shell-action:1.0">
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[SHELL SETTINGS FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <exec>[SHELL-COMMAND]</exec>
+ <argument>[ARG-VALUE]</argument>
+ ...
+ <argument>[ARG-VALUE]</argument>
+ <env-var>[VAR1=VALUE1]</env-var>
+ ...
+ <env-var>[VARN=VALUEN]</env-var>
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ <capture-output/>
+ </shell>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
+
+The `job-xml` element, if present, specifies a file containing configuration
+for the Shell job. As of schema 0.2, multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
+
+The `configuration` element, if present, contains configuration
+properties that are passed to the Shell job.
+
+The `exec` element must contain the path of the Shell command to
+execute. The arguments of Shell command can then be specified
+using one or more `argument` element.
+
+The `argument` element, if present, contains argument to be passed to
+the Shell command.
+
+The `env-var` element, if present, contains the environment to be passed
+to the Shell command. `env-var` should contain only one pair of environment variable
+and value. If the pair contains the variable such as $PATH, it should follow the
+Unix convention such as PATH=$PATH:mypath. Don't use ${PATH} which will be
+substituted by Oozie's EL evaluator.
+
+A `shell` action creates a Hadoop configuration. The Hadoop configuration is made available as a local file to the
+Shell application in its running directory. The exact file path is exposed to the spawned shell using the environment
+variable called `OOZIE_ACTION_CONF_XML`.The Shell application can access the environment variable to read the action
+configuration XML file path.
+
+If the `capture-output` element is present, it indicates Oozie to capture output of the STDOUT of the shell command
+execution. The Shell command output must be in Java Properties file format and it must not exceed 2KB. From within the
+workflow definition, the output of an Shell action node is accessible via the `String action:output(String node,
+String key)` function (Refer to section '4.2.6 Action EL Functions').
+
+All the above elements can be parameterized (templatized) using EL
+expressions.
+
+**Example:**
+
+How to run any shell script or perl script or CPP executable
+
+
+```
+<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'>
+ <start to='shell1' />
+ <action name='shell1'>
+ <shell xmlns="uri:oozie:shell-action:1.0">
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode}</name-node>
+ <configuration>
+ <property>
+ <name>mapred.job.queue.name</name>
+ <value>${queueName}</value>
+ </property>
+ </configuration>
+ <exec>${EXEC}</exec>
+ <argument>A</argument>
+ <argument>B</argument>
+ <file>${EXEC}#${EXEC}</file> <!--Copy the executable to compute node's current working directory -->
+ </shell>
+ <ok to="end" />
+ <error to="fail" />
+ </action>
+ <kill name="fail">
+ <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
+ </kill>
+ <end name='end' />
+</workflow-app>
+```
+
+The corresponding job properties file used to submit Oozie job could be as follows:
+
+
+```
+oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script
+
+#Execute is expected to be in the Workflow directory.
+#Shell Script to run
+EXEC=script.sh
+#CPP executable. Executable should be binary compatible to the compute node OS.
+#EXEC=hello
+#Perl script
+#EXEC=script.pl
+
+resourceManager=localhost:8032
+nameNode=hdfs://localhost:8020
+queueName=default
+
+```
+
+How to run any java program bundles in a jar.
+
+
+```
+<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'>
+ <start to='shell1' />
+ <action name='shell1'>
+ <shell xmlns="uri:oozie:shell-action:1.0">
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode}</name-node>
+ <configuration>
+ <property>
+ <name>mapred.job.queue.name</name>
+ <value>${queueName}</value>
+ </property>
+ </configuration>
+ <exec>java</exec>
+ <argument>-classpath</argument>
+ <argument>./${EXEC}:$CLASSPATH</argument>
+ <argument>Hello</argument>
+ <file>${EXEC}#${EXEC}</file> <!--Copy the jar to compute node current working directory -->
+ </shell>
+ <ok to="end" />
+ <error to="fail" />
+ </action>
+ <kill name="fail">
+ <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
+ </kill>
+ <end name='end' />
+</workflow-app>
+```
+
+The corresponding job properties file used to submit Oozie job could be as follows:
+
+
+```
+oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script
+
+#Hello.jar file is expected to be in the Workflow directory.
+EXEC=Hello.jar
+
+resourceManager=localhost:8032
+nameNode=hdfs://localhost:8020
+queueName=default
+```
+
+### Shell Action Configuration
+
+ * `oozie.action.shell.setup.hadoop.conf.dir` - Generates a config directory with various core/hdfs/yarn/mapred-site.xml files and points `HADOOP_CONF_DIR` and `YARN_CONF_DIR` env-vars to it, before the Script is invoked. XML is sourced from the action configuration. Useful when the Shell script passed uses various `hadoop` commands. Default is false.
+ * `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` - When `oozie.action.shell.setup.hadoop.conf.dir` is enabled, toggle if a log4j.properties file should also be written under the configuration files directory. Default is true.
+ * `oozie.action.shell.setup.hadoop.conf.dir.log4j.content` - When `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` is enabled, the content to write into the log4j.properties file under the configuration files directory. Default is a simple console based stderr logger, as presented below:
+
+```
+log4j.rootLogger=${hadoop.root.logger}
+hadoop.root.logger=INFO,console
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+log4j.appender.console.target=System.err
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
+```
+
+### Shell Action Logging
+
+Shell action's stdout and stderr output are redirected to the Oozie Launcher map-reduce job task STDOUT that runs the shell command.
+
+From Oozie web-console, from the Shell action pop up using the 'Console URL' link, it is possible
+to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
+
+### Shell Action Limitations
+Although Shell action can execute any shell command, there are some limitations.
+
+ * No interactive command is supported.
+ * Command can't be executed as different user using sudo.
+ * User has to explicitly upload the required 3rd party packages (such as jar, so lib, executable etc). Oozie provides a way using \<file\> and \<archive\> tag through Hadoop's Distributed Cache to upload.
+ * Since Oozie will execute the shell command into a Hadoop compute node, the default installation of utility in the compute node might not be fixed. However, the most common unix utilities are usually installed on all compute nodes. It is important to note that Oozie could only support the commands that are installed into the compute nodes or that are uploaded through Distributed Cache.
+
+## Appendix, Shell XML-Schema
+
+### AE.A Appendix A, Shell XML-Schema
+
+#### Shell Action Schema Version 1.0
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:shell="uri:oozie:shell-action:1.0"
+ elementFormDefault="qualified"
+ targetNamespace="uri:oozie:shell-action:1.0">
+.
+ <xs:include schemaLocation="oozie-common-1.0.xsd"/>
+.
+ <xs:element name="shell" type="shell:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:choice>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="launcher" type="shell:LAUNCHER" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Shell Action Schema Version 0.3
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:shell="uri:oozie:shell-action:0.3" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:shell-action:0.3">
+
+ <xs:element name="shell" type="shell:ACTION"/>
+
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="FLAG"/>
+
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+</xs:schema>
+```
+
+#### Shell Action Schema Version 0.2
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:shell="uri:oozie:shell-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:shell-action:0.2">
+
+ <xs:element name="shell" type="shell:ACTION"/>
+
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="FLAG"/>
+
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+</xs:schema>
+```
+
+#### Shell Action Schema Version 0.1
+
+```
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:shell="uri:oozie:shell-action:0.1" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:shell-action:0.1">
+
+ <xs:element name="shell" type="shell:ACTION"/>
+
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="FLAG"/>
+
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+
+</xs:schema>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
[12/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/WorkflowFunctionalSpec.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/WorkflowFunctionalSpec.md b/docs/src/site/markdown/WorkflowFunctionalSpec.md
new file mode 100644
index 0000000..463635b
--- /dev/null
+++ b/docs/src/site/markdown/WorkflowFunctionalSpec.md
@@ -0,0 +1,5457 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Specification, a Hadoop Workflow System
+**<center>(v5.0)</center>**
+
+The goal of this document is to define a workflow engine system specialized in coordinating the execution of Hadoop
+Map/Reduce and Pig jobs.
+
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Changelog
+
+
+**2016FEB19**
+
+ * 3.2.7 Updated notes on System.exit(int n) behavior
+
+**2015APR29**
+
+ * 3.2.1.4 Added notes about Java action retries
+ * 3.2.7 Added notes about Java action retries
+
+**2014MAY08**
+
+ * 3.2.2.4 Added support for fully qualified job-xml path
+
+**2013JUL03**
+
+ * Appendix A, Added new workflow schema 0.5 and SLA schema 0.2
+
+**2012AUG30**
+
+ * 4.2.2 Added two EL functions (replaceAll and appendAll)
+
+**2012JUL26**
+
+ * Appendix A, updated XML schema 0.4 to include `parameters` element
+ * 4.1 Updated to mention about `parameters` element as of schema 0.4
+
+**2012JUL23**
+
+ * Appendix A, updated XML schema 0.4 (Fs action)
+ * 3.2.4 Updated to mention that a `name-node`, a `job-xml`, and a `configuration` element are allowed in the Fs action as of
+schema 0.4
+
+**2012JUN19**
+
+ * Appendix A, added XML schema 0.4
+ * 3.2.2.4 Updated to mention that multiple `job-xml` elements are allowed as of schema 0.4
+ * 3.2.3 Updated to mention that multiple `job-xml` elements are allowed as of schema 0.4
+
+**2011AUG17**
+
+ * 3.2.4 fs 'chmod' xml closing element typo in Example corrected
+
+**2011AUG12**
+
+ * 3.2.4 fs 'move' action characteristics updated, to allow for consistent source and target paths and existing target path only if directory
+ * 18, Update the doc for user-retry of workflow action.
+
+**2011FEB19**
+
+ * 10, Update the doc to rerun from the failed node.
+
+**2010OCT31**
+
+ * 17, Added new section on Shared Libraries
+
+**2010APR27**
+
+ * 3.2.3 Added new "arguments" tag to PIG actions
+ * 3.2.5 SSH actions are deprecated in Oozie schema 0.1 and removed in Oozie schema 0.2
+ * Appendix A, Added schema version 0.2
+
+
+**2009OCT20**
+
+ * Appendix A, updated XML schema
+
+
+**2009SEP15**
+
+ * 3.2.6 Removing support for sub-workflow in a different Oozie instance (removing the 'oozie' element)
+
+
+**2009SEP07**
+
+ * 3.2.2.3 Added Map Reduce Pipes specifications.
+ * 3.2.2.4 Map-Reduce Examples. Previously was 3.2.2.3.
+
+
+**2009SEP02**
+
+ * 10 Added missing skip nodes property name.
+ * 3.2.1.4 Reworded action recovery explanation.
+
+
+**2009AUG26**
+
+ * 3.2.9 Added `java` action type
+ * 3.1.4 Example uses EL constant to refer to counter group/name
+
+
+**2009JUN09**
+
+ * 12.2.4 Added build version resource to admin end-point
+ * 3.2.6 Added flag to propagate workflow configuration to sub-workflows
+ * 10 Added behavior for workflow job parameters given in the rerun
+ * 11.3.4 workflows info returns pagination information
+
+
+**2009MAY18**
+
+ * 3.1.4 decision node, 'default' element, 'name' attribute changed to 'to'
+ * 3.1.5 fork node, 'transition' element changed to 'start', 'to' attribute change to 'path'
+ * 3.1.5 join node, 'transition' element remove, added 'to' attribute to 'join' element
+ * 3.2.1.4 Rewording on action recovery section
+ * 3.2.2 map-reduce action, added 'job-tracker', 'name-node' actions, 'file', 'file' and 'archive' elements
+ * 3.2.2.1 map-reduce action, remove from 'streaming' element 'file', 'file' and 'archive' elements
+ * 3.2.2.2 map-reduce action, reorganized streaming section
+ * 3.2.3 pig action, removed information about implementation (SSH), changed elements names
+ * 3.2.4 fs action, removed 'fs-uri' and 'user-name' elements, file system URI is now specified in path, user is propagated
+ * 3.2.6 sub-workflow action, renamed elements 'oozie-url' to 'oozie' and 'workflow-app' to 'app-path'
+ * 4 Properties that are valid Java identifiers can be used as ${NAME}
+ * 4.1 Renamed default properties file from 'configuration.xml' to 'default-configuration.xml'
+ * 4.2 Changes in EL Constants and Functions
+ * 5 Updated notification behavior and tokens
+ * 6 Changed user propagation behavior
+ * 7 Changed application packaging from ZIP to HDFS directory
+ * Removed application lifecycle and self containment model sections
+ * 10 Changed workflow job recovery, simplified recovery behavior
+ * 11 Detailed Web Services API
+ * 12 Updated Client API section
+ * 15 Updated Action Executor API section
+ * Appendix A XML namespace updated to 'uri:oozie:workflow:0.1'
+ * Appendix A Updated XML schema to changes in map-reduce/pig/fs/ssh actions
+ * Appendix B Updated workflow example to schema changes
+
+
+**2009MAR25**
+
+ * Changing all references of HWS to Oozie (project name)
+ * Typos, XML Formatting
+ * XML Schema URI correction
+
+
+**2009MAR09**
+
+ * Changed `CREATED` job state to `PREP` to have same states as Hadoop
+ * Renamed 'hadoop-workflow' element to 'workflow-app'
+ * Decision syntax changed to be 'switch/case' with no transition indirection
+ * Action nodes common root element 'action', with the action type as sub-element (using a single built-in XML schema)
+ * Action nodes have 2 explicit transitions 'ok to' and 'error to' enforced by XML schema
+ * Renamed 'fail' action element to 'kill'
+ * Renamed 'hadoop' action element to 'map-reduce'
+ * Renamed 'hdfs' action element to 'fs'
+ * Updated all XML snippets and examples
+ * Made user propagation simpler and consistent
+ * Added Oozie XML schema to Appendix A
+ * Added workflow example to Appendix B
+
+
+**2009FEB22**
+
+ * Opened [JIRA HADOOP-5303](https://issues.apache.org/jira/browse/HADOOP-5303)
+
+
+**27/DEC/2012:**
+
+ * Added information on dropping hcatalog table partitions in prepare block
+ * Added hcatalog EL functions section
+
+## 0 Definitions
+
+**Action:** An execution/computation task (Map-Reduce job, Pig job, a shell command). It can also be referred as task or
+'action node'.
+
+**Workflow:** A collection of actions arranged in a control dependency DAG (Directed Acyclic Graph). "control dependency"
+from one action to another means that the second action can't run until the first action has completed.
+
+**Workflow Definition:** A programmatic description of a workflow that can be executed.
+
+**Workflow Definition Language:** The language used to define a Workflow Definition.
+
+**Workflow Job:** An executable instance of a workflow definition.
+
+**Workflow Engine:** A system that executes workflows jobs. It can also be referred as a DAG engine.
+
+## 1 Specification Highlights
+
+A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, and
+sub-workflows.
+
+Flow control operations within the workflow applications can be done using decision, fork and join nodes. Cycles in
+workflows are not supported.
+
+Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow
+definition as `${VAR}` variables.
+
+A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to
+run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig
+scripts, and other resource files.
+
+Before running a workflow job, the corresponding workflow application must be deployed in Oozie.
+
+Deploying workflow application and running workflow jobs can be done via command line tools, a WS API and a Java API.
+
+Monitoring the system and workflow jobs can be done via a web console, command line tools, a WS API and a Java API.
+
+When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions
+must be provided. This set of properties is a Hadoop configuration.
+
+Possible states for a workflow jobs are: `PREP`, `RUNNING`, `SUSPENDED`, `SUCCEEDED`, `KILLED` and `FAILED`.
+
+In the case of a action start failure in a workflow job, depending on the type of failure, Oozie will attempt automatic
+retries, it will request a manual retry or it will fail the workflow job.
+
+Oozie can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.
+
+In the case of workflow job failure, the workflow job can be resubmitted skipping previously completed actions.
+Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow
+application code.
+
+<a name="WorkflowDefinition"></a>
+## 2 Workflow Definition
+
+A workflow definition is a DAG with control flow nodes (start, end, decision, fork, join, kill) or action nodes
+(map-reduce, pig, etc.), nodes are connected by transitions arrows.
+
+The workflow definition language is XML based and it is called hPDL (Hadoop Process Definition Language).
+
+Refer to the Appendix A for the[Oozie Workflow Definition XML Schema](WorkflowFunctionalSpec.html#OozieWFSchema). Appendix
+B has [Workflow Definition Examples](WorkflowFunctionalSpec.html#OozieWFExamples).
+
+### 2.1 Cycles in Workflow Definitions
+
+Oozie does not support cycles in workflow definitions, workflow definitions must be a strict DAG.
+
+At workflow application deployment time, if Oozie detects a cycle in the workflow definition it must fail the
+deployment.
+
+## 3 Workflow Nodes
+
+Workflow nodes are classified in control flow nodes and action nodes:
+
+ * **Control flow nodes:** nodes that control the start and end of the workflow and workflow job execution path.
+ * **Action nodes:** nodes that trigger the execution of a computation/processing task.
+
+Node names and transitions must be conform to the following pattern `[a-zA-Z][\-_a-zA-Z0-0]*`, of up to 20 characters
+long.
+
+### 3.1 Control Flow Nodes
+
+Control flow nodes define the beginning and the end of a workflow (the `start`, `end` and `kill` nodes) and provide a
+mechanism to control the workflow execution path (the `decision`, `fork` and `join` nodes).
+
+<a name="StartNode"></a>
+#### 3.1.1 Start Control Node
+
+The `start` node is the entry point for a workflow job, it indicates the first workflow node the workflow job must
+transition to.
+
+When a workflow is started, it automatically transitions to the node specified in the `start`.
+
+A workflow definition must have one `start` node.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <start to="[NODE-NAME]"/>
+ ...
+</workflow-app>
+```
+
+The `to` attribute is the name of first workflow node to execute.
+
+**Example:**
+
+
+```
+<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <start to="firstHadoopJob"/>
+ ...
+</workflow-app>
+```
+
+<a name="EndNode"></a>
+#### 3.1.2 End Control Node
+
+The `end` node is the end for a workflow job, it indicates that the workflow job has completed successfully.
+
+When a workflow job reaches the `end` it finishes successfully (SUCCEEDED).
+
+If one or more actions started by the workflow job are executing when the `end` node is reached, the actions will be
+killed. In this scenario the workflow job is still considered as successfully run.
+
+A workflow definition must have one `end` node.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <end name="[NODE-NAME]"/>
+ ...
+</workflow-app>
+```
+
+The `name` attribute is the name of the transition to do to end the workflow job.
+
+**Example:**
+
+
+```
+<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <end name="end"/>
+</workflow-app>
+```
+
+<a name="KillNode"></a>
+#### 3.1.3 Kill Control Node
+
+The `kill` node allows a workflow job to kill itself.
+
+When a workflow job reaches the `kill` it finishes in error (KILLED).
+
+If one or more actions started by the workflow job are executing when the `kill` node is reached, the actions will be
+killed.
+
+A workflow definition may have zero or more `kill` nodes.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <kill name="[NODE-NAME]">
+ <message>[MESSAGE-TO-LOG]</message>
+ </kill>
+ ...
+</workflow-app>
+```
+
+The `name` attribute in the `kill` node is the name of the Kill action node.
+
+The content of the `message` element will be logged as the kill reason for the workflow job.
+
+A `kill` node does not have transition elements because it ends the workflow job, as `KILLED`.
+
+**Example:**
+
+
+```
+<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <kill name="killBecauseNoInput">
+ <message>Input unavailable</message>
+ </kill>
+ ...
+</workflow-app>
+```
+
+<a name="DecisionNode"></a>
+#### 3.1.4 Decision Control Node
+
+A `decision` node enables a workflow to make a selection on the execution path to follow.
+
+The behavior of a `decision` node can be seen as a switch-case statement.
+
+A `decision` node consists of a list of predicates-transition pairs plus a default transition. Predicates are evaluated
+in order or appearance until one of them evaluates to `true` and the corresponding transition is taken. If none of the
+predicates evaluates to `true` the `default` transition is taken.
+
+Predicates are JSP Expression Language (EL) expressions (refer to section 4.2 of this document) that resolve into a
+boolean value, `true` or `false`. For example:
+
+
+```
+ ${fs:fileSize('/usr/foo/myinputdir') gt 10 * GB}
+```
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <decision name="[NODE-NAME]">
+ <switch>
+ <case to="[NODE_NAME]">[PREDICATE]</case>
+ ...
+ <case to="[NODE_NAME]">[PREDICATE]</case>
+ <default to="[NODE_NAME]"/>
+ </switch>
+ </decision>
+ ...
+</workflow-app>
+```
+
+The `name` attribute in the `decision` node is the name of the decision node.
+
+Each `case` elements contains a predicate and a transition name. The predicate ELs are evaluated
+in order until one returns `true` and the corresponding transition is taken.
+
+The `default` element indicates the transition to take if none of the predicates evaluates
+to `true`.
+
+All decision nodes must have a `default` element to avoid bringing the workflow into an error
+state if none of the predicates evaluates to true.
+
+**Example:**
+
+
+```
+<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <decision name="mydecision">
+ <switch>
+ <case to="reconsolidatejob">
+ ${fs:fileSize(secondjobOutputDir) gt 10 * GB}
+ </case> <case to="rexpandjob">
+ ${fs:fileSize(secondjobOutputDir) lt 100 * MB}
+ </case>
+ <case to="recomputejob">
+ ${ hadoop:counters('secondjob')[RECORDS][REDUCE_OUT] lt 1000000 }
+ </case>
+ <default to="end"/>
+ </switch>
+ </decision>
+ ...
+</workflow-app>
+```
+
+<a name="ForkJoinNodes"></a>
+#### 3.1.5 Fork and Join Control Nodes
+
+A `fork` node splits one path of execution into multiple concurrent paths of execution.
+
+A `join` node waits until every concurrent execution path of a previous `fork` node arrives to it.
+
+The `fork` and `join` nodes must be used in pairs. The `join` node assumes concurrent execution paths are children of
+the same `fork` node.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <fork name="[FORK-NODE-NAME]">
+ <path start="[NODE-NAME]" />
+ ...
+ <path start="[NODE-NAME]" />
+ </fork>
+ ...
+ <join name="[JOIN-NODE-NAME]" to="[NODE-NAME]" />
+ ...
+</workflow-app>
+```
+
+The `name` attribute in the `fork` node is the name of the workflow fork node. The `start` attribute in the `path`
+elements in the `fork` node indicate the name of the workflow node that will be part of the concurrent execution paths.
+
+The `name` attribute in the `join` node is the name of the workflow join node. The `to` attribute in the `join` node
+indicates the name of the workflow node that will executed after all concurrent execution paths of the corresponding
+fork arrive to the join node.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <fork name="forking">
+ <path start="firstparalleljob"/>
+ <path start="secondparalleljob"/>
+ </fork>
+ <action name="firstparallejob">
+ <map-reduce>
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <job-xml>job1.xml</job-xml>
+ </map-reduce>
+ <ok to="joining"/>
+ <error to="kill"/>
+ </action>
+ <action name="secondparalleljob">
+ <map-reduce>
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <job-xml>job2.xml</job-xml>
+ </map-reduce>
+ <ok to="joining"/>
+ <error to="kill"/>
+ </action>
+ <join name="joining" to="nextaction"/>
+ ...
+</workflow-app>
+```
+
+By default, Oozie performs some validation that any forking in a workflow is valid and won't lead to any incorrect behavior or
+instability. However, if Oozie is preventing a workflow from being submitted and you are very certain that it should work, you can
+disable forkjoin validation so that Oozie will accept the workflow. To disable this validation just for a specific workflow, simply
+set `oozie.wf.validate.ForkJoin` to `false` in the job.properties file. To disable this validation for all workflows, simply set
+`oozie.validate.ForkJoin` to `false` in the oozie-site.xml file. Disabling this validation is determined by the AND of both of
+these properties, so it will be disabled if either or both are set to false and only enabled if both are set to true (or not
+specified).
+
+<a name="ActionNodes"></a>
+### 3.2 Workflow Action Nodes
+
+Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task.
+
+#### 3.2.1 Action Basis
+
+The following sub-sections define common behavior and capabilities for all action types.
+
+##### 3.2.1.1 Action Computation/Processing Is Always Remote
+
+All computation/processing tasks triggered by an action node are remote to Oozie. No workflow application specific
+computation/processing task is executed within Oozie.
+
+##### 3.2.1.2 Actions Are Asynchronous
+
+All computation/processing tasks triggered by an action node are executed asynchronously by Oozie. For most types of
+computation/processing tasks triggered by workflow action, the workflow job has to wait until the
+computation/processing task completes before transitioning to the following node in the workflow.
+
+The exception is the `fs` action that is handled as a synchronous action.
+
+Oozie can detect completion of computation/processing tasks by two different means, callbacks and polling.
+
+When a computation/processing tasks is started by Oozie, Oozie provides a unique callback URL to the task, the task
+should invoke the given URL to notify its completion.
+
+For cases that the task failed to invoke the callback URL for any reason (i.e. a transient network failure) or when
+the type of task cannot invoke the callback URL upon completion, Oozie has a mechanism to poll computation/processing
+tasks for completion.
+
+##### 3.2.1.3 Actions Have 2 Transitions, `ok` and `error`
+
+If a computation/processing task -triggered by a workflow- completes successfully, it transitions to `ok`.
+
+If a computation/processing task -triggered by a workflow- fails to complete successfully, its transitions to `error`.
+
+If a computation/processing task exits in error, there computation/processing task must provide `error-code` and
+ `error-message` information to Oozie. This information can be used from `decision` nodes to implement a fine grain
+error handling at workflow application level.
+
+Each action type must clearly define all the error codes it can produce.
+
+##### 3.2.1.4 Action Recovery
+
+Oozie provides recovery capabilities when starting or ending actions.
+
+Once an action starts successfully Oozie will not retry starting the action if the action fails during its execution.
+The assumption is that the external system (i.e. Hadoop) executing the action has enough resilience to recover jobs
+once it has started (i.e. Hadoop task retries).
+
+Java actions are a special case with regard to retries. Although Oozie itself does not retry Java actions
+should they fail after they have successfully started, Hadoop itself can cause the action to be restarted due to a
+map task retry on the map task running the Java application. See the Java Action section below for more detail.
+
+For failures that occur prior to the start of the job, Oozie will have different recovery strategies depending on the
+nature of the failure.
+
+If the failure is of transient nature, Oozie will perform retries after a pre-defined time interval. The number of
+retries and timer interval for a type of action must be pre-configured at Oozie level. Workflow jobs can override such
+configuration.
+
+Examples of a transient failures are network problems or a remote system temporary unavailable.
+
+If the failure is of non-transient nature, Oozie will suspend the workflow job until an manual or programmatic
+intervention resumes the workflow job and the action start or end is retried. It is the responsibility of an
+administrator or an external managing system to perform any necessary cleanup before resuming the workflow job.
+
+If the failure is an error and a retry will not resolve the problem, Oozie will perform the error transition for the
+action.
+
+<a name="MapReduceAction"></a>
+#### 3.2.2 Map-Reduce Action
+
+The `map-reduce` action starts a Hadoop map/reduce job from a workflow. Hadoop jobs can be Java Map/Reduce jobs or
+streaming jobs.
+
+A `map-reduce` action can be configured to perform file system cleanup and directory creation before starting the
+map reduce job. This capability enables Oozie to retry a Hadoop job in the situation of a transient failure (Hadoop
+checks the non-existence of the job output directory and then creates it when the Hadoop job is starting, thus a retry
+without cleanup of the job output directory would fail).
+
+The workflow job will wait until the Hadoop map/reduce job completes before continuing to the next action in the
+workflow execution path.
+
+The counters of the Hadoop job and job exit status (`FAILED`, `KILLED` or `SUCCEEDED`) must be available to the
+workflow job after the Hadoop jobs ends. This information can be used from within decision nodes and other actions
+configurations.
+
+The `map-reduce` action has to be configured with all the necessary Hadoop JobConf properties to run the Hadoop
+map/reduce job.
+
+Hadoop JobConf properties can be specified as part of
+
+ * the `config-default.xml` or
+ * JobConf XML file bundled with the workflow application or
+ * \<global\> tag in workflow definition or
+ * Inline `map-reduce` action configuration or
+ * An implementation of OozieActionConfigurator specified by the \<config-class\> tag in workflow definition.
+
+The configuration properties are loaded in the following above order i.e. `streaming`, `job-xml`, `configuration`,
+and `config-class`, and the precedence order is later values override earlier values.
+
+Streaming and inline property values can be parameterized (templatized) using EL expressions.
+
+The Hadoop `mapred.job.tracker` and `fs.default.name` properties must not be present in the job-xml and inline
+configuration.
+
+<a name="FilesArchives"></a>
+##### 3.2.2.1 Adding Files and Archives for the Job
+
+The `file`, `archive` elements make available, to map-reduce jobs, files and archives. If the specified path is
+relative, it is assumed the file or archiver are within the application directory, in the corresponding sub-path.
+If the path is absolute, the file or archive it is expected in the given absolute path.
+
+Files specified with the `file` element, will be symbolic links in the home directory of the task.
+
+If a file is a native library (an '.so' or a '.so.#' file), it will be symlinked as and '.so' file in the task running
+directory, thus available to the task JVM.
+
+To force a symlink for a file on the task running directory, use a '#' followed by the symlink name. For example
+'mycat.sh#cat'.
+
+Refer to Hadoop distributed cache documentation for details more details on files and archives.
+
+##### 3.2.2.2 Configuring the MapReduce action with Java code
+
+Java code can be used to further configure the MapReduce action. This can be useful if you already have "driver" code for your
+MapReduce action, if you're more familiar with MapReduce's Java API, if there's some configuration that requires logic, or some
+configuration that's difficult to do in straight XML (e.g. Avro).
+
+Create a class that implements the org.apache.oozie.action.hadoop.OozieActionConfigurator interface from the "oozie-sharelib-oozie"
+artifact. It contains a single method that receives a `JobConf` as an argument. Any configuration properties set on this `JobConf`
+will be used by the MapReduce action.
+
+The OozieActionConfigurator has this signature:
+
+```
+public interface OozieActionConfigurator {
+ public void configure(JobConf actionConf) throws OozieActionConfiguratorException;
+}
+```
+where `actionConf` is the `JobConf` you can update. If you need to throw an Exception, you can wrap it in
+an `OozieActionConfiguratorException`, also in the "oozie-sharelib-oozie" artifact.
+
+For example:
+
+```
+package com.example;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileInputFormat;
+import org.apache.hadoop.mapred.FileOutputFormat;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.oozie.action.hadoop.OozieActionConfigurator;
+import org.apache.oozie.action.hadoop.OozieActionConfiguratorException;
+import org.apache.oozie.example.SampleMapper;
+import org.apache.oozie.example.SampleReducer;
+
+public class MyConfigClass implements OozieActionConfigurator {
+
+ @Override
+ public void configure(JobConf actionConf) throws OozieActionConfiguratorException {
+ if (actionConf.getUser() == null) {
+ throw new OozieActionConfiguratorException("No user set");
+ }
+ actionConf.setMapperClass(SampleMapper.class);
+ actionConf.setReducerClass(SampleReducer.class);
+ FileInputFormat.setInputPaths(actionConf, new Path("/user/" + actionConf.getUser() + "/input-data"));
+ FileOutputFormat.setOutputPath(actionConf, new Path("/user/" + actionConf.getUser() + "/output"));
+ ...
+ }
+}
+```
+
+To use your config class in your MapReduce action, simply compile it into a jar, make the jar available to your action, and specify
+the class name in the `config-class` element (this requires at least schema 0.5):
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <map-reduce>
+ ...
+ <job-xml>[JOB-XML-FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <config-class>com.example.MyConfigClass</config-class>
+ ...
+ </map-reduce>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+Another example of this can be found in the "map-reduce" example that comes with Oozie.
+
+A useful tip: The initial `JobConf` passed to the `configure` method includes all of the properties listed in the `configuration`
+section of the MR action in a workflow. If you need to pass any information to your OozieActionConfigurator, you can simply put
+them here.
+
+<a name="StreamingMapReduceAction"></a>
+##### 3.2.2.3 Streaming
+
+Streaming information can be specified in the `streaming` element.
+
+The `mapper` and `reducer` elements are used to specify the executable/script to be used as mapper and reducer.
+
+User defined scripts must be bundled with the workflow application and they must be declared in the `files` element of
+the streaming configuration. If the are not declared in the `files` element of the configuration it is assumed they
+will be available (and in the command PATH) of the Hadoop slave machines.
+
+Some streaming jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using
+the `file` and `archive` elements described in the previous section.
+
+The Mapper/Reducer can be overridden by a `mapred.mapper.class` or `mapred.reducer.class` properties in the `job-xml`
+file or `configuration` elements.
+
+<a name="PipesMapReduceAction"></a>
+##### 3.2.2.4 Pipes
+
+Pipes information can be specified in the `pipes` element.
+
+A subset of the command line options which can be used while using the Hadoop Pipes Submitter can be specified
+via elements - `map`, `reduce`, `inputformat`, `partitioner`, `writer`, `program`.
+
+The `program` element is used to specify the executable/script to be used.
+
+User defined program must be bundled with the workflow application.
+
+Some pipe jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using
+the `file` and `archive` elements described in the previous section.
+
+Pipe properties can be overridden by specifying them in the `job-xml` file or `configuration` element.
+
+##### 3.2.2.5 Syntax
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <map-reduce>
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <streaming>
+ <mapper>[MAPPER-PROCESS]</mapper>
+ <reducer>[REDUCER-PROCESS]</reducer>
+ <record-reader>[RECORD-READER-CLASS]</record-reader>
+ <record-reader-mapping>[NAME=VALUE]</record-reader-mapping>
+ ...
+ <env>[NAME=VALUE]</env>
+ ...
+ </streaming>
+ <!-- Either streaming or pipes can be specified for an action, not both -->
+ <pipes>
+ <map>[MAPPER]</map>
+ <reduce>[REDUCER]</reducer>
+ <inputformat>[INPUTFORMAT]</inputformat>
+ <partitioner>[PARTITIONER]</partitioner>
+ <writer>[OUTPUTFORMAT]</writer>
+ <program>[EXECUTABLE]</program>
+ </pipes>
+ <job-xml>[JOB-XML-FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <config-class>com.example.MyConfigClass</config-class>
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ </map-reduce>
+
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete before starting the job. This should be used
+exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation
+will be performed in the `fs.default.name` filesystem for hdfs URIs. The format for specifying hcatalog table URI is
+hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
+In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
+need to be placed in workflow lib directory or specified using `archive` tag.
+
+The `job-xml` element, if present, must refer to a Hadoop JobConf `job.xml` file bundled in the workflow application.
+By default the `job.xml` file is taken from the workflow application namenode, regardless the namenode specified for the action.
+To specify a `job.xml` on another namenode use a fully qualified file path.
+The `job-xml` element is optional and as of schema 0.4, multiple `job-xml` elements are allowed in order to specify multiple Hadoop JobConf `job.xml` files.
+
+The `configuration` element, if present, contains JobConf properties for the Hadoop job.
+
+Properties specified in the `configuration` element override properties specified in the file specified in the
+ `job-xml` element.
+
+As of schema 0.5, the `config-class` element, if present, contains a class that implements OozieActionConfigurator that can be used
+to further configure the MapReduce job.
+
+Properties specified in the `config-class` class override properties specified in `configuration` element.
+
+External Stats can be turned on/off by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in the configuration element of workflow.xml. The default value for this property is _false_.
+
+The `file` element, if present, must specify the target symbolic link for binaries by separating the original file and target with a # (file#target-sym-link). This is not required for libraries.
+
+The `mapper` and `reducer` process for streaming jobs, should specify the executable command with URL encoding. e.g. '%' should be replaced by '%25'.
+
+**Example:**
+
+
+```
+<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="myfirstHadoopJob">
+ <map-reduce>
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="hdfs://foo:8020/usr/tucu/output-data"/>
+ </prepare>
+ <job-xml>/myfirstjob.xml</job-xml>
+ <configuration>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>/usr/tucu/input-data</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>/usr/tucu/input-data</value>
+ </property>
+ <property>
+ <name>mapred.reduce.tasks</name>
+ <value>${firstJobReducers}</value>
+ </property>
+ <property>
+ <name>oozie.action.external.stats.write</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ </map-reduce>
+ <ok to="myNextAction"/>
+ <error to="errorCleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+In the above example, the number of Reducers to be used by the Map/Reduce job has to be specified as a parameter of
+the workflow job configuration when creating the workflow job.
+
+**Streaming Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="firstjob">
+ <map-reduce>
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${output}"/>
+ </prepare>
+ <streaming>
+ <mapper>/bin/bash testarchive/bin/mapper.sh testfile</mapper>
+ <reducer>/bin/bash testarchive/bin/reducer.sh</reducer>
+ </streaming>
+ <configuration>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>${input}</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>${output}</value>
+ </property>
+ <property>
+ <name>stream.num.map.output.key.fields</name>
+ <value>3</value>
+ </property>
+ </configuration>
+ <file>/users/blabla/testfile.sh#testfile</file>
+ <archive>/users/blabla/testarchive.jar#testarchive</archive>
+ </map-reduce>
+ <ok to="end"/>
+ <error to="kill"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+
+**Pipes Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="firstjob">
+ <map-reduce>
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${output}"/>
+ </prepare>
+ <pipes>
+ <program>bin/wordcount-simple#wordcount-simple</program>
+ </pipes>
+ <configuration>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>${input}</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>${output}</value>
+ </property>
+ </configuration>
+ <archive>/users/blabla/testarchive.jar#testarchive</archive>
+ </map-reduce>
+ <ok to="end"/>
+ <error to="kill"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+
+<a name="PigAction"></a>
+#### 3.2.3 Pig Action
+
+The `pig` action starts a Pig job.
+
+The workflow job will wait until the pig job completes before continuing to the next action.
+
+The `pig` action has to be configured with the resource-manager, name-node, pig script and the necessary parameters and
+configuration to run the Pig job.
+
+A `pig` action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
+starting the Pig job. This capability enables Oozie to retry a Pig job in the situation of a transient failure (Pig
+creates temporary directories for intermediate data, thus a retry without cleanup would fail).
+
+Hadoop JobConf properties can be specified as part of
+
+ * the `config-default.xml` or
+ * JobConf XML file bundled with the workflow application or
+ * \<global\> tag in workflow definition or
+ * Inline `pig` action configuration.
+
+The configuration properties are loaded in the following above order i.e. `job-xml` and `configuration`, and
+the precedence order is later values override earlier values.
+
+Inline property values can be parameterized (templatized) using EL expressions.
+
+The YARN `yarn.resourcemanager.address` and HDFS `fs.default.name` properties must not be present in the job-xml and inline
+configuration.
+
+As with Hadoop map-reduce jobs, it is possible to add files and archives to be available to the Pig job, refer to
+section [#FilesArchives][Adding Files and Archives for the Job].
+
+
+**Syntax for Pig actions in Oozie schema 1.0:**
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <pig>
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[JOB-XML-FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <script>[PIG-SCRIPT]</script>
+ <param>[PARAM-VALUE]</param>
+ ...
+ <param>[PARAM-VALUE]</param>
+ <argument>[ARGUMENT-VALUE]</argument>
+ ...
+ <argument>[ARGUMENT-VALUE]</argument>
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ </pig>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+**Syntax for Pig actions in Oozie schema 0.2:**
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.2">
+ ...
+ <action name="[NODE-NAME]">
+ <pig>
+ <job-tracker>[JOB-TRACKER]</job-tracker>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[JOB-XML-FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <script>[PIG-SCRIPT]</script>
+ <param>[PARAM-VALUE]</param>
+ ...
+ <param>[PARAM-VALUE]</param>
+ <argument>[ARGUMENT-VALUE]</argument>
+ ...
+ <argument>[ARGUMENT-VALUE]</argument>
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ </pig>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+**Syntax for Pig actions in Oozie schema 0.1:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1">
+ ...
+ <action name="[NODE-NAME]">
+ <pig>
+ <job-tracker>[JOB-TRACKER]</job-tracker>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[JOB-XML-FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <script>[PIG-SCRIPT]</script>
+ <param>[PARAM-VALUE]</param>
+ ...
+ <param>[PARAM-VALUE]</param>
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ </pig>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete before starting the job. This should be used
+exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation
+will be performed in the `fs.default.name` filesystem for hdfs URIs. The format for specifying hcatalog table URI is
+hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
+In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
+need to be placed in workflow lib directory or specified using `archive` tag.
+
+The `job-xml` element, if present, must refer to a Hadoop JobConf `job.xml` file bundled in the workflow application.
+The `job-xml` element is optional and as of schema 0.4, multiple `job-xml` elements are allowed in order to specify multiple Hadoop JobConf `job.xml` files.
+
+The `configuration` element, if present, contains JobConf properties for the underlying Hadoop jobs.
+
+Properties specified in the `configuration` element override properties specified in the file specified in the
+ `job-xml` element.
+
+External Stats can be turned on/off by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in the configuration element of workflow.xml. The default value for this property is _false_.
+
+The inline and job-xml configuration properties are passed to the Hadoop jobs submitted by Pig runtime.
+
+The `script` element contains the pig script to execute. The pig script can be templatized with variables of the
+form `${VARIABLE}`. The values of these variables can then be specified using the `params` element.
+
+NOTE: Oozie will perform the parameter substitution before firing the pig job. This is different from the
+[parameter substitution mechanism provided by Pig](http://wiki.apache.org/pig/ParameterSubstitution), which has a
+few limitations.
+
+The `params` element, if present, contains parameters to be passed to the pig script.
+
+**In Oozie schema 0.2:**
+The `arguments` element, if present, contains arguments to be passed to the pig script.
+
+
+All the above elements can be parameterized (templatized) using EL expressions.
+
+**Example for Oozie schema 0.2:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.2">
+ ...
+ <action name="myfirstpigjob">
+ <pig>
+ <job-tracker>foo:8021</job-tracker>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>oozie.action.external.stats.write</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <script>/mypigscript.pig</script>
+ <argument>-param</argument>
+ <argument>INPUT=${inputDir}</argument>
+ <argument>-param</argument>
+ <argument>OUTPUT=${outputDir}/pig-output3</argument>
+ </pig>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+
+**Example for Oozie schema 0.1:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
+ ...
+ <action name="myfirstpigjob">
+ <pig>
+ <job-tracker>foo:8021</job-tracker>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <script>/mypigscript.pig</script>
+ <param>InputDir=/home/tucu/input-data</param>
+ <param>OutputDir=${jobOutput}</param>
+ </pig>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+<a name="FsAction"></a>
+#### 3.2.4 Fs (HDFS) action
+
+The `fs` action allows to manipulate files and directories in HDFS from a workflow application. The supported commands
+are `move`, `delete`, `mkdir`, `chmod`, `touchz`, `setrep` and `chgrp`.
+
+The FS commands are executed synchronously from within the FS action, the workflow job will wait until the specified
+file commands are completed before continuing to the next action.
+
+Path names specified in the `fs` action can be parameterized (templatized) using EL expressions.
+Path name should be specified as a absolute path. In case of `move`, `delete`, `chmod` and `chgrp` commands, a glob pattern can also be specified instead of an absolute path.
+For `move`, glob pattern can only be specified for source path and not the target.
+
+Each file path must specify the file system URI, for move operations, the target must not specify the system URI.
+
+**IMPORTANT:** For the purposes of copying files within a cluster it is recommended to refer to the `distcp` action
+instead. Refer to [`distcp`](DG_DistCpActionExtension.html) action to copy files within a cluster.
+
+**IMPORTANT:** All the commands within `fs` action do not happen atomically, if a `fs` action fails half way in the
+commands being executed, successfully executed commands are not rolled back. The `fs` action, before executing any
+command must check that source paths exist and target paths don't exist (constraint regarding target relaxed for the `move` action. See below for details), thus failing before executing any command.
+Therefore the validity of all paths specified in one `fs` action are evaluated before any of the file operation are
+executed. Thus there is less chance of an error occurring while the `fs` action executes.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <fs>
+ <delete path='[PATH]' skip-trash='[true/false]'/>
+ ...
+ <mkdir path='[PATH]'/>
+ ...
+ <move source='[SOURCE-PATH]' target='[TARGET-PATH]'/>
+ ...
+ <chmod path='[PATH]' permissions='[PERMISSIONS]' dir-files='false' />
+ ...
+ <touchz path='[PATH]' />
+ ...
+ <chgrp path='[PATH]' group='[GROUP]' dir-files='false' />
+ ...
+ <setrep path='[PATH]' replication-factor='2'/>
+ </fs>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `delete` command deletes the specified path, if it is a directory it deletes recursively all its content and then
+deletes the directory. By default it does skip trash. It can be moved to trash by setting the value of skip-trash to
+'false'. It can also be used to drop hcat tables/partitions. This is the only FS command which supports HCatalog URIs as well.
+For eg:
+
+```
+<delete path='hcat://[metastore server]:[port]/[database name]/[table name]'/>
+OR
+<delete path='hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...'/>
+```
+
+The `mkdir` command creates the specified directory, it creates all missing directories in the path. If the directory
+already exist it does a no-op.
+
+In the `move` command the `source` path must exist. The following scenarios are addressed for a `move`:
+
+ * The file system URI(e.g. `hdfs://{nameNode}`) can be skipped in the `target` path. It is understood to be the same as that of the source. But if the target path does contain the system URI, it cannot be different than that of the source.
+ * The parent directory of the `target` path must exist
+ * For the `target` path, if it is a file, then it must not already exist.
+ * However, if the `target` path is an already existing directory, the `move` action will place your `source` as a child of the `target` directory.
+
+The `chmod` command changes the permissions for the specified path. Permissions can be specified using the Unix Symbolic
+representation (e.g. -rwxrw-rw-) or an octal representation (755).
+When doing a `chmod` command on a directory, by default the command is applied to the directory and the files one level
+within the directory. To apply the `chmod` command to the directory, without affecting the files within it,
+the `dir-files` attribute must be set to `false`. To apply the `chmod` command
+recursively to all levels within a directory, put a `recursive` element inside the \<chmod\> element.
+
+The `touchz` command creates a zero length file in the specified path if none exists. If one already exists, then touchz will perform a touch operation.
+Touchz works only for absolute paths.
+
+The `chgrp` command changes the group for the specified path.
+When doing a `chgrp` command on a directory, by default the command is applied to the directory and the files one level
+within the directory. To apply the `chgrp` command to the directory, without affecting the files within it,
+the `dir-files` attribute must be set to `false`.
+To apply the `chgrp` command recursively to all levels within a directory, put a `recursive` element inside the \<chgrp\> element.
+
+The `setrep` command changes replication factor of an hdfs file(s). Changing RF of directories or symlinks is not
+supported; this action requires an argument for RF.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="hdfscommands">
+ <fs>
+ <delete path='hdfs://foo:8020/usr/tucu/temp-data'/>
+ <mkdir path='archives/${wf:id()}'/>
+ <move source='${jobInput}' target='archives/${wf:id()}/processed-input'/>
+ <chmod path='${jobOutput}' permissions='-rwxrw-rw-' dir-files='true'><recursive/></chmod>
+ <chgrp path='${jobOutput}' group='testgroup' dir-files='true'><recursive/></chgrp>
+ <setrep path='archives/${wf:id()/filename(s)}' replication-factor='2'/>
+ </fs>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+In the above example, a directory named after the workflow job ID is created and the input of the job, passed as
+workflow configuration parameter, is archived under the previously created directory.
+
+As of schema 0.4, if a `name-node` element is specified, then it is not necessary for any of the paths to start with the file system
+URI as it is taken from the `name-node` element. This is also true if the name-node is specified in the global section
+(see [Global Configurations](WorkflowFunctionalSpec.html#GlobalConfigurations))
+
+As of schema 0.4, zero or more `job-xml` elements can be specified; these must refer to Hadoop JobConf `job.xml` formatted files
+bundled in the workflow application. They can be used to set additional properties for the FileSystem instance.
+
+As of schema 0.4, if a `configuration` element is specified, then it will also be used to set additional JobConf properties for the
+FileSystem instance. Properties specified in the `configuration` element override properties specified in the files specified
+by any `job-xml` elements.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.4">
+ ...
+ <action name="hdfscommands">
+ <fs>
+ <name-node>hdfs://foo:8020</name-node>
+ <job-xml>fs-info.xml</job-xml>
+ <configuration>
+ <property>
+ <name>some.property</name>
+ <value>some.value</value>
+ </property>
+ </configuration>
+ <delete path='/usr/tucu/temp-data'/>
+ </fs>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+<a name="SubWorkflowAction"></a>
+#### 3.2.5 Sub-workflow Action
+
+The `sub-workflow` action runs a child workflow job, the child workflow job can be in the same Oozie system or in
+another Oozie system.
+
+The parent workflow job will wait until the child workflow job has completed.
+
+There can be several sub-workflows defined within a single workflow, each under its own action element.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <sub-workflow>
+ <app-path>[WF-APPLICATION-PATH]</app-path>
+ <propagate-configuration/>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ </sub-workflow>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The child workflow job runs in the same Oozie system instance where the parent workflow job is running.
+
+The `app-path` element specifies the path to the workflow application of the child workflow job.
+
+The `propagate-configuration` flag, if present, indicates that the workflow job configuration should be propagated to
+the child workflow.
+
+The `configuration` section can be used to specify the job properties that are required to run the child workflow job.
+
+The configuration of the `sub-workflow` action can be parameterized (templatized) using EL expressions.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="a">
+ <sub-workflow>
+ <app-path>child-wf</app-path>
+ <configuration>
+ <property>
+ <name>input.dir</name>
+ <value>${wf:id()}/second-mr-output</value>
+ </property>
+ </configuration>
+ </sub-workflow>
+ <ok to="end"/>
+ <error to="kill"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+In the above example, the workflow definition with the name `child-wf` will be run on the Oozie instance at
+ `.http://myhost:11000/oozie`. The specified workflow application must be already deployed on the target Oozie instance.
+
+A configuration parameter `input.dir` is being passed as job property to the child workflow job.
+
+The subworkflow can inherit the lib jars from the parent workflow by setting `oozie.subworkflow.classpath.inheritance` to true
+in oozie-site.xml or on a per-job basis by setting `oozie.wf.subworkflow.classpath.inheritance` to true in a job.properties file.
+If both are specified, `oozie.wf.subworkflow.classpath.inheritance` has priority. If the subworkflow and the parent have
+conflicting jars, the subworkflow's jar has priority. By default, `oozie.wf.subworkflow.classpath.inheritance` is set to false.
+
+To prevent errant workflows from starting infinitely recursive subworkflows, `oozie.action.subworkflow.max.depth` can be specified
+in oozie-site.xml to set the maximum depth of subworkflow calls. For example, if set to 3, then a workflow can start subwf1, which
+can start subwf2, which can start subwf3; but if subwf3 tries to start subwf4, then the action will fail. The default is 50.
+
+<a name="JavaAction"></a>
+#### 3.2.6 Java Action
+
+The `java` action will execute the `public static void main(String[] args)` method of the specified main Java class.
+
+Java applications are executed in the Hadoop cluster as map-reduce job with a single Mapper task.
+
+The workflow job will wait until the java application completes its execution before continuing to the next action.
+
+The `java` action has to be configured with the resource-manager, name-node, main Java class, JVM options and arguments.
+
+To indicate an `ok` action transition, the main Java class must complete gracefully the `main` method invocation.
+
+To indicate an `error` action transition, the main Java class must throw an exception.
+
+The main Java class can call `System.exit(int n)`. Exit code zero is regarded as OK, while non-zero exit codes will
+cause the `java` action to do an `error` transition and exit.
+
+A `java` action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
+starting the Java application. This capability enables Oozie to retry a Java application in the situation of a transient
+or non-transient failure (This can be used to cleanup any temporary data which may have been created by the Java
+application in case of failure).
+
+A `java` action can create a Hadoop configuration for interacting with a cluster (e.g. launching a map-reduce job).
+Oozie prepares a Hadoop configuration file which includes the environments site configuration files (e.g. hdfs-site.xml,
+mapred-site.xml, etc) plus the properties added to the `<configuration>` section of the `java` action. The Hadoop configuration
+file is made available as a local file to the Java application in its running directory. It can be added to the `java` actions
+Hadoop configuration by referencing the system property: `oozie.action.conf.xml`. For example:
+
+
+```
+// loading action conf prepared by Oozie
+Configuration actionConf = new Configuration(false);
+actionConf.addResource(new Path("file:///", System.getProperty("oozie.action.conf.xml")));
+```
+
+If `oozie.action.conf.xml` is not added then the job will pick up the mapred-default properties and this may result
+in unexpected behaviour. For repeated configuration properties later values override earlier ones.
+
+Inline property values can be parameterized (templatized) using EL expressions.
+
+The YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties must not be present
+in the `job-xml` and in the inline configuration.
+
+As with `map-reduce` and `pig` actions, it is possible to add files and archives to be available to the Java
+application. Refer to section [#FilesArchives][Adding Files and Archives for the Job].
+
+The `capture-output` element can be used to propagate values back into Oozie context, which can then be accessed via
+EL-functions. This needs to be written out as a java properties format file. The filename is obtained via a System
+property specified by the constant `oozie.action.output.properties`
+
+**IMPORTANT:** In order for a Java action to succeed on a secure cluster, it must propagate the Hadoop delegation token like in the
+following code snippet (this is benign on non-secure clusters):
+
+```
+// propagate delegation related props from launcher job to MR job
+if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
+ jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
+}
+```
+
+**IMPORTANT:** Because the Java application is run from within a Map-Reduce job, from Hadoop 0.20. onwards a queue must
+be assigned to it. The queue name must be specified as a configuration property.
+
+**IMPORTANT:** The Java application from a Java action is executed in a single map task. If the task is abnormally terminated,
+such as due to a TaskTracker restart (e.g. during cluster maintenance), the task will be retried via the normal Hadoop task
+retry mechanism. To avoid workflow failure, the application should be written in a fashion that is resilient to such retries,
+for example by detecting and deleting incomplete outputs or picking back up from complete outputs. Furthermore, if a Java action
+spawns asynchronous activity outside the JVM of the action itself (such as by launching additional MapReduce jobs), the
+application must consider the possibility of collisions with activity spawned by the new instance.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <java>
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[JOB-XML]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <main-class>[MAIN-CLASS]</main-class>
+ <java-opts>[JAVA-STARTUP-OPTS]</java-opts>
+ <arg>ARGUMENT</arg>
+ ...
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ <capture-output />
+ </java>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete before starting the Java application. This should
+be used exclusively for directory cleanup or dropping of hcatalog table or table partitions for the Java application to be executed.
+In case of `delete`, a glob pattern can be used to specify path.
+The format for specifying hcatalog table URI is
+hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
+In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
+need to be placed in workflow lib directory or specified using `archive` tag.
+
+The `java-opts` and `java-opt` elements, if present, contains the command line parameters which are to be used to start the JVM that
+will execute the Java application. Using this element is equivalent to using the `mapred.child.java.opts`
+or `mapreduce.map.java.opts` configuration properties, with the advantage that Oozie will append to these properties instead of
+simply setting them (e.g. if you have one of these properties specified in mapred-site.xml, setting it again in
+the `configuration` element will override it, but using `java-opts` or `java-opt` will instead append to it, preserving the original
+value). You can have either one `java-opts`, multiple `java-opt`, or neither; you cannot use both at the same time. In any case,
+Oozie will set both `mapred.child.java.opts` and `mapreduce.map.java.opts` to the same value based on combining them. In other
+words, after Oozie is finished:
+
+```
+mapred.child.java.opts <-- "<mapred.child.java.opts> <mapreduce.map.java.opts> <java-opt...|java-opts>"
+mapreduce.map.java.opts <-- "<mapred.child.java.opts> <mapreduce.map.java.opts> <java-opt...|java-opts>"
+```
+In the case that parameters are repeated, the latest instance of the parameter is used by Java. This means that `java-opt`
+(or `java-opts`) has the highest priority, followed by `mapreduce.map.java.opts`, and finally `mapred.child.java.opts`. When
+multiple `java-opt` are specified, they are included from top to bottom ordering, where the bottom has highest priority.
+
+The `arg` elements, if present, contains arguments for the main function. The value of each `arg` element is considered
+a single argument and they are passed to the `main` method in the same order.
+
+All the above elements can be parameterized (templatized) using EL expressions.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="myfirstjavajob">
+ <java>
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.queue.name</name>
+ <value>default</value>
+ </property>
+ </configuration>
+ <main-class>org.apache.oozie.MyFirstMainClass</main-class>
+ <java-opts>-Dblah</java-opts>
+ <arg>argument1</arg>
+ <arg>argument2</arg>
+ </java>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+##### 3.2.6.1 Overriding an action's Main class
+
+This feature is useful for developers to change the Main classes without having to recompile or redeploy Oozie.
+
+For most actions (not just the Java action), you can override the Main class it uses by specifying the following `configuration`
+property and making sure that your class is included in the workflow's classpath. If you specify this in the Java action,
+the main-class element has priority.
+
+
+```
+<property>
+ <name>oozie.launcher.action.main.class</name>
+ <value>org.my.CustomMain</value>
+</property>
+```
+
+**Note:** Most actions typically pass information to their corresponding Main in specific ways; you should look at the action's
+existing Main to see how it works before creating your own. In fact, its probably simplest to just subclass the existing Main and
+add/modify/overwrite any behavior you want to change.
+
+<a name="GitAction"></a>
+#### 3.2.7 Git action
+
+The `git` action allows one to clone a Git repository into HDFS. The supported options are `git-uri`, `branch`, `key-path`
+and `destination-uri`.
+
+The `git clone` action is executed asynchronously by one of the YARN containers assigned to run on the cluster. If an SSH key is
+specified it will be created on the file system in a YARN container's local directory, relying on YARN NodeManager to remove the
+file after the action has run.
+
+Path names specified in the `git` action should be able to be parameterized (templatized) using EL expressions,
+e.g. `${wf:user()}` . Path name should be specified as an absolute path. Each file path must specify the file system URI.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <git>
+ <git-uri>[SOURCE-URI]</git-uri>
+ ...
+ <branch>[BRANCH]</branch>
+ ...
+ <key-path>[HDFS-PATH]</key-path>
+ ...
+ <destination-uri>[HDFS-PATH]</destination-uri>
+ </git>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
+ ...
+ <action name="clone_oozie">
+ <git>
+ <git-uri>https://github.com/apache/oozie</git-uri>
+ <destination-uri>hdfs://my_git_repo_directory</destination-uri>
+ </git>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+In the above example, a Git repository on e.g. GitHub.com is cloned to the HDFS directory `my_git_repo_directory` which should not
+exist previously on the filesystem. Note that repository addresses outside of GitHub.com but accessible to the YARN container
+running the Git action may also be used.
+
+If a `name-node` element is specified, then it is not necessary for any of the paths to start with the file system URI as it is
+taken from the `name-node` element.
+
+The `resource-manager` (Oozie 5.x) element has to be specified to name the YARN ResourceManager address.
+
+If any of the paths need to be served from another HDFS namenode, its address has to be part of
+that filesystem URI prefix:
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <git>
+ ...
+ <name-node>hdfs://name-node.first.company.com:8020</name-node>
+ ...
+ <key-path>hdfs://name-node.second.company.com:8020/[HDFS-PATH]</key-path>
+ ...
+ </git>
+ ...
+ </action>
+ ...
+</workflow-app>
+```
+
+This is also true if the name-node is specified in the global section (see
+[Global Configurations](WorkflowFunctionalSpec.html#GlobalConfigurations)).
+
+Be aware that `key-path` might point to a secure object store location other than the current `fs.defaultFS`. In that case,
+appropriate file permissions are still necessary (readable by submitting user), credentials provided, etc.
+
+As of workflow schema 1.0, zero or more `job-xml` elements can be specified; these must refer to Hadoop JobConf `job.xml` formatted
+files bundled in the workflow application. They can be used to set additional properties for the `FileSystem` instance.
+
+As of schema workflow schema 1.0, if a `configuration` element is specified, then it will also be used to set additional `JobConf`
+properties for the `FileSystem` instance. Properties specified in the `configuration` element are overridden by properties
+specified in the files specified by any `job-xml` elements.
+
+**Example:**
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <git>
+ ...
+ <name-node>hdfs://foo:8020</name-node>
+ <job-xml>fs-info.xml</job-xml>
+ <configuration>
+ <property>
+ <name>some.property</name>
+ <value>some.value</value>
+ </property>
+ </configuration>
+ </git>
+ ...
+ </action>
+ ...
+</workflow>
+```
+
+<a name="WorkflowParameterization"></a>
+## 4 Parameterization of Workflows
+
+Workflow definitions can be parameterized.
+
+When workflow node is executed by Oozie all the ELs are resolved into concrete values.
+
+The parameterization of workflow definitions it done using JSP Expression Language syntax from the
+[JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to
+support variables as parameters but also functions and complex expressions.
+
+EL expressions can be used in the configuration values of action and decision nodes. They can be used in XML attribute
+values and in XML element and attribute values.
+
+They cannot be used in XML element and attribute names. They cannot be used in the name of a node and they cannot be
+used within the `transition` elements of a node.
+
+<a name="WorkflowProperties"></a>
+### 4.1 Workflow Job Properties (or Parameters)
+
+When a workflow job is submitted to Oozie, the submitter may specify as many workflow job properties as required
+(similar to Hadoop JobConf properties).
+
+Workflow applications may define default values for the workflow job or action parameters. They must be defined in a
+ `config-default.xml` file bundled with the workflow application archive (refer to section '7 Workflow
+ Applications Packaging'). Job or action properties specified in the workflow definition have precedence over the default values.
+
+Properties that are a valid Java identifier, `[A-Za-z_][0-9A-Za-z_]*`, are available as '${NAME}'
+variables within the workflow definition.
+
+Properties that are not valid Java Identifier, for example 'job.tracker', are available via the `String
+wf:conf(String name)` function. Valid identifier properties are available via this function as well.
+
+Using properties that are valid Java identifiers result in a more readable and compact definition.
+
+By using properties
+**Example:**
+
+Parameterized Workflow definition:
+
+
+```
+<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name='firstjob'>
+ <map-reduce>
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode}</name-node>
+ <configuration>
+ <property>
+ <name>mapred.mapper.class</name>
+ <value>com.foo.FirstMapper</value>
+ </property>
+ <property>
+ <name>mapred.reducer.class</name>
+ <value>com.foo.FirstReducer</value>
+ </property>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>${inputDir}</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>${outputDir}</value>
+ </property>
+ </configuration>
+ </map-reduce>
+ <ok to='secondjob'/>
+ <error to='killcleanup'/>
+ </action>
+ ...
+</workflow-app>
+```
+
+When submitting a workflow job for the workflow definition above, 3 workflow job properties must be specified:
+
+ * `resourceManager:`
+ * `inputDir:`
+ * `outputDir:`
+
+If the above 3 properties are not specified, the job will fail.
+
+As of schema 0.4, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
+properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
+
+**Example:**
+
+The previous parameterized workflow definition with formal parameters:
+
+
+```
+<workflow-app name='hello-wf' xmlns="uri:oozie:workflow:1.0">
+ <parameters>
+ <property>
+ <name>inputDir</name>
+ </property>
+ <property>
+ <name>outputDir</name>
+ <value>out-dir</value>
+ </property>
+ </parameters>
+ ...
+ <action name='firstjob'>
+ <map-reduce>
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode}</name-node>
+ <configuration>
+ <property>
+ <name>mapred.mapper.class</name>
+ <value>com.foo.FirstMapper</value>
+ </property>
+ <property>
+ <name>mapred.reducer.class</name>
+ <value>com.foo.FirstReducer</value>
+ </property>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>${inputDir}</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>${outputDir}</value>
+ </property>
+ </configuration>
+ </map-reduce>
+ <ok to='secondjob'/>
+ <error to='killcleanup'/>
+ </action>
+ ...
+</workflow-app>
+```
+
+In the above example, if `inputDir` is not specified, Oozie will print an error message instead of submitting the job. If
+`outputDir` is not specified, Oozie will use the default value, `out-dir`.
+
+<a name="WorkflowELSupport"></a>
+### 4.2 Expression Language Functions
+
+Oozie, besides allowing the use of workflow job properties to parameterize workflow jobs, it provides a set of build
+in EL functions that enable a more complex parameterization of workflow action nodes as well as the predicates in
+decision nodes.
+
+#### 4.2.1 Basic EL Constants
+
+ * **KB:** 1024, one kilobyte.
+ * **MB:** 1024 *** KB, one megabyte.
+ * **GB:** 1024 *** MB, one gigabyte.
+ * **TB:** 1024 *** GB, one terabyte.
+ * **PB:** 1024 *** TG, one petabyte.
+
+All the above constants are of type `long`.
+
+#### 4.2.2 Basic EL Functions
+
+**String firstNotNull(String value1, String value2)**
+
+It returns the first not `null` value, or `null` if both are `null`.
+
+Note that if the output of this function is `null` and it is used as string, the EL library converts it to an empty
+string. This is the common behavior when using `firstNotNull()` in node configuration sections.
+
+**String concat(String s1, String s2)**
+
+It returns the concatenation of 2 strings. A string with `null` value is considered as an empty string.
+
+**String replaceAll(String src, String regex, String replacement)**
+
+Replace each occurrence of regular expression match in the first string with the `replacement` string and return the replaced string. A 'regex' string with `null` value is considered as no change. A 'replacement' string with `null` value is consider as an empty string.
+
+**String appendAll(String src, String append, String delimeter)**
+
+ Add the `append` string into each splitted sub-strings of the first string(`src`). The split is performed into `src` string using the `delimiter`. E.g. `appendAll("/a/b/,/c/b/,/c/d/", "ADD", ",")` will return `/a/b/ADD,/c/b/ADD,/c/d/ADD`. A `append` string with `null` value is consider as an empty string. A `delimiter` string with value `null` is considered as no append in the string.
+
+**String trim(String s)**
+
+It returns the trimmed value of the given string. A string with `null` value is considered as an empty string.
+
+**String urlEncode(String s)**
+
+It returns the URL UTF-8 encoded value of the given string. A string with `null` value is considered as an empty string.
+
+**String timestamp()**
+
+It returns the current datetime in ISO8601 format, down to minutes (yyyy-MM-ddTHH:mmZ), in the Oozie's processing timezone,
+i.e. 1997-07-16T19:20Z
+
+**String toJsonStr(Map<String, String>)** (since Oozie 3.3)
+
+It returns an XML encoded JSON representation of a Map<String, String>. This function is useful to encode as
+a single property the complete action-data of an action, **wf:actionData(String actionName)**, in order to pass
+it in full to another action.
+
+**String toPropertiesStr(Map<String, String>)** (since Oozie 3.3)
+
+It returns an XML encoded Properties representation of a Map<String, String>. This function is useful to encode as
+a single property the complete action-data of an action, **wf:actionData(String actionName)**, in order to pass
+it in full to another action.
+
+**String toConfigurationStr(Map<String, String>)** (since Oozie 3.3)
+
+It returns an XML encoded Configuration representation of a Map<String, String>. This function is useful to encode as
+a single property the complete action-data of an action, **wf:actionData(String actionName)**, in order to pass
+it in full to another action.
+
+#### 4.2.3 Workflow EL Functions
+
+**String wf:id()**
+
+It returns the workflow job ID for the current workflow job.
+
+**String wf:name()**
+
+It returns the workflow application name for the current workflow job.
+
+**String wf:appPath()**
+
+It returns the workflow application path for the current workflow job.
+
+**String wf:conf(String name)**
+
+It returns the value of the workflow job configuration property for the current workflow job, or an empty string if
+undefined.
+
+**String wf:user()**
+
+It returns the user name that started the current workflow job.
+
+**String wf:group()**
+
+It returns the group/ACL for the current workflow job.
+
+**String wf:callback(String stateVar)**
+
+It returns the callback URL for the current workflow action node, `stateVar` can be a valid exit state (`OK` or
+ `ERROR`) for the action or a token to be replaced with the exit state by the remote system executing the task.
+
+**String wf:transition(String node)**
+
+It returns the transition taken by the specified workflow action node, or an empty string if the action has not being
+executed or it has not completed yet.
+
+**String wf:lastErrorNode()**
+
+It returns the name of the last workflow action node that exit with an `ERROR` exit state, or an empty string if no
+action has exited with `ERROR` state in the current workflow job.
+
+**String wf:errorCode(String node)**
+
+It returns the error code for the specified action node, or an empty string if the action node has not exited
+with `ERROR` state.
+
+Each type of action node must define its complete error code list.
+
+**String wf:errorMessage(String message)**
+
+It returns the error message for the specified action node, or an empty string if no action node has not exited
+with `ERROR` state.
+
+The error message can be useful for debugging and notification purposes.
+
+**int wf:run()**
+
+It returns the run number for the current workflow job, normally `0` unless the workflow job is re-run, in which case
+indicates the current run.
+
+**Map<String, String> wf:actionData(String node)**
+
+This function is only applicable to action nodes that produce output data on completion.
+
+The output data is in a Java Properties format and via this EL function it is available as a `Map<String, String>`.
+
+**String wf:actionExternalId(String node)**
+
+It returns the external Id for an action node, or an empty string if the action has not being executed or it has not
+completed yet.
+
+**String wf:actionTrackerUri(String node)**
+
+It returns the tracker URI for an action node, or an empty string if the action has not being executed or it has not
+completed yet.
+
+**String wf:actionExternalStatus(String node)**
+
+It returns the external status for an action node, or an empty string if the action has not being executed or it has
+not completed yet.
+
+#### 4.2.4 Hadoop EL Constants
+
+ * **RECORDS:** Hadoop record counters group name.
+ * **MAP_IN:** Hadoop mapper input records counter name.
+ * **MAP_OUT:** Hadoop mapper output records counter name.
+ * **REDUCE_IN:** Hadoop reducer input records counter name.
+ * **REDUCE_OUT:** Hadoop reducer input record counter name.
+ * **GROUPS:** 1024 *** Hadoop mapper/reducer record groups counter name.
+
+#### 4.2.5 Hadoop EL Functions
+
+<a name="HadoopCountersEL"></a>
+**Map < String, Map < String, Long > > hadoop:counters(String node)**
+
+It returns the counters for a job submitted by a Hadoop action node. It returns `0` if the if the Hadoop job has not
+started yet and for undefined counters.
+
+The outer Map key is a counter group name. The inner Map value is a Map of counter names and counter values.
+
+The Hadoop EL constants defined in the previous section provide access to the Hadoop built in record counters.
+
+This function can also be used to access specific action statistics information. Examples of action stats and their usage through EL Functions (referenced in workflow xml) are given below.
+
+**Example of MR action stats:**
+
+```
+{
+ "ACTION_TYPE": "MAP_REDUCE",
+ "org.apache.hadoop.mapred.JobInProgress$Counter": {
+ "TOTAL_LAUNCHED_REDUCES": 1,
+ "TOTAL_LAUNCHED_MAPS": 1,
+ "DATA_LOCAL_MAPS": 1
+ },
+ "FileSystemCounters": {
+ "FILE_BYTES_READ": 1746,
+ "HDFS_BYTES_READ": 1409,
+ "FILE_BYTES_WRITTEN": 3524,
+ "HDFS_BYTES_WRITTEN": 1547
+ },
+ "org.apache.hadoop.mapred.Task$Counter": {
+ "REDUCE_INPUT_GROUPS": 33,
+ "COMBINE_OUTPUT_RECORDS": 0,
+ "MAP_INPUT_RECORDS": 33,
+ "REDUCE_SHUFFLE_BYTES": 0,
+ "REDUCE_OUTPUT_RECORDS": 33,
+ "SPILLED_RECORDS": 66,
+ "MAP_OUTPUT_BYTES": 1674,
+ "MAP_INPUT_BYTES": 1409,
+ "MAP_OUTPUT_RECORDS": 33,
+ "COMBINE_INPUT_RECORDS": 0,
+ "REDUCE_INPUT_RECORDS": 33
+ }
+}
+```
+
+Below is the workflow that describes how to access specific information using hadoop:counters() EL function from the MR stats.
+**Workflow xml:**
+
+```
+<workflow-app xmlns="uri:oozie:workflow:1.0" name="map-reduce-wf">
+ <start to="mr-node"/>
+ <action name="mr-node">
+ <map-reduce>
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode}</name-node>
+ <prepare>
+ <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.job.queue.name</name>
+ <value>${queueName}</value>
+ </property>
+ <property>
+ <name>mapred.mapper.class</name>
+ <value>org.apache.oozie.example.SampleMapper</value>
+ </property>
+ <property>
+ <name>mapred.reducer.class</name>
+ <value>org.apache.oozie.example.SampleReducer</value>
+ </property>
+ <property>
+ <name>mapred.map.tasks</name>
+ <value>1</value>
+ </property>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
+ </property>
+ <property>
+ <name>oozie.action.external.stats.write</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ </map-reduce>
+ <ok to="java1"/>
+ <error to="fail"/>
+ </action>
+ <action name="java1">
+ <java>
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode}</name-node>
+ <configuration>
+ <property>
+ <name>mapred.job.queue.name</name>
+ <value>${queueName}</value>
+ </property>
+ </configuration>
+ <main-class>MyTest</main-class>
+ <arg> ${hadoop:counters("mr-node")["FileSystemCounters"]["FILE_BYTES_READ"]}</arg>
+ <capture-output/>
+ </java>
+ <ok to="end" />
+ <error to="fail" />
+ </action>
+ <kill name="fail">
+ <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
+ </kill>
+ <end name="end"/>
+
<TRUNCATED>
[13/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/WebServicesAPI.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/WebServicesAPI.md b/docs/src/site/markdown/WebServicesAPI.md
new file mode 100644
index 0000000..a303802
--- /dev/null
+++ b/docs/src/site/markdown/WebServicesAPI.md
@@ -0,0 +1,2379 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Oozie Web Services API, V1 (Workflow, Coordinator, And Bundle)
+
+The Oozie Web Services API is a HTTP REST JSON API.
+
+All responses are in `UTF-8`.
+
+Assuming Oozie is running at `OOZIE_URL`, the following web services end points are supported:
+
+ * \<OOZIE_URL\>/versions
+ * \<OOZIE_URL\>/v1/admin
+ * \<OOZIE_URL\>/v1/job
+ * \<OOZIE_URL\>/v1/jobs
+ * \<OOZIE_URL\>/v2/job
+ * \<OOZIE_URL\>/v2/jobs
+ * \<OOZIE_URL\>/v2/admin
+ * \<OOZIE_URL\>/v2/sla
+
+Documentation on the API is below; in some cases, looking at the corresponding command in the
+[Command Line Documentation](DG_CommandLineTool.html) page will provide additional details and examples. Most of the functionality
+offered by the Oozie CLI is using the WS API. If you export `OOZIE_DEBUG` then the Oozie CLI will output the WS API
+details used by any commands you execute. This is useful for debugging purposes to or see how the Oozie CLI works with the WS API.
+
+### Versions End-Point
+
+_Identical to the corresponding Oozie v0 WS API_
+
+This endpoint is for clients to perform protocol negotiation.
+
+It support only HTTP GET request and not sub-resources.
+
+It returns the supported Oozie protocol versions by the server.
+
+Current returned values are `0, 1, 2`.
+
+**Request:**
+
+
+```
+GET /oozie/versions
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+[0,1]
+```
+
+### Admin End-Point
+
+This endpoint is for obtaining Oozie system status and configuration information.
+
+It supports the following sub-resources: `status, os-env, sys-props, configuration, instrumentation, systems, available-timezones`.
+
+#### System Status
+
+_Identical to the corresponding Oozie v0 WS API_
+
+A HTTP GET request returns the system status.
+
+**Request:**
+
+
+```
+GET /oozie/v1/admin/status
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{"systemMode":NORMAL}
+```
+
+With a HTTP PUT request it is possible to change the system status between `NORMAL`, `NOWEBSERVICE`, and `SAFEMODE`.
+
+**Request:**
+
+
+```
+PUT /oozie/v1/admin/status?systemmode=SAFEMODE
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+```
+
+#### OS Environment
+
+_Identical to the corresponding Oozie v0 WS API_
+
+A HTTP GET request returns the Oozie system OS environment.
+
+**Request:**
+
+
+```
+GET /oozie/v1/admin/os-env
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ TERM: "xterm",
+ JAVA_HOME: "/usr/java/latest",
+ XCURSOR_SIZE: "",
+ SSH_CLIENT: "::ffff:127.0.0.1 49082 22",
+ XCURSOR_THEME: "default",
+ INPUTRC: "/etc/inputrc",
+ HISTSIZE: "1000",
+ PATH: "/usr/java/latest/bin"
+ KDE_FULL_SESSION: "true",
+ LANG: "en_US.UTF-8",
+ ...
+}
+```
+
+#### Java System Properties
+
+_Identical to the corresponding Oozie v0 WS API_
+
+A HTTP GET request returns the Oozie Java system properties.
+
+**Request:**
+
+
+```
+GET /oozie/v1/admin/java-sys-properties
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ java.vm.version: "11.0-b15",
+ sun.jnu.encoding: "UTF-8",
+ java.vendor.url: "http://java.sun.com/",
+ java.vm.info: "mixed mode",
+ ...
+}
+```
+
+#### Oozie Configuration
+
+_Identical to the corresponding Oozie v0 WS API_
+
+A HTTP GET request returns the Oozie system configuration.
+
+**Request:**
+
+
+```
+GET /oozie/v1/admin/configuration
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ oozie.service.SchedulerService.threads: "5",
+ oozie.service.ActionService.executor.classes: "
+ org.apache.oozie.dag.action.decision.DecisionActionExecutor,
+ org.apache.oozie.dag.action.hadoop.HadoopActionExecutor,
+ org.apache.oozie.dag.action.hadoop.FsActionExecutor
+ ",
+ oozie.service.CallableQueueService.threads.min: "10",
+ oozie.service.DBLiteWorkflowStoreService.oozie.autoinstall: "true",
+ ...
+}
+```
+
+#### Oozie Instrumentation
+
+_Identical to the corresponding Oozie v0 WS API_
+
+Deprecated and by default disabled since 5.0.0.
+
+A HTTP GET request returns the Oozie instrumentation information. Keep in mind that timers and counters that the Oozie server
+hasn't incremented yet will not show up.
+
+**Note:** If Instrumentation is enabled, then Metrics is unavailable.
+
+**Request:**
+
+
+```
+GET /oozie/v1/admin/instrumentation
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ timers: [
+ {
+ group: "db",
+ data: [
+ {
+ ownMinTime: 2,
+ ownTimeStdDev: 0,
+ totalTimeStdDev: 0,
+ ownTimeAvg: 3,
+ ticks: 117,
+ name: "update-workflow",
+ ownMaxTime: 32,
+ totalMinTime: 2,
+ totalMaxTime: 32,
+ totalTimeAvg: 3
+ },
+ ...
+ ]
+ },
+ ...
+ ],
+ samplers: [
+ {
+ group: "callablequeue",
+ data: [
+ {
+ name: "threads.active",
+ value: 1.8333333333333333
+ },
+ {
+ name: "delayed.queue.size",
+ value: 0
+ },
+ {
+ name: "queue.size",
+ value: 0
+ }
+ ]
+ },
+ ...
+ ],
+ variables: [
+ {
+ group: "jvm",
+ data: [
+ {
+ name: "max.memory",
+ value: 506920960
+ },
+ {
+ name: "total.memory",
+ value: 56492032
+ },
+ {
+ name: "free.memory",
+ value: 45776800
+ }
+ ]
+ },
+ ...
+ ]
+}
+```
+
+#### Oozie Metrics
+
+_Available in the Oozie v2 WS API and later_
+
+A HTTP GET request returns the Oozie metrics information. Keep in mind that timers and counters that the Oozie server
+hasn't incremented yet will not show up.
+
+
+**Note:** If Metrics is enabled, then Instrumentation is unavailable.
+
+**Note:** by default enabled since 5.0.0.
+
+**Request:**
+
+
+```
+GET /oozie/v2/admin/metrics
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ "gauges" : {
+ "jvm.memory.non-heap.committed" : {
+ "value" : 62590976
+ },
+ "oozie.mode" : {
+ "value" : "NORMAL"
+ },
+ ...
+ },
+ "timers" : {
+ "commands.action.end.call.timer" : {
+ "mean" : 108.5,
+ "p50" : 111.5,
+ "p75" : 165.5,
+ "p999" : 169,
+ "count" : 4,
+ "p95" : 169,
+ "max" : 169,
+ "mean_rate" : 0,
+ "duration_units" : "milliseconds",
+ "p98" : 169,
+ "m1_rate" : 0,
+ "rate_units" : "calls/millisecond",
+ "m15_rate" : 0,
+ "stddev" : 62.9417720330995,
+ "m5_rate" : 0,
+ "p99" : 169,
+ "min" : 42
+ },
+ ...
+ },
+ "histograms" : {
+ "callablequeue.threads.active.histogram" : {
+ "p999" : 1,
+ "mean" : 0.0625,
+ "min" : 0,
+ "p75" : 0,
+ "p95" : 1,
+ "count" : 48,
+ "p98" : 1,
+ "stddev" : 0.24462302739504083,
+ "max" : 1,
+ "p99" : 1,
+ "p50" : 0
+ },
+ ...
+ },
+ "counters" : {
+ "commands.job.info.executions" : {
+ "count" : 9
+ },
+ "jpa.CoordJobsGetPendingJPAExecutor" : {
+ "count" : 1
+ },
+ "jpa.GET_WORKFLOW" : {
+ "count" : 10
+ },
+ ...
+ }
+}
+```
+
+#### Version
+
+_Identical to the corresponding Oozie v0 WS API_
+
+A HTTP GET request returns the Oozie build version.
+
+**Request:**
+
+
+```
+GET /oozie/v1/admin/build-version
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{buildVersion: "3.0.0-SNAPSHOT" }
+```
+
+#### Available Time Zones
+
+A HTTP GET request returns the available time zones.
+
+**Request:**
+
+
+```
+GET /oozie/v1/admin/available-timezones
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ "available-timezones":[
+ {
+ "timezoneDisplayName":"SST (Pacific\/Midway)",
+ "timezoneId":"Pacific\/Midway"
+ },
+ {
+ "timezoneDisplayName":"NUT (Pacific\/Niue)",
+ "timezoneId":"Pacific\/Niue"
+ },
+ {
+ "timezoneDisplayName":"SST (Pacific\/Pago_Pago)",
+ "timezoneId":"Pacific\/Pago_Pago"
+ },
+ {
+ "timezoneDisplayName":"SST (Pacific\/Samoa)",
+ "timezoneId":"Pacific\/Samoa"
+ },
+ {
+ "timezoneDisplayName":"SST (US\/Samoa)",
+ "timezoneId":"US\/Samoa"
+ },
+ {
+ "timezoneDisplayName":"HAST (America\/Adak)",
+ "timezoneId":"America\/Adak"
+ },
+ {
+ "timezoneDisplayName":"HAST (America\/Atka)",
+ "timezoneId":"America\/Atka"
+ },
+ {
+ "timezoneDisplayName":"HST (HST)",
+ "timezoneId":"HST"
+ },
+ ...
+ ]
+}
+```
+
+#### Queue Dump
+
+A HTTP GET request returns the queue dump of the Oozie system. This is an administrator debugging feature.
+
+**Request:**
+
+
+```
+
+GET /oozie/v1/admin/queue-dump
+```
+
+#### Available Oozie Servers
+
+A HTTP GET request returns the list of available Oozie Servers. This is useful when Oozie is configured
+for [High Availability](AG_Install.html#HA); if not, it will simply return the one Oozie Server.
+
+**Request:**
+
+
+```
+GET /oozie/v2/admin/available-oozie-servers
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ "hostA": "http://hostA:11000/oozie",
+ "hostB": "http://hostB:11000/oozie",
+ "hostC": "http://hostC:11000/oozie",
+}
+```
+
+#### List available sharelib
+A HTTP GET request to get list of available sharelib.
+If the name of the sharelib is passed as an argument (regex supported) then all corresponding files are also listed.
+
+**Request:**
+
+
+```
+GET /oozie/v2/admin/list_sharelib
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+{
+ "sharelib":
+ [
+ "oozie",
+ "hive",
+ "distcp",
+ "hcatalog",
+ "sqoop",
+ "mapreduce-streaming",
+ "pig"
+ ]
+}
+```
+
+**Request:**
+
+
+```
+GET /oozie/v2/admin/list_sharelib?lib=pig*
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+
+{
+ "sharelib":
+ [
+ {
+ "pig":
+ {
+ "sharelibFiles":
+ [
+ hdfs://localhost:9000/user/purushah/share/lib/lib_20131114095729/pig/pig.jar
+ hdfs://localhost:9000/user/purushah/share/lib/lib_20131114095729/pig/piggybank.jar
+ ]
+ }
+ }
+ ]
+}
+```
+
+
+#### Update system sharelib
+This webservice call makes the oozie server(s) to pick up the latest version of sharelib present
+under oozie.service.WorkflowAppService.system.libpath directory based on the sharelib directory timestamp or reloads
+the sharelib metafile if one is configured. The main purpose is to update the sharelib on the oozie server without restarting.
+
+
+**Request:**
+
+
+```
+GET /oozie/v2/admin/update_sharelib
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+[
+ {
+ "sharelibUpdate":{
+ "host":"server1",
+ "status":"Server not found"
+ }
+ },
+ {
+ "sharelibUpdate":{
+ "host":"server2",
+ "status":"Successful",
+ "sharelibDirOld":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218",
+ "sharelibDirNew":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218"
+ }
+ },
+ {
+ "sharelibUpdate":{
+ "host":"server3",
+ "status":"Successful",
+ "sharelibDirOld":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218",
+ "sharelibDirNew":"hdfs://localhost:51951/user/purushah/share/lib/lib_20140107181218"
+ }
+ }
+]
+```
+
+#### Purge Command
+
+Oozie admin purge command cleans up the Oozie Workflow/Coordinator/Bundle records based on the parameters.
+The unit for parameters is day.
+
+Purge command will delete the workflow records (wf=30) older than 30 days, coordinator records (coord=7) older than 7 days and
+bundle records (bundle=7) older than 7 days. The limit (limit=10) defines, number of records to be fetch at a time. Turn
+(oldCoordAction`true/false) `on/off= coordinator action record purging for long running coordinators. If any of the parameter is
+not provided, then it will be taken from the `oozie-default/oozie-site` configuration.
+
+**Request:**
+
+
+```
+
+GET /oozie/v2/admin/purge?wf=30&coord=7&bundle=7&limit=10&oldCoordAction=true
+
+```
+
+**Response:**
+
+
+```
+
+{
+ "purge": "Purge command executed successfully"
+}
+
+```
+
+### Job and Jobs End-Points
+
+_Modified in Oozie v1 WS API_
+
+These endpoints are for submitting, managing and retrieving information of workflow, coordinator, and bundle jobs.
+
+#### Job Submission
+
+#### Standard Job Submission
+
+An HTTP POST request with an XML configuration as payload creates a job.
+
+The type of job is determined by the presence of one of the following 3 properties:
+
+ * `oozie.wf.application.path` : path to a workflow application directory, creates a workflow job
+ * `oozie.coord.application.path` : path to a coordinator application file, creates a coordinator job
+ * `oozie.bundle.application.path` : path to a bundle application file, creates a bundle job
+
+Or, if none of those are present, the jobtype parameter determines the type of job to run. It can either be mapreduce or pig.
+
+**Request:**
+
+
+```
+POST /oozie/v1/jobs
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>user.name</name>
+ <value>bansalm</value>
+ </property>
+ <property>
+ <name>oozie.wf.application.path</name>
+ <value>hdfs://foo:8020/user/bansalm/myapp/</value>
+ </property>
+ ...
+</configuration>
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 201 CREATED
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "job-3"
+}
+```
+
+A created job will be in `PREP` status. If the query string parameter 'action=start' is provided in
+the POST URL, the job will be started immediately and its status will be `RUNNING`.
+
+Coordinator jobs with start time in the future they will not create any action until the start time
+happens.
+
+A coordinator job will remain in `PREP` status until it's triggered, in which case it will change to `RUNNING` status.
+The 'action=start' parameter is not valid for coordinator jobs.
+
+#### Proxy MapReduce Job Submission
+
+You can submit a Workflow that contains a single MapReduce action without writing a workflow.xml. Any required Jars or other files
+must already exist in HDFS.
+
+The following properties are required; any additional parameters needed by the MapReduce job can also be specified here:
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `mapred.mapper.class`: The map-task classname
+ * `mapred.reducer.class`: The reducer-task classname
+ * `mapred.input.dir`: The map-task input directory
+ * `mapred.output.dir`: The reduce-task output directory
+ * `user.name`: The username of the user submitting the job
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
+
+**Request:**
+
+
+```
+POST /oozie/v1/jobs?jobtype=mapreduce
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>fs.default.name</name>
+ <value>hdfs://localhost:8020</value>
+ </property>
+ <property>
+ <name>mapred.job.tracker</name>
+ <value>localhost:8021</value>
+ </property>
+ <property>
+ <name>mapred.mapper.class</name>
+ <value>org.apache.oozie.example.SampleMapper</value>
+ </property>
+ <property>
+ <name>mapred.reducer.class</name>
+ <value>org.apache.oozie.example.SampleReducer</value>
+ </property>
+ <property>
+ <name>mapred.input.dir</name>
+ <value>hdfs://localhost:8020/user/rkanter/examples/input-data/text</value>
+ </property>
+ <property>
+ <name>mapred.output.dir</name>
+ <value>hdfs://localhost:8020/user/rkanter/examples/output-data/map-reduce</value>
+ </property>
+ <property>
+ <name>user.name</name>
+ <value>rkanter</value>
+ </property>
+ <property>
+ <name>oozie.libpath</name>
+ <value>hdfs://localhost:8020/user/rkanter/examples/apps/map-reduce/lib</value>
+ </property>
+ <property>
+ <name>oozie.proxysubmission</name>
+ <value>true</value>
+ </property>
+</configuration>
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 201 CREATED
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "job-3"
+}
+```
+
+#### Proxy Pig Job Submission
+
+You can submit a Workflow that contains a single Pig action without writing a workflow.xml. Any required Jars or other files must
+already exist in HDFS.
+
+The following properties are required:
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `user.name`: The username of the user submitting the job
+ * `oozie.pig.script`: Contains the pig script you want to run (the actual script, not a file path)
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
+
+The following properties are optional:
+
+ * `oozie.pig.script.params.size`: The number of parameters you'll be passing to Pig
+ required
+ * `oozie.pig.script.params.n`: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
+ * `oozie.pig.options.size`: The number of options you'll be passing to Pig
+ * `oozie.pig.options.n`: An argument to pass to Pig, the 'n' should be an integer starting with 0 to indicate the option number
+
+The `oozie.pig.options.n` parameters are sent directly to Pig without any modification unless they start with `-D`, in which case
+they are put into the `<configuration>` element of the action.
+
+In addition to passing parameters to Pig with `oozie.pig.script.params.n`, you can also create a properties file on HDFS and
+reference it with the `-param_file` option in `oozie.pig.script.options.n`; both are shown in the following example.
+
+
+```
+$ hadoop fs -cat /user/rkanter/pig_params.properties
+INPUT=/user/rkanter/examples/input-data/text
+```
+
+**Request:**
+
+
+```
+POST /oozie/v1/jobs?jobtype=pig
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>fs.default.name</name>
+ <value>hdfs://localhost:8020</value>
+ </property>
+ <property>
+ <name>mapred.job.tracker</name>
+ <value>localhost:8021</value>
+ </property>
+ <property>
+ <name>user.name</name>
+ <value>rkanter</value>
+ </property>
+ <property>
+ <name>oozie.pig.script</name>
+ <value>
+ A = load '$INPUT' using PigStorage(':');
+ B = foreach A generate $0 as id;
+ store B into '$OUTPUT' USING PigStorage();
+ </value>
+ </property>
+ <property>
+ <name>oozie.pig.script.params.size</name>
+ <value>1</value>
+ </property>
+ <property>
+ <name>oozie.pig.script.params.0</name>
+ <value>OUTPUT=/user/rkanter/examples/output-data/pig</value>
+ </property>
+ <property>
+ <name>oozie.pig.options.size</name>
+ <value>2</value>
+ </property>
+ <property>
+ <name>oozie.pig.options.0</name>
+ <value>-param_file</value>
+ </property>
+ <property>
+ <name>oozie.pig.options.1</name>
+ <value>hdfs://localhost:8020/user/rkanter/pig_params.properties</value>
+ </property>
+ <property>
+ <name>oozie.libpath</name>
+ <value>hdfs://localhost:8020/user/rkanter/share/lib/pig</value>
+ </property>
+ <property>
+ <name>oozie.proxysubmission</name>
+ <value>true</value>
+ </property>
+</configuration>
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 201 CREATED
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "job-3"
+}
+```
+
+#### Proxy Hive Job Submission
+
+You can submit a Workflow that contains a single Hive action without writing a workflow.xml. Any required Jars or other files must
+already exist in HDFS.
+
+The following properties are required:
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `user.name`: The username of the user submitting the job
+ * `oozie.hive.script`: Contains the hive script you want to run (the actual script, not a file path)
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
+
+The following properties are optional:
+
+ * `oozie.hive.script.params.size`: The number of parameters you'll be passing to Hive
+ * `oozie.hive.script.params.n`: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
+ * `oozie.hive.options.size`: The number of options you'll be passing to Hive
+ * `oozie.hive.options.n`: An argument to pass to Hive, the 'n' should be an integer starting with 0 to indicate the option number
+
+The `oozie.hive.options.n` parameters are sent directly to Hive without any modification unless they start with `-D`, in which case
+they are put into the `<configuration>` element of the action.
+
+**Request:**
+
+
+```
+POST /oozie/v1/jobs?jobtype=hive
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>fs.default.name</name>
+ <value>hdfs://localhost:8020</value>
+ </property>
+ <property>
+ <name>mapred.job.tracker</name>
+ <value>localhost:8021</value>
+ </property>
+ <property>
+ <name>user.name</name>
+ <value>rkanter</value>
+ </property>
+ <property>
+ <name>oozie.hive.script</name>
+ <value>
+ CREATE EXTERNAL TABLE test (a INT) STORED AS TEXTFILE LOCATION '${INPUT}';
+ INSERT OVERWRITE DIRECTORY '${OUTPUT}' SELECT * FROM test;
+ </value>
+ </property>
+ <property>
+ <name>oozie.hive.script.params.size</name>
+ <value>2</value>
+ </property>
+ <property>
+ <name>oozie.hive.script.params.0</name>
+ <value>OUTPUT=/user/rkanter/examples/output-data/hive</value>
+ </property>
+ <property>
+ <name>oozie.hive.script.params.1</name>
+ <value>INPUT=/user/rkanter/examples/input-data/table</value>
+ </property>
+ <property>
+ <name>oozie.libpath</name>
+ <value>hdfs://localhost:8020/user/rkanter/share/lib/hive</value>
+ </property>
+ <property>
+ <name>oozie.proxysubmission</name>
+ <value>true</value>
+ </property>
+</configuration>
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 201 CREATED
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "job-3"
+}
+```
+
+#### Proxy Sqoop Job Submission
+
+You can submit a Workflow that contains a single Sqoop command without writing a workflow.xml. Any required Jars or other
+ files must already exist in HDFS.
+
+The following properties are required:
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `user.name`: The username of the user submitting the job
+ * `oozie.sqoop.command`: The sqoop command you want to run where each argument occupies one line or separated by "\n"
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
+
+The following properties are optional:
+
+ * `oozie.sqoop.options.size`: The number of options you'll be passing to Sqoop Hadoop job
+ * `oozie.sqoop.options.n`: An argument to pass to Sqoop hadoop job conf, the 'n' should be an integer starting with 0 to indicate the option number
+
+**Request:**
+
+
+```
+POST /oozie/v1/jobs?jobtype=sqoop
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>fs.default.name</name>
+ <value>hdfs://localhost:8020</value>
+ </property>
+ <property>
+ <name>mapred.job.tracker</name>
+ <value>localhost:8021</value>
+ </property>
+ <property>
+ <name>user.name</name>
+ <value>bzhang</value>
+ </property>
+ <property>
+ <name>oozie.sqoop.command</name>
+ <value>
+ import
+ --connect
+ jdbc:mysql://localhost:3306/oozie
+ --username
+ oozie
+ --password
+ oozie
+ --table
+ WF_JOBS
+ --target-dir
+ /user/${wf:user()}/${examplesRoot}/output-data/sqoop
+ </value>
+ </property>
+ <name>oozie.libpath</name>
+ <value>hdfs://localhost:8020/user/bzhang/share/lib/sqoop</value>
+ </property>
+ <property>
+ <name>oozie.proxysubmission</name>
+ <value>true</value>
+ </property>
+</configuration>
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 201 CREATED
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "job-3"
+}
+```
+
+#### Managing a Job
+
+A HTTP PUT request starts, suspends, resumes, kills, update or dryruns a job.
+
+**Request:**
+
+
+```
+PUT /oozie/v1/job/job-3?action=start
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+```
+
+Valid values for the 'action' parameter are 'start', 'suspend', 'resume', 'kill', 'dryrun', 'rerun', and 'change'.
+
+Rerunning and changing a job require additional parameters, and are described below:
+
+##### Re-Running a Workflow Job
+
+A workflow job in `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun specifying a list
+of workflow nodes to skip during the rerun. All the nodes in the skip list must have complete its
+execution.
+
+The rerun job will have the same job ID.
+
+A rerun request is done with a HTTP PUT request with a `rerun` action.
+
+**Request:**
+
+
+```
+PUT /oozie/v1/job/job-3?action=rerun
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <property>
+ <name>user.name</name>
+ <value>tucu</value>
+ </property>
+ <property>
+ <name>oozie.wf.application.path</name>
+ <value>hdfs://foo:8020/user/tucu/myapp/</value>
+ </property>
+ <property>
+ <name>oozie.wf.rerun.skip.nodes</name>
+ <value>firstAction,secondAction</value>
+ </property>
+ ...
+</configuration>
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+```
+
+##### Re-Running a coordinator job
+
+A coordinator job in `RUNNING` `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun by specifying the coordinator actions
+to re-execute.
+
+A rerun request is done with an HTTP PUT request with a `coord-rerun` `action`.
+
+The `type` of the rerun can be `date` or `action`.
+
+The `scope` of the rerun depends on the type:
+* `date`: a comma-separated list of date ranges. Each date range element is specified with dates separated by `::`
+* `action`: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by `-`
+
+The `refresh` parameter can be `true` or `false` to specify if the user wants to refresh an action's input and output events.
+
+The `nocleanup` parameter can be `true` or `false` to specify is the user wants to cleanup output events for the rerun actions.
+
+**Request:**
+
+
+```
+PUT /oozie/v1/job/job-3?action=coord-rerun&type=action&scope=1-2&refresh=false&nocleanup=false
+.
+```
+
+or
+
+
+```
+PUT /oozie/v1/job/job-3?action=coord-rerun&type=date2009-02-01T00:10Z::2009-03-01T00:10Z&scope=&refresh=false&nocleanup=false
+.
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+```
+
+##### Re-Running a bundle job
+
+A coordinator job in `RUNNING` `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun by specifying the coordinators to
+re-execute.
+
+A rerun request is done with an HTTP PUT request with a `bundle-rerun` `action`.
+
+A comma separated list of coordinator job names (not IDs) can be specified in the `coord-scope` parameter.
+
+The `date-scope` parameter is a comma-separated list of date ranges. Each date range element is specified with dates separated
+by `::`. If empty or not included, Oozie will figure this out for you
+
+The `refresh` parameter can be `true` or `false` to specify if the user wants to refresh the coordinator's input and output events.
+
+The `nocleanup` parameter can be `true` or `false` to specify is the user wants to cleanup output events for the rerun coordinators.
+
+**Request:**
+
+
+```
+PUT /oozie/v1/job/job-3?action=bundle-rerun&coord-scope=coord-1&refresh=false&nocleanup=false
+.
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+```
+
+
+##### Changing endtime/concurrency/pausetime of a Coordinator Job
+
+A coordinator job not in `KILLED` status can have it's endtime, concurrency, or pausetime changed.
+
+A change request is done with an HTTP PUT request with a `change` `action`.
+
+The `value` parameter can contain any of the following:
+* endtime: the end time of the coordinator job.
+* concurrency: the concurrency of the coordinator job.
+* pausetime: the pause time of the coordinator job.
+
+Multiple arguments can be passed to the `value` parameter by separating them with a ';' character.
+
+If an already-succeeded job changes its end time, its status will become running.
+
+**Request:**
+
+
+```
+PUT /oozie/v1/job/job-3?action=change&value=endtime=2011-12-01T05:00Z
+.
+```
+
+or
+
+
+```
+PUT /oozie/v1/job/job-3?action=change&value=concurrency=100
+.
+```
+
+or
+
+
+```
+PUT /oozie/v1/job/job-3?action=change&value=pausetime=2011-12-01T05:00Z
+.
+```
+
+or
+
+
+```
+PUT /oozie/v1/job/job-3?action=change&value=endtime=2011-12-01T05:00Z;concurrency=100;pausetime=2011-12-01T05:00Z
+.
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+```
+
+##### Updating coordinator definition and properties
+Existing coordinator definition and properties will be replaced by new definition and properties. Refer [Updating coordinator definition and properties](DG_CommandLineTool.html#Updating_coordinator_definition_and_properties)
+
+
+```
+PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+{"update":
+ {"diff":"**********Job definition changes**********\n******************************************\n**********Job conf changes****************\n@@ -8,16 +8,12 @@\n
+ <value>hdfs:\/\/localhost:9000\/user\/purushah\/examples\/apps\/aggregator\/coordinator.xml<\/value>\r\n <\/property>\r\n <property>\r\n
+ - <name>user.name<\/name>\r\n
+ - <value>purushah<\/value>\r\n
+ - <\/property>\r\n
+ - <property>\r\n <name>start<\/name>\r\n
+ <value>2010-01-01T01:00Z<\/value>\r\n <\/property>\r\n <property>\r\n
+ - <name>newproperty<\/name>\r\n
+ - <value>new<\/value>\r\n
+ + <name>user.name<\/name>\r\n
+ + <value>purushah<\/value>\r\n <\/property>\r\n <property>\r\n
+ <name>queueName<\/name>\r\n******************************************\n"
+ }
+}
+```
+
+
+#### Job Information
+
+A HTTP GET request retrieves the job information.
+
+**Request:**
+
+
+```
+GET /oozie/v1/job/job-3?show=info&timezone=GMT
+```
+
+**Response for a workflow job:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "0-200905191240-oozie-W",
+ appName: "indexer-workflow",
+ appPath: "hdfs://user/bansalm/indexer.wf",
+ externalId: "0-200905191230-oozie-pepe",
+ user: "bansalm",
+ status: "RUNNING",
+ conf: "<configuration> ... </configuration>",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: null,
+ run: 0,
+ actions: [
+ {
+ id: "0-200905191240-oozie-W@indexer",
+ name: "indexer",
+ type: "map-reduce",
+ conf: "<configuration> ...</configuration>",
+ startTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ status: "OK",
+ externalId: "job-123-200903101010",
+ externalStatus: "SUCCEEDED",
+ trackerUri: "foo:8021",
+ consoleUrl: "http://foo:50040/jobdetailshistory.jsp?jobId=...",
+ transition: "reporter",
+ data: null,
+ errorCode: null,
+ errorMessage: null,
+ retries: 0
+ },
+ ...
+ ]
+}
+```
+
+**Response for a coordinator job:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ id: "0-200905191240-oozie-C",
+ appName: "indexer-Coord",
+ appPath: "hdfs://user/bansalm/myapp/logprocessor-coord.xml",
+ externalId: "0-200905191230-oozie-pepe",
+ user: "bansalm",
+ status: "RUNNING",
+ conf: "<configuration> ... </configuration>",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
+ frequency: "${days(1)}"
+ actions: [
+ {
+ id: "0000010-130426111815091-oozie-bansalm-C@1",
+ createdTime: "Fri, 26 Apr 2013 20:57:07 GMT",
+ externalId: "",
+ missingDependencies: "",
+ runConf: null,
+ createdConf: null,
+ consoleUrl: null,
+ nominalTime: "Fri, 01 Jan 2010 01:00:00 GMT",
+ ...
+}
+```
+
+**Response for a bundle job:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ jobType: "bundle",
+ id: "0-200905191240-oozie-B",
+ appName: "new-bundle",
+ appPath: "hdfs://user/bansalm/myapp/logprocessor-bundle.xml",
+ externalId: "0-200905191230-oozie-pepe",
+ user: "bansalm",
+ status: "RUNNING",
+ conf: "<configuration> ... </configuration>",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 31 Dec 2009 00:00:00 GMT"
+ bundleCoordJobs: [
+ {
+ status: "RUNNING",
+ concurrency: 1,
+ conf: "<configuration> ... </configuration>",
+ executionPolicy: "FIFO",
+ toString: "Coordinator application id[0000010-130426111815091-oozie-bansalm-C] status[RUNNING]",
+ coordJobName: "coord-1",
+ endTime: "Fri, 01 Jan 2010 03:00:00 GMT",
+ ...
+ }
+ ...
+}
+```
+
+**Getting all the Workflows corresponding to a Coordinator Action:**
+
+A coordinator action kicks off different workflows for its original run and all subsequent reruns.
+Getting a list of those workflow ids is a useful tool to keep track of your actions' runs and
+to go debug the workflow job logs if required. Along with ids, it also lists their statuses,
+and start and end times for quick reference.
+
+Both v1 and v2 API are supported. v0 is not supported.
+
+
+```
+GET /oozie/v2/job/0000001-111219170928042-oozie-joe-C@1?show=allruns
+```
+
+**Response**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{"workflows":[
+ {
+ "startTime":"Mon, 24 Mar 2014 23:40:53 GMT",
+ "id":"0000001-140324163709596-oozie-chit-W",
+ "status":"SUCCEEDED",
+ "endTime":"Mon, 24 Mar 2014 23:40:54 GMT"
+ },
+ {
+ "startTime":"Mon, 24 Mar 2014 23:44:01 GMT",
+ "id":"0000000-140324164318985-oozie-chit-W",
+ "status":"SUCCEEDED",
+ "endTime":"Mon, 24 Mar 2014 23:44:01 GMT"
+ },
+ {
+ "startTime":"Mon, 24 Mar 2014 23:44:24 GMT",
+ "id":"0000001-140324164318985-oozie-chit-W",
+ "status":"SUCCEEDED",
+ "endTime":"Mon, 24 Mar 2014 23:44:24 GMT"
+ }
+]}
+```
+
+An alternate API is also available for the same output. With this API, one can pass the coordinator **JOB** Id
+followed by query params - `type=action` and `scope=<action-number>`. One single action number can be passed at a time.
+
+
+```
+GET /oozie/v2/job/0000001-111219170928042-oozie-joe-C?show=allruns&type=action&scope=1
+```
+
+**Retrieve a subset of actions**
+
+Query parameters, `offset` and `length` can be specified with a workflow job to retrieve specific actions. Default is offset=0, len=1000
+
+```
+GET /oozie/v1/job/0000002-130507145349661-oozie-joe-W?show=info&offset=5&len=10
+```
+Query parameters, `offset`, `length`, `filter` can be specified with a coordinator job to retrieve specific actions.
+Query parameter, `order` with value "desc" can be used to retrieve the latest coordinator actions materialized instead of actions from @1.
+Query parameters `filter` can be used to retrieve coordinator actions matching specific status.
+Default is offset=0, len=0 for v2/job (i.e., does not return any coordinator actions) and offset=0, len=1000 with v1/job and v0/job.
+So if you need actions to be returned with v2 API, specifying `len` parameter is necessary.
+Default `order` is "asc".
+
+```
+GET /oozie/v1/job/0000001-111219170928042-oozie-joe-C?show=info&offset=5&len=10&filter=status%3DKILLED&order=desc
+```
+Note that the filter is URL encoded, its decoded value is `status=KILLED`.
+
+```
+GET /oozie/v1/job/0000001-111219170928042-oozie-joe-C?show=info&filter=status%21%3DSUCCEEDED&order=desc
+```
+This retrieves coordinator actions except for SUCCEEDED status, which is useful for debugging.
+
+**Retrieve information of the retry attempts of the workflow action:**
+
+
+```
+GET oozie/v2/job/0000000-161212175234862-oozie-puru-W@pig-node?show=retries
+```
+
+**Response**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ "retries":
+ [
+ {
+ "startTime": "Tue, 13 Dec 2016 01:54:13 GMT",
+ "consoleUrl": "http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2648",
+ "endTime": "Tue, 13 Dec 2016 01:54:20 GMT",
+ "attempt": "1"
+ },
+ {
+ "startTime": "Tue, 13 Dec 2016 01:55:20 GMT",
+ "consoleUrl": "http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2649",
+ "endTime": "Tue, 13 Dec 2016 01:55:24 GMT",
+ "attempt": "2"
+ }
+ ]
+}
+```
+
+#### Job Application Definition
+
+A HTTP GET request retrieves the workflow or a coordinator job definition file.
+
+**Request:**
+
+
+```
+GET /oozie/v1/job/job-3?show=definition
+```
+
+**Response for a workflow job:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<workflow-app name='xyz-app' xmlns="uri:oozie:workflow:0.1">
+ <start to='firstaction' />
+ ...
+ <end name='end' />
+</workflow-app>
+```
+
+**Response for a coordinator job:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<coordinator-app name='abc-app' xmlns="uri:oozie:coordinator:0.1" frequency="${days(1)}
+ start="2009-01-01T00:00Z" end="2009-12-31T00:00Z" timezone="America/Los_Angeles">
+ <datasets>
+ ...
+ </datasets>
+ ...
+</coordinator-app>
+```
+
+**Response for a bundle job:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8"?>
+<bundle-app name='abc-app' xmlns="uri:oozie:coordinator:0.1"
+ start="2009-01-01T00:00Z" end="2009-12-31T00:00Z"">
+ <datasets>
+ ...
+ </datasets>
+ ...
+</bundle-app>
+```
+
+#### Job Log
+
+An HTTP GET request retrieves the job log.
+
+**Request:**
+
+
+```
+GET /oozie/v1/job/job-3?show=log
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: text/plain;charset=UTF-8
+.
+...
+23:21:31,272 TRACE oozieapp:526 - USER[bansalm] GROUP[other] TOKEN[-] APP[test-wf] JOB[0-20090518232130-oozie-tucu] ACTION[mr-1] Start
+23:21:31,305 TRACE oozieapp:526 - USER[bansalm] GROUP[other] TOKEN[-] APP[test-wf] JOB[0-20090518232130-oozie-tucu] ACTION[mr-1] End
+...
+```
+
+#### Job Error Log
+
+An HTTP GET request retrieves the job error log.
+
+**Request:**
+
+
+```
+GET /oozie/v2/job/0000000-150121110331712-oozie-puru-B?show=errorlog
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: text/plain;charset=UTF-8
+2015-01-21 11:33:29,090 WARN CoordSubmitXCommand:523 - SERVER[-] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150121110331712-oozie-puru-B] ACTION[] SAXException :
+org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-type.2.4.a: Invalid content was found starting with element 'concurrency'. One of '{"uri:oozie:coordinator:0.2":controls, "uri:oozie:coordinator:0.2":datasets, "uri:oozie:coordinator:0.2":input-events, "uri:oozie:coordinator:0.2":output-events, "uri:oozie:coordinator:0.2":action}' is expected.
+ at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
+ at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
+...
+```
+
+
+#### Job Audit Log
+
+An HTTP GET request retrieves the job audit log.
+
+**Request:**
+
+
+```
+GET /oozie/v2/job/0000000-150322000230582-oozie-puru-C?show=auditlog
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: text/plain;charset=UTF-8
+2015-03-22 00:04:35,494 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 00:05:13,823 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 00:06:59,561 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 23:22:20,012 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+```
+
+
+#### Filtering the server logs with logfilter options
+User can provide multiple option to filter logs using -logfilter opt1=val1;opt2=val1;opt3=val1. This can be used to fetch only just logs of interest faster as fetching Oozie server logs is slow due to the overhead of pattern matching.
+
+
+```
+GET /oozie/v1/job/0000003-140319184715726-oozie-puru-C?show=log&logfilter=limit=3;loglevel=WARN
+```
+
+
+Refer to the [Filtering the server logs with logfilter options](DG_CommandLineTool.html#Filtering_the_server_logs_with_logfilter_options) for more details.
+
+
+#### Job graph
+
+An `HTTP GET` request returns the image of the workflow DAG (rendered as a PNG or SVG image, or as a DOT string).
+
+ * The nodes that are being executed are painted yellow
+ * The nodes that have successfully executed are painted green
+ * The nodes that have failed execution are painted red
+ * The nodes that are yet to be executed are pained gray
+ * An arc painted green marks the successful path taken so far
+ * An arc painted red marks the failure of the node and highlights the _error_ action
+ * An arc painted gray marks a path not taken yet
+
+**PNG request:**
+
+```
+GET /oozie/v1/job/job-3?show=graph[&show-kill=true][&format=png]
+```
+
+**PNG response:**
+
+```
+HTTP/1.1 200 OK
+Content-Type: image/png
+Content-Length: {image_size_in_bytes}
+
+{image_bits}
+```
+
+**SVG request:**
+
+```
+GET /oozie/v1/job/job-3?show=graph[&show-kill=true]&format=svg
+```
+
+**SVG response:**
+
+```
+HTTP/1.1 200 OK
+Content-Type: image/svg+xml
+Content-Length: {image_size_in_bytes}
+
+{image_bits}
+```
+
+**DOT request:**
+
+```
+GET /oozie/v1/job/job-3?show=graph[&show-kill=true]&format=dot
+```
+
+**DOT response:**
+
+```
+HTTP/1.1 200 OK
+Content-Type: text/plain
+Content-Length: {dot_size_in_bytes}
+
+{dot_bytes}
+```
+
+The optional `show-kill` parameter shows `kill` node in the graph. Valid values for this parameter are `1`, `yes`, and `true`.
+This parameter has no effect when workflow fails and the failure node leads to the `kill` node; in that case `kill` node is shown
+always.
+
+The optional `format` parameter describes whether the response has to be rendered as a PNG image, or an SVG image, or a DOT string.
+When omitted, `format` is considered as `png` for backwards compatibility. Oozie Web UI uses the `svg` `format`.
+
+The node labels are the node names provided in the workflow XML.
+
+This API returns `HTTP 400` when run on a resource other than a workflow, viz. bundle and coordinator.
+
+#### Job Status
+
+An `HTTP GET` request that returns the current status (e.g. `SUCCEEDED`, `KILLED`, etc) of a given job. If you are only interested
+in the status, and don't want the rest of the information that the `info` query provides, it is recommended to use this call
+as it is more efficient.
+
+**Request**
+
+```
+GET /oozie/v2/job/0000000-140908152307821-oozie-rkan-C?show=status
+```
+
+**Response**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ "status" : "SUCCEEDED"
+}
+```
+
+It accepts any valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.
+
+#### Changing job SLA definition and alerting
+An `HTTP PUT` request to change job SLA alert status/SLA definition.
+
+ * All sla commands takes actions-list or date parameter.
+ * `date`: a comma-separated list of date ranges. Each date range element is specified with dates separated by `::`
+ * `action-list`: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by `-`
+ * For bundle jobs additional `coordinators` (coord_name/id) parameter can be passed.
+ * Sla change command need extra parameter `value` to specify new sla definition.
+ * Changing SLA definition
+
+ SLA definition of should-start, should-end, nominal-time and max-duration can be changed.
+
+
+```
+PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-change&value=<key>=<value>;...;<key>=<value>
+```
+
+ * Disabling SLA alert
+
+
+```
+PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-disable&action-list=3-4
+```
+Will disable SLA alert for actions 3 and 4.
+
+
+```
+PUT /oozie/v1/job/0000003-140319184715726-oozie-puru-C?action=sla-disable&date=2009-02-01T00:10Z::2009-03-01T00:10Z
+```
+Will disable SLA alert for actions whose nominal time is in-between 2009-02-01T00:10Z 2009-03-01T00:10Z (inclusive).
+
+
+
+```
+PUT /oozie/v1/job/0000004-140319184715726-oozie-puru-B?action=sla-disable&date=2009-02-01T00:10Z::2009-03-01T00:10Z&coordinators=abc
+```
+For bundle jobs additional coordinators (list of comma separated coord_name/id) parameter can be passed.
+
+ * Enabling SLA alert
+
+
+```
+PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-enable&action-list=1,14,17-20
+```
+Will enable SLA alert for actions 1,14,17,18,19,20.
+
+### Getting missing dependencies of coordinator action(s)
+
+
+```
+GET oozie/v2/job/0000000-170104115137443-oozie-puru-C?show=missing-dependencies&action-list=1,20
+```
+
+**Response**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+
+{
+"missingDependencies":
+[{
+ "blockedOn": "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS",
+ "dataSets":
+ [
+ {
+ "missingDependencies":
+ [
+ "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS"
+ ],
+ "dataSet": "input-2"
+ }
+ ],
+ "id": 1
+ },
+ {
+ "blockedOn": "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/20/00/_SUCCESS",
+ "dataSets":
+ [
+ {
+ "missingDependencies":
+ [
+ "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/20/00/_SUCCESS",
+ "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/19/40/_SUCCESS",
+ "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/19/20/_SUCCESS",
+ "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/19/00/_SUCCESS",
+ "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/18/40/_SUCCESS",
+ "hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/18/20/_SUCCESS"
+ ],
+ "dataSet": "input-2"
+ }
+ ],
+ "id": 20
+ }]
+}
+```
+#### Jobs Information
+
+A HTTP GET request retrieves workflow and coordinator jobs information.
+
+**Request:**
+
+
+```
+GET /oozie/v1/jobs?filter=user%3Dbansalm&offset=1&len=50&timezone=GMT
+```
+
+Note that the filter is URL encoded, its decoded value is `user=bansalm`.
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ offset: 1,
+ len: 50,
+ total: 1002,
+**jobs: [
+ {
+** jobType: "workflow"
+ id: "0-200905191240-oozie-W",
+ appName: "indexer-workflow",
+ appPath: "hdfs://user/tucu/indexer-wf",
+ user: "bansalm",
+ group: "other",
+ status: "RUNNING",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: null,
+ info: "run=0",
+ },
+ {
+** jobType: "coordinator"
+ id: "0-200905191240-oozie-C",
+ appName: "logprocessor-coord",
+ appPath: "hdfs://user/bansalm/myapp/logprocessor-coord.xml",
+ user: "bansalm",
+ group: "other",
+ status: "RUNNING",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
+ info: "nextAction=5",
+ },
+ {
+** jobType: "bundle"
+ id: "0-200905191240-oozie-B",
+ appName: "logprocessor-bundle",
+ appPath: "hdfs://user/bansalm/myapp/logprocessor-bundle.xml",
+ user: "bansalm",
+ group: "other",
+ status: "RUNNING",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
+ },
+ ...
+ ]
+}
+```
+
+No action information is returned when querying for multiple jobs.
+
+
+The syntax for the filter is `[NAME=VALUE][;NAME=VALUE]*`
+
+Valid filter names are:
+
+ * text: any text that might be a part of application name or a part of user name or a complete job ID
+ * name: the application name from the workflow/coordinator/bundle definition
+ * user: the user who submitted the job
+ * group: the group for the job (support for the group filter is discontinued. version: 3.2.0 OOZIE-228).
+ * id: the id of the workflow/coordinator/bundle job
+ * status: the status of the job
+ * startCreatedTime : the start of the window about workflow job's created time
+ * endCreatedTime : the end of above window
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
+
+The query will do an AND among all the filter names.
+
+The query will do an OR among all the filter values for the same name. Multiple values must be specified as different
+name value pairs.
+
+Additionally the `offset` and `len` parameters can be used for pagination. The start parameter is base 1.
+
+Moreover, the `jobtype` parameter could be used to determine what type of job is looking for.
+The valid values of job type are: `wf`, `coordinator` or `bundle`.
+
+startCreatedTime and endCreatedTime should be specified either in **ISO8601 (UTC)** format **(yyyy-MM-dd'T'HH:mm'Z')** or
+a offset value in days or hours or minutes from the current time. For example, -2d means the (current time - 2 days),
+-3h means the (current time - 3 hours), -5m means the (current time - 5 minutes).
+
+#### Bulk modify jobs
+
+A HTTP PUT request can kill, suspend, or resume all jobs that satisfy the url encoded parameters.
+
+**Request:**
+
+
+```
+PUT /oozie/v1/jobs?action=kill&filter=name%3Dcron-coord&offset=1&len=50&jobtype=coordinator
+```
+
+This request will kill all the coordinators with name=cron-coord up to 50 of them.
+
+Note that the filter is URL encoded, its decoded value is `name=cron-coord`.
+
+The syntax for the filter is `[NAME=VALUE][;NAME=VALUE]*`
+
+Valid filter names are:
+
+ * name: the application name from the workflow/coordinator/bundle definition
+ * user: the user that submitted the job
+ * group: the group for the job
+ * status: the status of the job
+
+The query will do an AND among all the filter names.
+
+The query will do an OR among all the filter values for the same name. Multiple values must be specified as different
+name value pairs.
+
+Additionally the `offset` and `len` parameters can be used for pagination. The start parameter is base 1.
+
+Moreover, the `jobtype` parameter could be used to determine what type of job is looking for.
+The valid values of job type are: `wf`, `coordinator` or `bundle`
+
+**Response:**
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ offset: 1,
+ len: 50,
+ total: 2,
+**jobs: [
+ {
+** jobType: "coordinator"
+ id: "0-200905191240-oozie-C",
+ appName: "cron-coord",
+ appPath: "hdfs://user/bansalm/app/cron-coord.xml",
+ user: "bansalm",
+ group: "other",
+ status: "KILLED",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
+ info: "nextAction=5",
+ },
+ {
+** jobType: "coordinator"
+ id: "0-200905191240-oozie-C",
+ appName: "cron-coord",
+ appPath: "hdfs://user/bansalm/myapp/cron-coord.xml",
+ user: "bansalm",
+ group: "other",
+ status: "KILLED",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 31 Dec 2009 00:00:00 GMT",
+ },
+ ...
+ ]
+}
+```
+
+
+```
+PUT /oozie/v1/jobs?action=suspend&filter=status%3Drunning&offset=1&len=50&jobtype=wf
+```
+
+This request will suspend all the workflows with status=running up to 50 of them.
+Note that the filter is URL encoded, its decoded value is `status=running`.
+
+**Response:**
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ offset: 1,
+ len: 50,
+ total: 50,
+**jobs: [
+ {
+** jobType: "workflow"
+ id: "0-200905191240-oozie-W",
+ appName: "indexer-workflow",
+ appPath: "hdfs://user/tucu/indexer-wf",
+ user: "bansalm",
+ group: "other",
+ status: "SUSPENDED",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: null,
+ info: "run=0",
+ },
+ {
+** jobType: "workflow"
+ id: "0-200905191240-oozie-W",
+ appName: "logprocessor-wf",
+ appPath: "hdfs://user/bansalm/myapp/workflow.xml",
+ user: "bansalm",
+ group: "other",
+ status: "SUSPENDED",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: null,
+ info: "run=0",
+ },
+ ...
+ ]
+}
+```
+
+#### Jobs information using Bulk API
+
+A HTTP GET request retrieves a bulk response for all actions, corresponding to a particular bundle, that satisfy user specified criteria.
+This is useful for monitoring purposes, where user can find out about the status of downstream jobs with a single bulk request.
+The criteria are used for filtering the actions returned. Valid options (_case insensitive_) for these request criteria are:
+
+ * **bundle**: the application name from the bundle definition
+ * **coordinators**: the application name(s) from the coordinator definition.
+ * **actionStatus**: the status of coordinator action (Valid values are WAITING, READY, SUBMITTED, RUNNING, SUSPENDED, TIMEDOUT, SUCCEEDED, KILLED, FAILED)
+ * **startCreatedTime**: the start of the window you want to look at, of the actions' created time
+ * **endCreatedTime**: the end of above window
+ * **startScheduledTime**: the start of the window you want to look at, of the actions' scheduled i.e. nominal time.
+ * **endScheduledTime**: the end of above window
+
+Specifying 'bundle' is REQUIRED. All the rest are OPTIONAL but that might result in thousands of results depending on the size of your job. (pagination comes into play then)
+
+If no 'actionStatus' values provided, by default KILLED,FAILED will be used.
+For e.g if the query string is only "bundle=MyBundle", the response will have all actions (across all coordinators) whose status is KILLED or FAILED
+
+The query will do an AND among all the filter names, and OR among each filter name's values.
+
+
+The syntax for the request criteria is `[NAME=VALUE][;NAME=VALUE]*`
+
+For 'coordinators' and 'actionStatus', if user wants to check for multiple values, they can be passed in a comma-separated manner.
+**Note**: The query will do an OR among them. Hence no need to repeat the criteria name
+
+All the time values should be specified in **ISO8601 (UTC)** format i.e. **yyyy-MM-dd'T'HH:mm'Z'**
+
+Additionally the `offset` and `len` parameters can be used as usual for pagination. The start parameter is base 1.
+
+If you specify a coordinator in the list, that does not exist, no error is thrown; simply the response will be empty or pertaining to the other valid coordinators.
+However, if bundle name provided does not exist, an error is thrown.
+
+**Request:**
+
+
+```
+GET /oozie/v1/jobs?bulk=bundle%3Dmy-bundle-app;coordinators%3Dmy-coord-1,my-coord-5;actionStatus%3DKILLED&offset=1&len=50
+```
+
+Note that the filter is URL encoded, its decoded value is `user=chitnis`. If typing in browser URL, one can type decoded value itself i.e. using '='
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ offset: 1,
+ len: 50,
+ total: 1002,
+** bulkresponses: [
+** {
+ bulkbundle:
+ {
+ bundleJobName: "my-bundle-app",
+ bundleJobId: "0-200905191240-oozie-B",
+ status: "SUSPENDED",
+ },
+ bulkcoord:
+ {
+ coordJobName: "my-coord-1",
+ status: "SUSPENDED",
+ },
+ bulkaction:
+ {
+ id: "0-200905191240-oozie-C@21",
+ coordJobId: "0-200905191240-oozie-C",
+ actionNumber: 21,
+ externalId: "job_00076_0009",
+ status: "KILLED",
+ externalStatus: "FAILED",
+ errorCode: "E0902",
+ errorMessage: "Input file corrupt",
+ createdTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ nominalTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ missingDependencies: "hdfs://nn:port/user/joe/file.txt"
+ },
+ },
+** {
+ bulkbundle:
+ {
+ bundleJobName: "my-bundle-app",
+ bundleJobId: "0-200905191240-oozie-B",
+ status: "SUSPENDED",
+ }
+ bulkcoord:
+ {
+ coordJobName: "my-coord-5",
+ status: "SUSPENDED",
+ }
+ bulkaction:
+ {
+ id: "0-200905191245-oozie-C@114",
+ coordJobId: "0-200905191245-oozie-C",
+ actionNumber: 114,
+ externalId: "job_00076_0265",
+ status: "KILLED",
+ externalStatus: "KILLED",
+ errorCode: "E0603",
+ errorMessage: "SQL error in operation ...",
+ createdTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ nominalTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ missingDependencies:
+ }
+ }
+ ...
+ ]
+}
+```
+
+## Oozie Web Services API, V2 (Workflow , Coordinator And Bundle)
+
+The Oozie Web Services API is a HTTP REST JSON API.
+
+All responses are in `UTF-8`.
+
+Assuming Oozie is running at `OOZIE_URL`, the following web services end points are supported:
+
+ * \<OOZIE_URL\>/versions
+ * \<OOZIE_URL\>/v2/admin
+ * \<OOZIE_URL\>/v2/job
+ * \<OOZIE_URL\>/v2/jobs
+
+**Changes in v2 job API:**
+
+There is a difference in the JSON format of Job Information API (*/job) particularly for map-reduce action.
+No change for other actions.
+In v1, externalId and consoleUrl point to spawned child job ID, and externalChildIDs is null in map-reduce action.
+In v2, externalId and consoleUrl point to launcher job ID, and externalChildIDs is spawned child job ID in map-reduce action.
+
+v2 supports retrieving of JMS topic on which job notifications are sent
+
+**REST API URL:**
+
+
+```
+GET http://localhost:11000/oozie/v2/job/0000002-130507145349661-oozie-vira-W?show=jmstopic
+```
+
+**Changes in v2 admin API:**
+
+v2 adds support for retrieving JMS connection information related to JMS notifications.
+
+**REST API URL:**
+
+
+```
+GET http://localhost:11000/oozie/v2/admin/jmsinfo
+```
+
+v2/jobs remain the same as v1/jobs
+
+### Job and Jobs End-Points
+
+#### Job Information
+
+A HTTP GET request retrieves the job information.
+
+**Request:**
+
+
+```
+GET /oozie/v2/job/job-3?show=info&timezone=GMT
+```
+
+**Response for a workflow job:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+**jobType: "workflow",
+ id: "0-200905191240-oozie-W",
+ appName: "indexer-workflow",
+ appPath: "hdfs://user/bansalm/indexer.wf",
+ externalId: "0-200905191230-oozie-pepe",
+ user: "bansalm",
+ group: "other",
+ status: "RUNNING",
+ conf: "<configuration> ... </configuration>",
+ createdTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ startTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ endTime: null,
+ run: 0,
+ actions: [
+ {
+ id: "0-200905191240-oozie-W@indexer",
+ name: "indexer",
+ type: "map-reduce",
+ conf: "<configuration> ...</configuration>",
+ startTime: "Thu, 01 Jan 2009 00:00:00 GMT",
+ endTime: "Fri, 02 Jan 2009 00:00:00 GMT",
+ status: "OK",
+ externalId: "job-123-200903101010",
+ externalStatus: "SUCCEEDED",
+ trackerUri: "foo:8021",
+ consoleUrl: "http://foo:50040/jobdetailshistory.jsp?jobId=job-123-200903101010",
+ transition: "reporter",
+ data: null,
+ stats: null,
+ externalChildIDs: "job-123-200903101011"
+ errorCode: null,
+ errorMessage: null,
+ retries: 0
+ },
+ ...
+ ]
+}
+```
+
+#### Managing a Job
+##### Ignore a Coordinator Job or Action
+
+A ignore request is done with an HTTP PUT request with a `ignore`
+
+The `type` parameter supports `action` only.
+The `scope` parameter can contain coordinator action id(s) to be ignored.
+Multiple action ids can be passed to the `scope` parameter
+
+**Request:**
+
+Ignore a coordinator job
+
+```
+PUT /oozie/v2/job/job-3?action=ignore
+```
+
+Ignore coordinator actions
+
+```
+PUT /oozie/v2/job/job-3?action=ignore&type=action&scope=3-4
+```
+
+### Validate End-Point
+
+This endpoint is to validate a workflow, coordinator, bundle XML file.
+
+#### Validate a local file
+
+**Request:**
+
+
+```
+POST /oozie/v2/validate?file=/home/test/myApp/workflow.xml
+Content-Type: application/xml;charset=UTF-8
+.
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<workflow-app xmlns="uri:oozie:workflow:0.3" name="test">
+ <start to="shell"/>
+ <action name="shell">
+ <shell xmlns="uri:oozie:shell-action:0.3">
+ <job-tracker>${jobTracker}</job-tracker>
+ <name-node>${nameNode}</name-node>
+ <exec>script.sh</exec>
+ <argument></argument>
+ <file>script.sh</file>
+ <capture-output/>
+ </shell>
+ <ok to="end"/>
+ <error to="fail"/>
+ </action>
+ <kill name="fail">
+ <message>failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
+ </kill>
+ <end name="end"/>
+</workflow-app>
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ validate: "Valid workflow-app"
+}
+```
+
+#### Validate a file in HDFS
+
+You can validate a workflow, coordinator, bundle XML file in HDFS. The XML file must already exist in HDFS.
+
+**Request:**
+
+
+```
+POST /oozie/v2/validate?file=hdfs://localhost:8020/user/test/myApp/workflow.xml
+Content-Type: application/xml;charset=UTF-8
+.
+```
+
+**Response:**
+
+
+```
+HTTP/1.1 200 OK
+Content-Type: application/json;charset=UTF-8
+.
+{
+ validate: "Valid workflow-app"
+}
+```
+
+
+
+
+
+
+
+
[04/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_SparkActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SparkActionExtension.twiki b/docs/src/site/twiki/DG_SparkActionExtension.twiki
deleted file mode 100644
index 5a56cca..0000000
--- a/docs/src/site/twiki/DG_SparkActionExtension.twiki
+++ /dev/null
@@ -1,436 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Spark Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Spark Action
-
-The `spark` action runs a Spark job.
-
-The workflow job will wait until the Spark job completes before
-continuing to the next action.
-
-To run the Spark job, you have to configure the `spark` action with
-the `resource-manager`, `name-node`, Spark `master` elements as
-well as the necessary elements, arguments and configuration.
-
-Spark options can be specified in an element called `spark-opts`.
-
-A `spark` action can be configured to create or delete HDFS directories
-before starting the Spark job.
-
-Oozie EL expressions can be used in the inline configuration. Property
-values specified in the `configuration` element override values specified
-in the `job-xml` file.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <spark xmlns="uri:oozie:spark-action:1.0">
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[SPARK SETTINGS FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <master>[SPARK MASTER URL]</master>
- <mode>[SPARK MODE]</mode>
- <name>[SPARK JOB NAME]</name>
- <class>[SPARK MAIN CLASS]</class>
- <jar>[SPARK DEPENDENCIES JAR / PYTHON FILE]</jar>
- <spark-opts>[SPARK-OPTIONS]</spark-opts>
- <arg>[ARG-VALUE]</arg>
- ...
- <arg>[ARG-VALUE]</arg>
- ...
- </spark>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-
-The `job-xml` element, if present, specifies a file containing configuration
-for the Spark job. Multiple `job-xml` elements are allowed in order to
-specify multiple `job.xml` files.
-
-The `configuration` element, if present, contains configuration
-properties that are passed to the Spark job.
-
-The `master` element indicates the url of the Spark Master. Ex: `spark://host:port`, `mesos://host:port`, yarn-cluster, yarn-client,
-or local.
-
-The `mode` element if present indicates the mode of spark, where to run spark driver program. Ex: client,cluster. This is typically
-not required because you can specify it as part of `master` (i.e. master`yarn, mode`client is equivalent to master=yarn-client).
-A local `master` always runs in client mode.
-
-Depending on the `master` (and `mode`) entered, the Spark job will run differently as follows:
-
- * local mode: everything runs here in the Launcher Job.
- * yarn-client mode: the driver runs here in the Launcher Job and the executor in Yarn.
- * yarn-cluster mode: the driver and executor run in Yarn.
-
-The `name` element indicates the name of the spark application.
-
-The `class` element if present, indicates the spark's application main class.
-
-The `jar` element indicates a comma separated list of jars or python files.
-
-The `spark-opts` element, if present, contains a list of Spark options that can be passed to Spark. Spark configuration
-options can be passed by specifying '--conf key=value' or other Spark CLI options.
-Values containing whitespaces can be enclosed by double quotes.
-
-Some examples of the `spark-opts` element:
-
- * '--conf key=value'
- * '--conf key1=value1 value2'
- * '--conf key1="value1 value2"'
- * '--conf key1=value1 key2="value2 value3"'
- * '--conf key=value --verbose --properties-file user.properties'
-
-There are several ways to define properties that will be passed to Spark. They are processed in the following order:
-
- * propagated from `oozie.service.SparkConfigurationService.spark.configurations`
- * read from a localized `spark-defaults.conf` file
- * read from a file defined in `spark-opts` via the `--properties-file`
- * properties defined in `spark-opts` element
-
-(The latter takes precedence over the former.)
-The server propagated properties, the `spark-defaults.conf` and the user-defined properties file are merged together into a
-single properties file as Spark handles only one file in its `--properties-file` option.
-
-The `arg` element if present, contains arguments that can be passed to spark application.
-
-In case some property values are present both in `spark-defaults.conf` and as property key/value pairs generated by Oozie, the user
-configured values from `spark-defaults.conf` are prepended to the ones generated by Oozie, as part of the Spark arguments list.
-
-Following properties to prepend to Spark arguments:
-
- * `spark.executor.extraClassPath`
- * `spark.driver.extraClassPath`
- * `spark.executor.extraJavaOptions`
- * `spark.driver.extraJavaOptions`
-
-All the above elements can be parameterized (templatized) using EL
-expressions.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="myfirstsparkjob">
- <spark xmlns="uri:oozie:spark-action:1.0">
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- </configuration>
- <master>local[*]</master>
- <mode>client</mode>
- <name>Spark Example</name>
- <class>org.apache.spark.examples.mllib.JavaALS</class>
- <jar>/lib/spark-examples_2.10-1.1.0.jar</jar>
- <spark-opts>--executor-memory 20G --num-executors 50
- --conf spark.executor.extraJavaOptions="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"</spark-opts>
- <arg>inputpath=hdfs://localhost/input/file.txt</arg>
- <arg>value=2</arg>
- </spark>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-### Spark Action Logging
-
-Spark action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Spark.
-
-From Oozie web-console, from the Spark action pop up using the 'Console URL' link, it is possible
-to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
-
-### Spark on YARN
-
-To ensure that your Spark job shows up in the Spark History Server, make sure to specify these three Spark configuration properties
-either in `spark-opts` with `--conf` or from `oozie.service.SparkConfigurationService.spark.configurations` in oozie-site.xml.
-
-1. spark.yarn.historyServer.address=SPH-HOST:18088
-
-2. spark.eventLog.dir=`hdfs://NN:8020/user/spark/applicationHistory`
-
-3. spark.eventLog.enabled=true
-
-### PySpark with Spark Action
-
-To submit PySpark scripts with Spark Action, pyspark dependencies must be available in sharelib or in workflow's lib/ directory.
-For more information, please refer to [installation document.](AG_Install.html#Oozie_Share_Lib)
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ....
- <action name="myfirstpysparkjob">
- <spark xmlns="uri:oozie:spark-action:1.0">
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- </configuration>
- <master>yarn-cluster</master>
- <name>Spark Example</name>
- <jar>pi.py</jar>
- <spark-opts>--executor-memory 20G --num-executors 50
- --conf spark.executor.extraJavaOptions="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"</spark-opts>
- <arg>100</arg>
- </spark>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `jar` element indicates python file. Refer to the file by it's localized name, because only local files are allowed
-in PySpark. The py file should be in the lib/ folder next to the workflow.xml or added using the `file` element so that
-it's localized to the working directory with just its name.
-
-### Using Symlink in \<jar\>
-
-A symlink must be specified using [file](WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job) element. Then, you can use
-the symlink name in `jar` element.
-
-**Example:**
-
-Specifying relative path for symlink:
-
-Make sure that the file is within the application directory i.e. `oozie.wf.application.path` .
-
-```
- <spark xmlns="uri:oozie:spark-action:1.0">
- ...
- <jar>py-spark-example-symlink.py</jar>
- ...
- ...
- <file>py-spark.py#py-spark-example-symlink.py</file>
- ...
- </spark>
-```
-
-Specifying full path for symlink:
-
-```
- <spark xmlns="uri:oozie:spark-action:1.0">
- ...
- <jar>spark-example-symlink.jar</jar>
- ...
- ...
- <file>hdfs://localhost:8020/user/testjars/all-oozie-examples.jar#spark-example-symlink.jar</file>
- ...
- </spark>
-```
-
-
-
-## Appendix, Spark XML-Schema
-
-### AE.A Appendix A, Spark XML-Schema
-
-#### Spark Action Schema Version 1.0
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:spark="uri:oozie:spark-action:1.0" elementFormDefault="qualified"
- targetNamespace="uri:oozie:spark-action:1.0">
-.
- <xs:include schemaLocation="oozie-common-1.0.xsd"/>
-.
- <xs:element name="spark" type="spark:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:choice>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="launcher" type="spark:LAUNCHER" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Spark Action Schema Version 0.2
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:spark="uri:oozie:spark-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:spark-action:0.2">
-
- <xs:element name="spark" type="spark:ACTION"/>
-
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="spark:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="spark:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
-</xs:schema>
-```
-
-#### Spark Action Schema Version 0.1
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:spark="uri:oozie:spark-action:0.1" elementFormDefault="qualified"
- targetNamespace="uri:oozie:spark-action:0.1">
-
- <xs:element name="spark" type="spark:ACTION"/>
-
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="spark:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="spark:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="master" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="mode" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="class" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="jar" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="spark-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="spark:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="spark:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
-</xs:schema>
-```
-[::Go back to Oozie Documentation Index::](index.html)
-
-
-
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_SqoopActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SqoopActionExtension.twiki b/docs/src/site/twiki/DG_SqoopActionExtension.twiki
deleted file mode 100644
index b186c5a..0000000
--- a/docs/src/site/twiki/DG_SqoopActionExtension.twiki
+++ /dev/null
@@ -1,348 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Sqoop Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Sqoop Action
-
-**IMPORTANT:** The Sqoop action requires Apache Hadoop 1.x or 2.x.
-
-The `sqoop` action runs a Sqoop job.
-
-The workflow job will wait until the Sqoop job completes before
-continuing to the next action.
-
-To run the Sqoop job, you have to configure the `sqoop` action with the `resource-manager`, `name-node` and Sqoop `command`
-or `arg` elements as well as configuration.
-
-A `sqoop` action can be configured to create or delete HDFS directories
-before starting the Sqoop job.
-
-Sqoop configuration can be specified with a file, using the `job-xml`
-element, and inline, using the `configuration` elements.
-
-Oozie EL expressions can be used in the inline configuration. Property
-values specified in the `configuration` element override values specified
-in the `job-xml` file.
-
-Note that YARN `yarn.resourcemanager.address` / `resource-manager` and HDFS `fs.default.name` / `name-node` properties must not
-be present in the inline configuration.
-
-As with Hadoop `map-reduce` jobs, it is possible to add files and
-archives in order to make them available to the Sqoop job. Refer to the
-[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
-section for more information about this feature.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <sqoop xmlns="uri:oozie:sqoop-action:1.0">
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <command>[SQOOP-COMMAND]</command>
- <arg>[SQOOP-ARGUMENT]</arg>
- ...
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- </sqoop>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-
-The `job-xml` element, if present, specifies a file containing configuration
-for the Sqoop job. As of schema 0.3, multiple `job-xml` elements are allowed in order to
-specify multiple `job.xml` files.
-
-The `configuration` element, if present, contains configuration
-properties that are passed to the Sqoop job.
-
-**Sqoop command**
-
-The Sqoop command can be specified either using the `command` element or multiple `arg`
-elements.
-
-When using the `command` element, Oozie will split the command on every space
-into multiple arguments.
-
-When using the `arg` elements, Oozie will pass each argument value as an argument to Sqoop.
-
-The `arg` variant should be used when there are spaces within a single argument.
-
-Consult the Sqoop documentation for a complete list of valid Sqoop commands.
-
-All the above elements can be parameterized (templatized) using EL
-expressions.
-
-**Examples:**
-
-Using the `command` element:
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="myfirsthivejob">
- <sqoop xmlns="uri:oozie:sqoop-action:1.0">
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- </configuration>
- <command>import --connect jdbc:hsqldb:file:db.hsqldb --table TT --target-dir hdfs://localhost:8020/user/tucu/foo -m 1</command>
- </sqoop>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-The same Sqoop action using `arg` elements:
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="myfirstsqoopjob">
- <sqoop xmlns="uri:oozie:sqoop-action:1.0">
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- </configuration>
- <arg>import</arg>
- <arg>--connect</arg>
- <arg>jdbc:hsqldb:file:db.hsqldb</arg>
- <arg>--table</arg>
- <arg>TT</arg>
- <arg>--target-dir</arg>
- <arg>hdfs://localhost:8020/user/tucu/foo</arg>
- <arg>-m</arg>
- <arg>1</arg>
- </sqoop>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-NOTE: The `arg` elements syntax, while more verbose, allows to have spaces in a single argument, something useful when
-using free from queries.
-
-### Sqoop Action Counters
-
-The counters of the map-reduce job run by the Sqoop action are available to be used in the workflow via the
-[hadoop:counters() EL function](WorkflowFunctionalSpec.html#HadoopCountersEL).
-
-If the Sqoop action run an import all command, the `hadoop:counters()` EL will return the aggregated counters
-of all map-reduce jobs run by the Sqoop import all command.
-
-### Sqoop Action Logging
-
-Sqoop action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Sqoop.
-
-From Oozie web-console, from the Sqoop action pop up using the 'Console URL' link, it is possible
-to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
-
-The logging level of the Sqoop action can set in the Sqoop action configuration using the
-property `oozie.sqoop.log.level`. The default value is `INFO`.
-
-## Appendix, Sqoop XML-Schema
-
-### AE.A Appendix A, Sqoop XML-Schema
-
-#### Sqoop Action Schema Version 1.0
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:sqoop="uri:oozie:sqoop-action:1.0"
- elementFormDefault="qualified"
- targetNamespace="uri:oozie:sqoop-action:1.0">
-.
- <xs:include schemaLocation="oozie-common-1.0.xsd"/>
-.
- <xs:element name="sqoop" type="sqoop:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:choice>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="launcher" type="sqoop:LAUNCHER" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:choice>
- <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/>
- </xs:choice>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Sqoop Action Schema Version 0.3
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:sqoop="uri:oozie:sqoop-action:0.3" elementFormDefault="qualified"
- targetNamespace="uri:oozie:sqoop-action:0.3">
-
- <xs:element name="sqoop" type="sqoop:ACTION"/>
-
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:choice>
- <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/>
- </xs:choice>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="sqoop:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="sqoop:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
-</xs:schema>
-```
-
-#### Sqoop Action Schema Version 0.2
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:sqoop="uri:oozie:sqoop-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:sqoop-action:0.2">
-
- <xs:element name="sqoop" type="sqoop:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="sqoop:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="configuration" type="sqoop:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:choice>
- <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="1" maxOccurs="unbounded"/>
- </xs:choice>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="sqoop:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="sqoop:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_SshActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SshActionExtension.twiki b/docs/src/site/twiki/DG_SshActionExtension.twiki
deleted file mode 100644
index e53e1c3..0000000
--- a/docs/src/site/twiki/DG_SshActionExtension.twiki
+++ /dev/null
@@ -1,161 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Ssh Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Ssh Action
-
-The `ssh` action starts a shell command on a remote machine as a remote secure shell in background. The workflow job
-will wait until the remote shell command completes before continuing to the next action.
-
-The shell command must be present in the remote machine and it must be available for execution via the command path.
-
-The shell command is executed in the home directory of the specified user in the remote host.
-
-The output (STDOUT) of the ssh job can be made available to the workflow job after the ssh job ends. This information
-could be used from within decision nodes. If the output of the ssh job is made available to the workflow job the shell
-command must follow the following requirements:
-
- * The format of the output must be a valid Java Properties file.
- * The size of the output must not exceed 2KB.
-
-Note: Ssh Action will fail if any output is written to standard error / output upon login (e.g. .bashrc of the remote
-user contains ls -a).
-
-Note: Ssh Action will fail if oozie fails to ssh connect to host for action status check
-(e.g., the host is under heavy load, or network is bad) after a configurable number (3 by default) of retries.
-The first retry will wait a configurable period of time ( 3 seconds by default) before check.
-The following retries will wait 2 times of previous wait time.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <ssh xmlns="uri:oozie:ssh-action:0.1">
- <host>[USER]@[HOST]</host>
- <command>[SHELL]</command>
- <args>[ARGUMENTS]</args>
- ...
- <capture-output/>
- </ssh>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `host` indicates the user and host where the shell will be executed.
-
-**IMPORTANT:** The `oozie.action.ssh.allow.user.at.host` property, in the `oozie-site.xml` configuration, indicates if
-an alternate user than the one submitting the job can be used for the ssh invocation. By default this property is set
-to `true`.
-
-The `command` element indicates the shell command to execute.
-
-The `args` element, if present, contains parameters to be passed to the shell command. If more than one `args` element
-is present they are concatenated in order. When an `args` element contains a space, even when quoted, it will be considered as
-separate arguments (i.e. "Hello World" becomes "Hello" and "World"). Starting with ssh schema 0.2, you can use the `arg` element
-(note that this is different than the `args` element) to specify arguments that have a space in them (i.e. "Hello World" is
-preserved as "Hello World"). You can use either `args` elements, `arg` elements, or neither; but not both in the same action.
-
-If the `capture-output` element is present, it indicates Oozie to capture output of the STDOUT of the ssh command
-execution. The ssh command output must be in Java Properties file format and it must not exceed 2KB. From within the
-workflow definition, the output of an ssh action node is accessible via the =String action:output(String node,
-String key)= function (Refer to section '4.2.6 Action EL Functions').
-
-The configuration of the `ssh` action can be parameterized (templatized) using EL expressions.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="myssjob">
- <ssh xmlns="uri:oozie:ssh-action:0.1">
- <host>foo@bar.com<host>
- <command>uploaddata</command>
- <args>jdbc:derby://bar.com:1527/myDB</args>
- <args>hdfs://foobar.com:8020/usr/tucu/myData</args>
- </ssh>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-In the above example, the `uploaddata` shell command is executed with two arguments, `jdbc:derby://foo.com:1527/myDB`
-and `hdfs://foobar.com:8020/usr/tucu/myData`.
-
-The `uploaddata` shell must be available in the remote host and available in the command path.
-
-The output of the command will be ignored because the `capture-output` element is not present.
-
-## Appendix, Ssh XML-Schema
-
-### AE.A Appendix A, Ssh XML-Schema
-
-#### Ssh Action Schema Version 0.2
-
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:ssh="uri:oozie:ssh-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:ssh-action:0.2">
-.
- <xs:element name="ssh" type="ssh:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="host" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:choice>
- <xs:element name="args" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:choice>
- <xs:element name="capture-output" type="ssh:FLAG" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="FLAG"/>
-.
-</xs:schema>
-```
-
-#### Ssh Action Schema Version 0.1
-
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:ssh="uri:oozie:ssh-action:0.1" elementFormDefault="qualified"
- targetNamespace="uri:oozie:ssh-action:0.1">
-.
- <xs:element name="ssh" type="ssh:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="host" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="command" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="args" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="capture-output" type="ssh:FLAG" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="FLAG"/>
-.
-</xs:schema>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_WorkflowReRun.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_WorkflowReRun.twiki b/docs/src/site/twiki/DG_WorkflowReRun.twiki
deleted file mode 100644
index c128681..0000000
--- a/docs/src/site/twiki/DG_WorkflowReRun.twiki
+++ /dev/null
@@ -1,42 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Workflow ReRrun
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-## Configs
-
- * oozie.wf.application.path
- * Only one of following two configurations is mandatory. Both should not be defined at the same time
- * oozie.wf.rerun.skip.nodes
- * oozie.wf.rerun.failnodes
- * Skip nodes are comma separated list of action names. They can be any action nodes including decision node.
- * The valid value of `oozie.wf.rerun.failnodes` is true or false.
- * If secured hadoop version is used, the following two properties needs to be specified as well
- * mapreduce.jobtracker.kerberos.principal
- * dfs.namenode.kerberos.principal.
- * Configurations can be passed as -D param.
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -rerun 14-20090525161321-oozie-joe -Doozie.wf.rerun.skip.nodes=<>
-```
-
-## Pre-Conditions
-
- * Workflow with id wfId should exist.
- * Workflow with id wfId should be in SUCCEEDED/KILLED/FAILED.
- * If specified , nodes in the config oozie.wf.rerun.skip.nodes must be completed successfully.
-
-## ReRun
-
- * Reloads the configs.
- * If no configuration is passed, existing coordinator/workflow configuration will be used. If configuration is passed then, it will be merged with existing workflow configuration. Input configuration will take the precedence.
- * Currently there is no way to remove an existing configuration but only override by passing a different value in the input configuration.
- * Creates a new Workflow Instance with the same wfId.
- * Deletes the actions that are not skipped from the DB and copies data from old Workflow Instance to new one for skipped actions.
- * Action handler will skip the nodes given in the config with the same exit transition as before.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/ENG_MiniOozie.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/ENG_MiniOozie.twiki b/docs/src/site/twiki/ENG_MiniOozie.twiki
deleted file mode 100644
index e793676..0000000
--- a/docs/src/site/twiki/ENG_MiniOozie.twiki
+++ /dev/null
@@ -1,83 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Running MiniOozie Tests
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## System Requirements
-
- * Unix box (tested on Mac OS X and Linux)
- * Java JDK 1.8+
- * Eclipse (tested on 3.5 and 3.6)
- * [Maven 3.0.1+](http://maven.apache.org/)
-
-The Maven command (mvn) must be in the command path.
-
-## Installing Oozie Jars To Maven Cache
-
-Oozie source tree is at Apache SVN or Apache GIT. MiniOozie sample project is under Oozie source tree.
-
-The following command downloads Oozie trunk to local:
-
-
-```
-$ svn co https://svn.apache.org/repos/asf/incubator/oozie/trunk
-```
-
-OR
-
-
-```
-$ git clone git://github.com/apache/oozie.git
-```
-
-To run MiniOozie tests, the required jars like oozie-core, oozie-client, oozie-core-tests need to be
-available in remote maven repositories or local maven repository. The local maven cache for the above
-jars can be created and installed using the command:
-
-
-```
-$ mvn clean install -DskipTests -DtestJarSimple
-```
-
-The following properties should be specified to install correct jars for MiniOozie:
-
- * -DskipTests : ignore executing Oozie unittests
- * -DtestJarSimple= : build only required test classes to oozie-core-tests
-
-MiniOozie is a folder named 'minitest' under Oozie source tree. Two sample tests are included in the project.
-The following command to execute tests under MiniOozie:
-
-
-```
-$ cd minitest
-$ mvn clean test
-```
-
-## Create Tests Using MiniOozie
-
-MiniOozie is a JUnit test class to test Oozie applications such as workflow and coordinator. The test case
-needs to extend from MiniOozieTestCase and does the same as the example class 'WorkflowTest.java' to create Oozie
-workflow application properties and workflow XML. The example file is under Oozie source tree:
-
- * `minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java`
-
-## IDE Setup
-
-Eclipse and IntelliJ can use directly MiniOozie Maven project files. MiniOozie project can be imported to
-Eclipse and IntelliJ as independent project.
-
-The test directories under MiniOozie are:
-
- * `minitest/src/test/java` : as test-source directory
- * `minitest/src/test/resources` : as test-resource directory
-
-
-Also asynchronous actions like FS action can be used / tested using `LocalOozie` / `OozieClient` API.
-Please see `fs-decision.xml` workflow example.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
[21/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
OOZIE-2734 amend [docs] Switch from TWiki to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Project: http://git-wip-us.apache.org/repos/asf/oozie/repo
Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/6a6f2199
Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/6a6f2199
Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/6a6f2199
Branch: refs/heads/master
Commit: 6a6f21996f537ed093106ee78c623c72803105bb
Parents: 4e5b3cb
Author: Andras Piros <an...@cloudera.com>
Authored: Fri Sep 14 16:34:39 2018 +0200
Committer: Andras Piros <an...@cloudera.com>
Committed: Fri Sep 14 16:34:39 2018 +0200
----------------------------------------------------------------------
README.md | 61 +
README.txt | 61 -
.../src/site/markdown/AG_ActionConfiguration.md | 87 +
.../src/site/markdown/AG_HadoopConfiguration.md | 94 +
docs/src/site/markdown/AG_Install.md | 1447 +++++
docs/src/site/markdown/AG_Monitoring.md | 199 +
docs/src/site/markdown/AG_OozieUpgrade.md | 86 +
docs/src/site/markdown/BundleFunctionalSpec.md | 418 ++
.../site/markdown/CoordinatorFunctionalSpec.md | 4890 ++++++++++++++++
.../site/markdown/DG_ActionAuthentication.md | 132 +
docs/src/site/markdown/DG_CommandLineTool.md | 2295 ++++++++
docs/src/site/markdown/DG_CoordinatorRerun.md | 53 +
.../site/markdown/DG_CustomActionExecutor.md | 83 +
.../site/markdown/DG_DistCpActionExtension.md | 220 +
.../site/markdown/DG_EmailActionExtension.md | 136 +
docs/src/site/markdown/DG_Examples.md | 204 +
docs/src/site/markdown/DG_FluentJobAPI.md | 384 ++
.../src/site/markdown/DG_HCatalogIntegration.md | 147 +
.../site/markdown/DG_Hive2ActionExtension.md | 315 +
.../src/site/markdown/DG_HiveActionExtension.md | 476 ++
docs/src/site/markdown/DG_JMSNotifications.md | 236 +
docs/src/site/markdown/DG_Overview.md | 86 +
docs/src/site/markdown/DG_SLAMonitoring.md | 459 ++
.../site/markdown/DG_ShellActionExtension.md | 494 ++
.../site/markdown/DG_SparkActionExtension.md | 436 ++
.../site/markdown/DG_SqoopActionExtension.md | 348 ++
docs/src/site/markdown/DG_SshActionExtension.md | 161 +
docs/src/site/markdown/DG_WorkflowReRun.md | 42 +
docs/src/site/markdown/ENG_MiniOozie.md | 83 +
docs/src/site/markdown/WebServicesAPI.md | 2379 ++++++++
.../src/site/markdown/WorkflowFunctionalSpec.md | 5457 ++++++++++++++++++
.../src/site/twiki/AG_ActionConfiguration.twiki | 87 -
.../src/site/twiki/AG_HadoopConfiguration.twiki | 94 -
docs/src/site/twiki/AG_Install.twiki | 1447 -----
docs/src/site/twiki/AG_Monitoring.twiki | 199 -
docs/src/site/twiki/AG_OozieUpgrade.twiki | 86 -
docs/src/site/twiki/BundleFunctionalSpec.twiki | 418 --
.../site/twiki/CoordinatorFunctionalSpec.twiki | 4890 ----------------
.../site/twiki/DG_ActionAuthentication.twiki | 132 -
docs/src/site/twiki/DG_CommandLineTool.twiki | 2295 --------
docs/src/site/twiki/DG_CoordinatorRerun.twiki | 53 -
.../site/twiki/DG_CustomActionExecutor.twiki | 83 -
.../site/twiki/DG_DistCpActionExtension.twiki | 220 -
.../site/twiki/DG_EmailActionExtension.twiki | 136 -
docs/src/site/twiki/DG_Examples.twiki | 204 -
docs/src/site/twiki/DG_FluentJobAPI.twiki | 384 --
.../src/site/twiki/DG_HCatalogIntegration.twiki | 147 -
.../site/twiki/DG_Hive2ActionExtension.twiki | 315 -
.../src/site/twiki/DG_HiveActionExtension.twiki | 476 --
docs/src/site/twiki/DG_JMSNotifications.twiki | 236 -
docs/src/site/twiki/DG_Overview.twiki | 86 -
docs/src/site/twiki/DG_SLAMonitoring.twiki | 459 --
.../site/twiki/DG_ShellActionExtension.twiki | 494 --
.../site/twiki/DG_SparkActionExtension.twiki | 436 --
.../site/twiki/DG_SqoopActionExtension.twiki | 348 --
docs/src/site/twiki/DG_SshActionExtension.twiki | 161 -
docs/src/site/twiki/DG_WorkflowReRun.twiki | 42 -
docs/src/site/twiki/ENG_MiniOozie.twiki | 83 -
docs/src/site/twiki/WebServicesAPI.twiki | 2379 --------
.../src/site/twiki/WorkflowFunctionalSpec.twiki | 5457 ------------------
release-log.txt | 1 +
61 files changed, 21909 insertions(+), 21908 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..f9c6bdf
--- /dev/null
+++ b/README.md
@@ -0,0 +1,61 @@
+Apache Oozie
+=============
+
+What is Oozie
+--------------
+
+Oozie is an extensible, scalable and reliable system to define, manage, schedule, and execute complex Hadoop workloads via web services. More specifically, this includes:
+
+ * XML-based declarative framework to specify a job or a complex workflow of dependent jobs.
+ * Support different types of job such as Hadoop Map-Reduce, Pipe, Streaming, Pig, Hive and custom java applications.
+ * Workflow scheduling based on frequency and/or data availability.
+ * Monitoring capability, automatic retry and failure handing of jobs.
+ * Extensible and pluggable architecture to allow arbitrary grid programming paradigms.
+ * Authentication, authorization, and capacity-aware load throttling to allow multi-tenant software as a service.
+
+Oozie Overview
+----------
+
+Oozie is a server based Workflow Engine specialized in running workflow jobs with actions that run Hadoop Map/Reduce and Pig jobs.
+
+Oozie is a Java Web-Application that runs in a Java servlet-container.
+
+For the purposes of Oozie, a workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in a control dependency DAG (Directed Acyclic Graph). "control dependency" from one action to another means that the second action can't run until the first action has completed.
+
+Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to JBOSS JBPM jPDL).
+
+Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.
+
+Oozie workflows contain control flow nodes and action nodes.
+
+Control flow nodes define the beginning and the end of a workflow ( start , end and fail nodes) and provide a mechanism to control the workflow execution path ( decision , fork and join nodes).
+
+Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and Oozie sub-workflow. Oozie can be extended to support additional type of actions.
+
+Oozie workflows can be parameterized (using variables like ${inputDir} within the workflow definition). When submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different output directories) several identical workflow jobs can concurrently.
+
+Documentations :
+-----------------
+Oozie web service is bundle with the built-in details documentation.
+
+More inforamtion could be found at:
+http://oozie.apache.org/
+
+Oozie Quick Start:
+http://oozie.apache.org/docs/5.0.0/DG_QuickStart.html
+
+
+Supported Hadoop Versions:
+----------------------------
+
+This version of Oozie was primarily tested against Hadoop 2.4.x and 2.6.x.
+
+
+
+If you have any questions/issues, please send an email to:
+
+user@oozie.apache.org
+
+Subscribe using the link:
+
+http://oozie.apache.org/mail-lists.html
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/README.txt
----------------------------------------------------------------------
diff --git a/README.txt b/README.txt
deleted file mode 100644
index f9c6bdf..0000000
--- a/README.txt
+++ /dev/null
@@ -1,61 +0,0 @@
-Apache Oozie
-=============
-
-What is Oozie
---------------
-
-Oozie is an extensible, scalable and reliable system to define, manage, schedule, and execute complex Hadoop workloads via web services. More specifically, this includes:
-
- * XML-based declarative framework to specify a job or a complex workflow of dependent jobs.
- * Support different types of job such as Hadoop Map-Reduce, Pipe, Streaming, Pig, Hive and custom java applications.
- * Workflow scheduling based on frequency and/or data availability.
- * Monitoring capability, automatic retry and failure handing of jobs.
- * Extensible and pluggable architecture to allow arbitrary grid programming paradigms.
- * Authentication, authorization, and capacity-aware load throttling to allow multi-tenant software as a service.
-
-Oozie Overview
-----------
-
-Oozie is a server based Workflow Engine specialized in running workflow jobs with actions that run Hadoop Map/Reduce and Pig jobs.
-
-Oozie is a Java Web-Application that runs in a Java servlet-container.
-
-For the purposes of Oozie, a workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in a control dependency DAG (Directed Acyclic Graph). "control dependency" from one action to another means that the second action can't run until the first action has completed.
-
-Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to JBOSS JBPM jPDL).
-
-Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.
-
-Oozie workflows contain control flow nodes and action nodes.
-
-Control flow nodes define the beginning and the end of a workflow ( start , end and fail nodes) and provide a mechanism to control the workflow execution path ( decision , fork and join nodes).
-
-Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and Oozie sub-workflow. Oozie can be extended to support additional type of actions.
-
-Oozie workflows can be parameterized (using variables like ${inputDir} within the workflow definition). When submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different output directories) several identical workflow jobs can concurrently.
-
-Documentations :
------------------
-Oozie web service is bundle with the built-in details documentation.
-
-More inforamtion could be found at:
-http://oozie.apache.org/
-
-Oozie Quick Start:
-http://oozie.apache.org/docs/5.0.0/DG_QuickStart.html
-
-
-Supported Hadoop Versions:
-----------------------------
-
-This version of Oozie was primarily tested against Hadoop 2.4.x and 2.6.x.
-
-
-
-If you have any questions/issues, please send an email to:
-
-user@oozie.apache.org
-
-Subscribe using the link:
-
-http://oozie.apache.org/mail-lists.html
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/AG_ActionConfiguration.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/AG_ActionConfiguration.md b/docs/src/site/markdown/AG_ActionConfiguration.md
new file mode 100644
index 0000000..e020d1a
--- /dev/null
+++ b/docs/src/site/markdown/AG_ActionConfiguration.md
@@ -0,0 +1,87 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Action Configuration
+
+Oozie supports providing default configuration for actions of a particular
+action type and default configuration for all actions
+
+## Hadoop Default Configuration Values
+
+Oozie supports action configuration equivalent to the component's `*-site.xml` and `*.properties` files.
+
+The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.action.configurations`
+and its value must follow the pattern _\[AUTHORITY=ACTION-CONF-DIR_,\]*. Where _AUTHORITY_ is the _HOST:PORT_ of
+the Hadoop service (JobTracker/ResourceManager or HDFS). The _ACTION-CONF-DIR_ is the action configuration directory. If the specified directory is a relative path, it will be looked under the Oozie configuration directory. An absolute path can
+also be specified. Oozie will load and process the action configuration files in the following order.
+
+ 1. All files in _default_/*.xml (sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.
+ 1. _default_.xml, if present.
+ 1. All supported files in _actionname_/\*, e.g. _actionname_/\*.xml and _actionname_/*.properties (based on filename extension, sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.
+ 1. _actionname_.xml, if present.
+
+
+For example, for _Hive_ action (which has the _actionname_ defined as _hive_ ), the list of files (under relevant _ACTION-CONF-DIR_ ) processed would be,
+
+ 1. All files in _default_/*.xml, if present
+ 1. _default_.xml, if present.
+ 1. All files in _hive_/\*.xml and _hive_/\*.properties, if present
+ 1. _hive_.xml, if present.
+
+
+Files processed earlier for an action have the lowest precedence and can have the configuration parameters redefined. All files and directories are relative to the _ACTION-CONF-DIR_ directory.
+
+In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard
+will be used as default if there is no action configuration for the requested Hadoop service.
+
+For example, the configuration in the `oozie-site.xml` would look like:
+
+
+```
+...
+ <property>
+ <name>oozie.service.HadoopAccessorService.action.configurations</name>
+ <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value>
+ </property>
+...
+```
+
+The action configuration files use the Hadoop configuration syntax.
+
+By default Oozie does not define any default action configurations.
+
+## Dependency deduplication
+
+Using Oozie with Hadoop 3 may require to have dependency file names distinguishable,
+ which means having two files on sharelib and in your app's dependencies with identical names, leads to job submission failure.
+To avoid this you can enable the deduplicator by setting oozie.action.dependency.deduplicate=true in oozie-site.xml
+(false, by default).
+Dependencies which are closer to your application has higher priority: action jar > user workflow libs > action libs > system lib,
+where dependency with greater prio is used.
+
+Real world example:
+You have an application workflow which is uploaded to HDFS in /apps/app directory. You have your app.jar and dependency jars.
+You also define a spark action in your workflow and set use system libs; the HDFS tree is similar to this:
+
+```
+ + /apps/app/
+ - app.jar
+ - workflow.xml
+ + libs
+ - app.jar
+ - jackson-annotations-1.0.jar
+ + share/lib/
+ + spark
+ - app.jar
+ - jackson-annotations-1.0.jar
+ + oozie
+ - jackson-annotations-1.0.jar
+```
+The deduplicator code will create the following list of files:
+`/apps/app/app.jar,/apps/app/libs/jackson-annotations-1.0.jar`
+And no other files will be passed at job submission.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/AG_HadoopConfiguration.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/AG_HadoopConfiguration.md b/docs/src/site/markdown/AG_HadoopConfiguration.md
new file mode 100644
index 0000000..ab71d7c
--- /dev/null
+++ b/docs/src/site/markdown/AG_HadoopConfiguration.md
@@ -0,0 +1,94 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Hadoop Configuration
+
+## Hadoop Services Whitelisting
+
+Oozie supports whitelisting Hadoop services (JobTracker, HDFS), via 2 configuration properties:
+
+
+```
+...
+ <property>
+ <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
+ <value> </value>
+ <description>
+ Whitelisted job tracker for Oozie service.
+ </description>
+ </property>
+ <property>
+ <name>oozie.service.HadoopAccessorService.nameNode.whitelist</name>
+ <value> </value>
+ <description>
+ Whitelisted job tracker for Oozie service.
+ </description>
+ </property>
+...
+```
+
+The value must follow the pattern `[AUTHORITY,...]`. Where `AUTHORITY` is the `HOST:PORT` of
+the Hadoop service (JobTracker, HDFS).
+
+If the value is empty any HOST:PORT is accepted. Empty is the default value.
+
+## Hadoop Default Configuration Values
+
+Oozie supports Hadoop configuration equivalent to the Hadoop `*-site.xml` files.
+
+The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.hadoop.configurations`
+and its value must follow the pattern `[<AUTHORITY>=<HADOOP_CONF_DIR>,]*`. Where `<AUTHORITY>` is the `HOST:PORT` of
+the Hadoop service (JobTracker, HDFS). The `<HADOOP_CONF_DIR>` is a Hadoop configuration directory. If the specified
+ directory is a relative path, it will be looked under the Oozie configuration directory. And absolute path can
+ also be specified. Oozie will load the Hadoop `*-site.xml` files in the following order: core-site.xml, hdfs-site.xml,
+ mapred-site.xml, yarn-site.xml, hadoop-site.xml, ssl-client.xml.
+
+In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard
+will be used as default if there is no configuration for the requested Hadoop service.
+
+For example, the configuration in the `oozie-site.xml` would look like:
+
+
+```
+...
+ <property>
+ <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
+ <value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value>
+ </property>
+...
+```
+
+The Hadoop configuration files use the Hadoop configuration syntax.
+
+By default Oozie defines `*=hadoop-conf` and the default values of the `hadoop-site.xml` file are:
+
+
+```
+<configuration>
+ <property>
+ <name>mapreduce.jobtracker.kerberos.principal</name>
+ <value>mapred/_HOST@LOCALREALM</value>
+ </property>
+ <property>
+ <name>yarn.resourcemanager.principal</name>
+ <value>yarn/_HOST@LOCALREALM</value>
+ </property>
+ <property>
+ <name>dfs.namenode.kerberos.principal</name>
+ <value>hdfs/_HOST@LOCALREALM</value>
+ </property>
+ <property>
+ <name>mapreduce.framework.name</name>
+ <value>yarn</value>
+ </property>
+</configuration>
+```
+
+## Limitations
+
+All actions in a workflow application must interact with the same Hadoop JobTracker and NameNode.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/AG_Install.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/AG_Install.md b/docs/src/site/markdown/AG_Install.md
new file mode 100644
index 0000000..270b98f
--- /dev/null
+++ b/docs/src/site/markdown/AG_Install.md
@@ -0,0 +1,1447 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Installation and Configuration
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Basic Setup
+
+Follow the instructions at [Oozie Quick Start](DG_QuickStart.html).
+
+## Environment Setup
+
+**IMPORTANT:** Oozie ignores any set value for `OOZIE_HOME`, Oozie computes its home automatically.
+
+When running Oozie with its embedded Jetty server, the `conf/oozie-env.sh` file can be
+used to configure the following environment variables used by Oozie:
+
+**JETTY_OPTS** : settings for the Embedded Jetty that runs Oozie. Java System properties
+for Oozie should be specified in this variable. No default value.
+
+**OOZIE_CONFIG_FILE** : Oozie configuration file to load from Oozie configuration directory.
+Default value `oozie-site.xml`.
+
+**OOZIE_LOGS** : Oozie logs directory. Default value `logs/` directory in the Oozie installation
+directory.
+
+**OOZIE_LOG4J_FILE** : Oozie Log4J configuration file to load from Oozie configuration directory.
+Default value `oozie-log4j.properties`.
+
+**OOZIE_LOG4J_RELOAD** : Reload interval of the Log4J configuration file, in seconds.
+Default value `10`
+
+**OOZIE_CHECK_OWNER** : If set to `true`, Oozie setup/start/run/stop scripts will check that the
+owner of the Oozie installation directory matches the user invoking the script. The default
+value is undefined and interpreted as a `false`.
+
+**OOZIE_INSTANCE_ID** : The instance id of the Oozie server. When using HA, each server instance should have a unique instance id.
+Default value `${OOZIE_HTTP_HOSTNAME}`
+
+## Oozie Server Setup
+
+The `oozie-setup.sh` script prepares the embedded Jetty server to run Oozie.
+
+The `oozie-setup.sh` script options are:
+
+
+```
+Usage : oozie-setup.sh <Command and OPTIONS>
+ sharelib create -fs FS_URI [-locallib SHARED_LIBRARY] [-extralib EXTRA_SHARED_LIBRARY] [-concurrency CONCURRENCY]
+ (create sharelib for oozie,
+ FS_URI is the fs.default.name
+ for hdfs uri; SHARED_LIBRARY, path to the
+ Oozie sharelib to install, it can be a tarball
+ or an expanded version of it. If omitted,
+ the Oozie sharelib tarball from the Oozie
+ installation directory will be used.
+ EXTRA_SHARED_LIBRARY represents extra sharelib resources.
+ This option requires a pair of sharelibname
+ and comma-separated list of pathnames in the following format:
+ sharelib-name=path-name-1,path-name-2
+ In case of more than one sharelib, this option can be specified
+ multiple times.
+ CONCURRENCY is a number of threads to be used
+ for copy operations.
+ By default 1 thread will be used)
+ (action fails if sharelib is already installed
+ in HDFS)
+ sharelib upgrade -fs FS_URI [-locallib SHARED_LIBRARY] ([deprecated][use create command to create new version]
+ upgrade existing sharelib, fails if there
+ is no existing sharelib installed in HDFS)
+ db create|upgrade|postupgrade -run [-sqlfile <FILE>] (create, upgrade or postupgrade oozie db with an
+ optional sql File)
+ export <file> exports the oozie database to the specified
+ file in zip format
+ import <file> imports the oozie database from the zip file
+ created by export
+ (without options prints this usage information)
+```
+
+If a directory `libext/` is present in Oozie installation directory, the `oozie-setup.sh` script will
+include all JARs in Jetty's `webapp/WEB_INF/lib/` directory.
+
+If the ExtJS ZIP file is present in the `libext/` directory, it will be added to the Jetty's `webapp/` directory as well.
+The ExtJS library file name be `ext-2.2.zip`.
+
+### Setting Up Oozie with an Alternate Tomcat
+
+Use the `addtowar.sh` script to prepare the Oozie server only if Oozie will run with a different
+servlet container than the embedded Jetty provided with the distribution.
+
+The `addtowar.sh` script adds Hadoop JARs, JDBC JARs and the ExtJS library to the Oozie WAR file.
+
+The `addtowar.sh` script options are:
+
+
+```
+ Usage : addtowar <OPTIONS>
+ Options: -inputwar INPUT_OOZIE_WAR
+ -outputwar OUTPUT_OOZIE_WAR
+ [-hadoop HADOOP_VERSION HADOOP_PATH]
+ [-extjs EXTJS_PATH]
+ [-jars JARS_PATH] (multiple JAR path separated by ':')
+ [-secureWeb WEB_XML_PATH] (path to secure web.xml)
+```
+
+The original `oozie.war` file is in the Oozie server installation directory.
+
+After the Hadoop JARs and the ExtJS library has been added to the `oozie.war` file Oozie is ready to run.
+
+Delete any previous deployment of the `oozie.war` from the servlet container (if using Tomcat, delete
+`oozie.war` and `oozie` directory from Tomcat's `webapps/` directory)
+
+Deploy the prepared `oozie.war` file (the one that contains the Hadoop JARs and the ExtJS library) in the
+servlet container (if using Tomcat, copy the prepared `oozie.war` file to Tomcat's `webapps/` directory).
+
+**IMPORTANT:** Only one Oozie instance can be deployed per Tomcat instance.
+
+## Database Configuration
+
+Oozie works with HSQL, Derby, MySQL, Oracle, PostgreSQL or SQL Server databases.
+
+By default, Oozie is configured to use Embedded Derby.
+
+Oozie bundles the JDBC drivers for HSQL, Embedded Derby and PostgreSQL.
+
+HSQL is normally used for test cases as it is an in-memory database and all data is lost every time Oozie is stopped.
+
+If using Derby, MySQL, Oracle, PostgreSQL, or SQL Server, the Oozie database schema must be created using the `ooziedb.sh` command
+line tool.
+
+If using MySQL, Oracle, or SQL Server, the corresponding JDBC driver JAR file must be copied to Oozie's `libext/` directory and
+it must be added to Oozie WAR file using the `bin/addtowar.sh` or the `oozie-setup.sh` scripts using the `-jars` option.
+
+**IMPORTANT:** It is recommended to set the database's timezone to GMT (consult your database's documentation on how to do this).
+Databases don't handle Daylight Saving Time shifts correctly, and may cause problems if you run any Coordinators with actions
+scheduled to materialize during the 1 hour period where we "fall back". For Derby, you can add '-Duser.timezone=GMT'
+to `JETTY_OPTS` in oozie-env.sh to set this. Alternatively, if using MySQL, you can have Oozie use GMT with MySQL without
+setting MySQL's timezone to GMT by adding 'useLegacyDatetimeCode=false&serverTimezone=GMT' arguments to the JDBC
+URL, `oozie.service.JPAService.jdbc.url`. Be advised that changing the timezone on an existing Oozie database while Coordinators
+are already running may cause Coordinators to shift by the offset of their timezone from GMT once after making this change.
+
+The SQL database used by Oozie is configured using the following configuration properties (default values shown):
+
+
+```
+ oozie.db.schema.name=oozie
+ oozie.service.JPAService.create.db.schema=false
+ oozie.service.JPAService.validate.db.connection=false
+ oozie.service.JPAService.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
+ oozie.service.JPAService.jdbc.url=jdbc:derby:${oozie.data.dir}/${oozie.db.schema.name}-db;create=true
+ oozie.service.JPAService.jdbc.username=sa
+ oozie.service.JPAService.jdbc.password=
+ oozie.service.JPAService.pool.max.active.conn=10
+```
+
+**NOTE:** If the `oozie.db.schema.create` property is set to `true` (default value is `false`) the Oozie tables
+will be created automatically without having to use the `ooziedb` command line tool. Setting this property to
+ `true` it is recommended only for development.
+
+**NOTE:** If the `oozie.db.schema.create` property is set to true, the `oozie.service.JPAService.validate.db.connection`
+property value is ignored and Oozie handles it as set to `false`.
+
+Once `oozie-site.xml` has been configured with the database configuration execute the `ooziedb.sh` command line tool to
+create the database:
+
+
+```
+$ bin/ooziedb.sh create -sqlfile oozie.sql -run
+
+Validate DB Connection.
+DONE
+Check DB schema does not exist
+DONE
+Check OOZIE_SYS table does not exist
+DONE
+Create SQL schema
+DONE
+DONE
+Create OOZIE_SYS table
+DONE
+
+Oozie DB has been created for Oozie version '3.2.0'
+
+The SQL commands have been written to: oozie.sql
+
+$
+```
+
+NOTE: If using MySQL, Oracle, or SQL Server, copy the corresponding JDBC driver JAR file to the `libext/` directory before running
+the `ooziedb.sh` command line tool.
+
+NOTE: If instead using the '-run' option, the `-sqlfile <FILE>` option is used, then all the
+database changes will be written to the specified file and the database won't be modified.
+
+If using HSQL there is no need to use the `ooziedb` command line tool as HSQL is an in-memory database. Use the
+following configuration properties in the oozie-site.xml:
+
+
+```
+ oozie.db.schema.name=oozie
+ oozie.service.JPAService.create.db.schema=true
+ oozie.service.JPAService.validate.db.connection=false
+ oozie.service.JPAService.jdbc.driver=org.hsqldb.jdbcDriver
+ oozie.service.JPAService.jdbc.url=jdbc:hsqldb:mem:${oozie.db.schema.name}
+ oozie.service.JPAService.jdbc.username=sa
+ oozie.service.JPAService.jdbc.password=
+ oozie.service.JPAService.pool.max.active.conn=10
+```
+
+If you are interested in fine tuning how Oozie can retry database operations on failing database connectivity or errors, you can
+set following properties to other values. Here are the default ones:
+
+
+```
+ oozie.service.JPAService.retry.initial-wait-time.ms=100
+ oozie.service.JPAService.retry.maximum-wait-time.ms=30000
+ oozie.service.JPAService.retry.max-retries=10
+```
+
+If you set either `oozie.service.JPAService.retry.max-retries` or `oozie.service.JPAService.retry.maximum-wait-time.ms` to `0`,
+no retry attempts will be made on any database connectivity issues. Exact settings for these properties depend also on how much load
+is on Oozie regarding workflow and coordinator jobs.
+
+The database operation retry functionality kicks in when there is a `javax.persistence.PersistenceException` those root cause is not
+part of the normal everyday operation - filtered against a blacklist consisting of descendants like `NoSuchResultException`,
+`NonUniqueResultException`, and the like. This way Oozie won't retry database operations on errors that are more related to the
+current query, or otherwise part of the everyday life. This way it's ensured that this blacklist is database agnostic.
+
+It has been tested with a MySQL / failing every minute 10 seconds / an Oozie coordinator job of an Oozie workflow consisting of four
+workflow actions (some of them are asynchronous). On this setup Oozie was recovering after each and every database outage.
+
+To set up such a failing MySQL scenario following has to be performed:
+
+ * Set `oozie.service.JPAService.connection.data.source` to `org.apache.oozie.util.db.BasicDataSourceWrapper`
+ within `oozie-site.xml`
+ * Set `oozie.service.JPAService.jdbc.driver` to `org.apache.oozie.util.db.FailingMySQLDriverWrapper` within `oozie-site.xml`
+ * Restart Oozie server
+ * Submit / start some workflows, coordinators etc.
+ * See how Oozie is retrying on injected database errors by looking at the Oozie server logs, grepping `JPAException` instances
+ with following message prefix: `Deliberately failing to prepare statement.`
+
+## Database Migration
+
+Oozie provides an easy way to switch between databases without losing any data. Oozie servers should be stopped during the
+database migration process.
+The export of the database can be done using the following command:
+
+```
+$ bin/oozie-setup.sh export /tmp/oozie_db.zip
+1 rows exported from OOZIE_SYS
+50 rows exported from WF_JOBS
+340 rows exported from WF_ACTIONS
+10 rows exported from COORD_JOBS
+70 rows exported from COORD_ACTIONS
+0 rows exported from BUNDLE_JOBS
+0 rows exported from BUNDLE_ACTIONS
+0 rows exported from SLA_REGISTRATION
+0 rows exported from SLA_SUMMARY
+```
+
+The database configuration is read from `oozie-site.xml`. After updating the configuration to point to the new database,
+the tables have to be created with ooziedb.sh in the [Database configuration](AG_Install.html#Database_Configuration)
+section above.
+Once the tables are created, they can be filled with data using the following command:
+
+
+```
+$ bin/oozie-setup.sh import /tmp/oozie_db.zip
+Loading to Oozie database version 3
+50 rows imported to WF_JOBS
+340 rows imported to WF_ACTIONS
+10 rows imported to COORD_JOBS
+70 rows imported to COORD_ACTIONS
+0 rows imported to BUNDLE_JOBS
+0 rows imported to BUNDLE_ACTIONS
+0 rows imported to SLA_REGISTRATION
+0 rows imported to SLA_SUMMARY
+```
+
+NOTE: The database version of the zip must match the version of the Oozie database it's imported to.
+
+After starting the Oozie server, the history and the currently running workflows should be available.
+
+**IMPORTANT:** The tool was primarily developed to make the migration from embedded databases (e.g. Derby) to standalone databases
+ (e.g. MySQL, PosgreSQL, Oracle, MS SQL Server), though it will work between any supported databases.
+It is **not** optimized to handle databases over 1 Gb. If the database size is larger, it should be purged before migration.
+
+## Oozie Configuration
+
+By default, Oozie configuration is read from Oozie's `conf/` directory
+
+The Oozie configuration is distributed in 3 different files:
+
+ * `oozie-site.xml` : Oozie server configuration
+ * `oozie-log4j.properties` : Oozie logging configuration
+ * `adminusers.txt` : Oozie admin users list
+
+### Oozie Configuration Properties
+
+All Oozie configuration properties and their default values are defined in the `oozie-default.xml` file.
+
+Oozie resolves configuration property values in the following order:
+
+ * If a Java System property is defined, it uses its value
+ * Else, if the Oozie configuration file (`oozie-site.xml`) contains the property, it uses its value
+ * Else, it uses the default value documented in the `oozie-default.xml` file
+
+**NOTE:** The `oozie-default.xml` file found in Oozie's `conf/` directory is not used by Oozie, it is there
+for reference purposes only.
+
+### Precedence of Configuration Properties
+
+For compatibility reasons across Hadoop / Oozie versions, some configuration properties can be defined using multiple keys
+in the launcher configuration. Beginning with Oozie 5.0.0, some of them can be overridden, some others will be prepended to default
+configuration values.
+
+#### Overriding Configuration Values
+
+Overriding happens for following configuration entries with `oozie.launcher` prefix, by switching `oozie.launcher.override`
+(on by default).
+
+For those, following is the general approach:
+
+ * check whether a YARN compatible entry is present. If yes, use it to override default value
+ * check whether a MapReduce v2 compatible entry is present. If yes, use it to override default value
+ * check whether a MapReduce v1 compatible entry is present. If yes, use it to override default value
+ * use default value
+
+Such properties are (legend: YARN / MapReduce v2 / MapReduce v1):
+
+ * max attempts of the MapReduce Application Master:
+ * N / A
+ * `mapreduce.map.maxattempts`
+ * `mapred.map.max.attempts`
+ * memory amount in MB of the MapReduce Application Master:
+ * `yarn.app.mapreduce.am.resource.mb`
+ * `mapreduce.map.memory.mb`
+ * `mapred.job.map.memory.mb`
+ * CPU vcore count of the MapReduce Application Master:
+ * `yarn.app.mapreduce.am.resource.cpu-vcores`
+ * `mapreduce.map.cpu.vcores`
+ * N / A
+ * logging level of the MapReduce Application Master:
+ * N / A
+ * `mapreduce.map.log.level`
+ * `mapred.map.child.log.level`
+ * MapReduce Application Master JVM options:
+ * `yarn.app.mapreduce.am.command-opts`
+ * `mapreduce.map.java.opts`
+ * `mapred.child.java.opts`
+ * MapReduce Application Master environment variable settings:
+ * `yarn.app.mapreduce.am.env`
+ * `mapreduce.map.env`
+ * `mapred.child.env`
+ * MapReduce Application Master job priority:
+ * N / A
+ * `mapreduce.job.priority`
+ * `mapred.job.priority`
+ * MapReduce Application Master job queue name:
+ * N / A
+ * `mapreduce.job.queuename`
+ * `mapred.job.queue.name`
+ * MapReduce View ACL settings:
+ * N / A
+ * `mapreduce.job.acl-view-job`
+ * N / A
+ * MapReduce Modify ACL settings:
+ * N / A
+ * `mapreduce.job.acl-modify-job`
+ * N / A
+
+This list can be extended or modified by adding new configuration entries or updating existing values
+beginning with `oozie.launcher.override.` within `oozie-site.xml`. Examples can be found in `oozie-default.xml`.
+
+#### Prepending Configuration Values
+
+Prepending happens for following configuration entries with `oozie.launcher` prefix, by switching `oozie.launcher.prepend`
+(on by default).
+
+For those, following is the general approach:
+
+ * check whether a YARN compatible entry is present. If yes, use it to prepend to default value
+ * use default value
+
+Such properties are (legend: YARN only):
+
+ * MapReduce Application Master JVM options: `yarn.app.mapreduce.am.admin-command-opts`
+ * MapReduce Application Master environment settings: `yarn.app.mapreduce.am.admin.user.env`
+
+This list can be extended or modified by adding new configuration entries or updating existing values
+beginning with `oozie.launcher.prepend.` within `oozie-site.xml`. Examples can be found in `oozie-default.xml`.
+
+### Logging Configuration
+
+By default, Oozie log configuration is defined in the `oozie-log4j.properties` configuration file.
+
+If the Oozie log configuration file changes, Oozie reloads the new settings automatically.
+
+By default, Oozie logs to Oozie's `logs/` directory.
+
+Oozie logs in 4 different files:
+
+ * oozie.log: web services log streaming works from this log
+ * oozie-ops.log: messages for Admin/Operations to monitor
+ * oozie-instrumentation.log: instrumentation data, every 60 seconds (configurable)
+ * oozie-audit.log: audit messages, workflow jobs changes
+
+The embedded Jetty and embedded Derby log files are also written to Oozie's `logs/` directory.
+
+### Oozie User Authentication Configuration
+
+Oozie supports Kerberos HTTP SPNEGO authentication, pseudo/simple authentication and anonymous access
+for client connections.
+
+Anonymous access (**default**) does not require the user to authenticate and the user ID is obtained from
+the job properties on job submission operations, other operations are anonymous.
+
+Pseudo/simple authentication requires the user to specify the user name on the request, this is done by
+the PseudoAuthenticator class by injecting the `user.name` parameter in the query string of all requests.
+The `user.name` parameter value is taken from the client process Java System property `user.name`.
+
+Kerberos HTTP SPNEGO authentication requires the user to perform a Kerberos HTTP SPNEGO authentication sequence.
+
+If Pseudo/simple or Kerberos HTTP SPNEGO authentication mechanisms are used, Oozie will return the user an
+authentication token HTTP Cookie that can be used in later requests as identity proof.
+
+Oozie uses Apache Hadoop-Auth (Java HTTP SPNEGO) library for authentication.
+This library can be extended to support other authentication mechanisms.
+
+Oozie user authentication is configured using the following configuration properties (default values shown):
+
+
+```
+ oozie.authentication.type=simple
+ oozie.authentication.token.validity=36000
+ oozie.authentication.signature.secret=
+ oozie.authentication.cookie.domain=
+ oozie.authentication.simple.anonymous.allowed=true
+ oozie.authentication.kerberos.principal=HTTP/localhost@${local.realm}
+ oozie.authentication.kerberos.keytab=${oozie.service.HadoopAccessorService.keytab.file}
+```
+
+The `type` defines authentication used for Oozie HTTP endpoint, the supported values are:
+simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#.
+
+The `token.validity` indicates how long (in seconds) an authentication token is valid before it has
+to be renewed.
+
+The `signature.secret` is the signature secret for signing the authentication tokens. It is recommended to not set this, in which
+case Oozie will randomly generate one on startup.
+
+The `oozie.authentication.cookie.domain` The domain to use for the HTTP cookie that stores the
+authentication token. In order to authentication to work correctly across all Hadoop nodes web-consoles
+the domain must be correctly set.
+
+The `simple.anonymous.allowed` indicates if anonymous requests are allowed. This setting is meaningful
+only when using 'simple' authentication.
+
+The `kerberos.principal` indicates the Kerberos principal to be used for HTTP endpoint.
+The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
+
+The `kerberos.keytab` indicates the location of the keytab file with the credentials for the principal.
+It should be the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
+
+### Oozie Hadoop Authentication Configuration
+
+Oozie works with Hadoop versions which support Kerberos authentication.
+
+Oozie Hadoop authentication is configured using the following configuration properties (default values shown):
+
+
+```
+ oozie.service.HadoopAccessorService.kerberos.enabled=false
+ local.realm=LOCALHOST
+ oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
+ oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}
+```
+
+The above default values are for a Hadoop 0.20 secure distribution (with support for Kerberos authentication).
+
+To enable Kerberos authentication, the following property must be set:
+
+
+```
+ oozie.service.HadoopAccessorService.kerberos.enabled=true
+```
+
+When using Kerberos authentication, the following properties must be set to the correct values (default values shown):
+
+
+```
+ local.realm=LOCALHOST
+ oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
+ oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}
+```
+
+**IMPORTANT:** When using Oozie with a Hadoop 20 with Security distribution, the Oozie user in Hadoop must be configured
+as a proxy user.
+
+### User ProxyUser Configuration
+
+Oozie supports impersonation or proxyuser functionality (identical to Hadoop proxyuser capabilities and conceptually
+similar to Unix 'sudo').
+
+Proxyuser enables other systems that are Oozie clients to submit jobs on behalf of other users.
+
+Because proxyuser is a powerful capability, Oozie provides the following restriction capabilities
+(similar to Hadoop):
+
+ * Proxyuser is an explicit configuration on per proxyuser user basis.
+ * A proxyuser user can be restricted to impersonate other users from a set of hosts.
+ * A proxyuser user can be restricted to impersonate users belonging to a set of groups.
+
+There are 2 configuration properties needed to set up a proxyuser:
+
+ * oozie.service.ProxyUserService.proxyuser.#USER#.hosts: hosts from where the user #USER# can impersonate other users.
+ * oozie.service.ProxyUserService.proxyuser.#USER#.groups: groups the users being impersonated by user #USER# must belong to.
+
+Both properties support the '*' wildcard as value. Although this is recommended only for testing/development.
+
+### User Authorization Configuration
+
+Oozie has a basic authorization model:
+
+ * Users have read access to all jobs
+ * Users have write access to their own jobs
+ * Users have write access to jobs based on an Access Control List (list of users and groups)
+ * Users have read access to admin operations
+ * Admin users have write access to all jobs
+ * Admin users have write access to admin operations
+
+If security is disabled all users are admin users.
+
+Oozie security is set via the following configuration property (default value shown):
+
+
+```
+ oozie.service.AuthorizationService.security.enabled=false
+```
+
+NOTE: the old ACL model where a group was provided is still supported if the following property is set
+in `oozie-site.xml`:
+
+
+```
+ oozie.service.AuthorizationService.default.group.as.acl=true
+```
+
+#### Defining Admin Users
+
+Admin users are determined from the list of admin groups, specified in
+ `oozie.service.AuthorizationService.admin.groups` property. Use commas to separate multiple groups, spaces, tabs
+and ENTER characters are trimmed.
+
+If the above property for admin groups is not set, then defining the admin users can happen in the following manners.
+The list of admin users can be in the `conf/adminusers.txt` file. The syntax of this file is:
+
+ * One user name per line
+ * Empty lines and lines starting with '#' are ignored
+
+Admin users can also be defined in
+`oozie.serviceAuthorizationService.admin.users` property. Use commas to separate multiple admin users, spaces, tabs
+and ENTER characters are trimmed.
+
+In case there are admin users defined using both methods, the effective list of admin users will be the union
+of the admin users found in the adminusers.txt and those specified with `oozie.serviceAuthorizationService.admin.users`.
+
+#### Defining Access Control Lists
+
+Access Control Lists are defined in the following ways:
+
+ * workflow job submission over CLI: configuration property `group.name` of `job.properties`
+ * workflow job submission over HTTP: configuration property `group.name` of the XML submitted over HTTP
+ * workflow job re-run: configuration property `oozie.job.acl` (preferred) or configuration property `group.name` of
+ `job.properties`
+ * coordinator job submission over CLI: configuration property `oozie.job.acl` (preferred) or configuration property `group.name`
+ of `job.properties`
+ * bundle job submission over CLI: configuration property `oozie.job.acl` (preferred) or configuration property `group.name` of
+ `job.properties`
+
+For all other workflow, coordinator, or bundle actions the ACL set in beforehand will be used as basis.
+
+Once the ACL for the job is defined, Oozie will check over HDFS whether the user trying to perform a specific action is part of the
+necessary group(s). For implementation details please check out `org.apache.hadoop.security.Groups#getGroups(String user)`.
+
+Note that it's enough that the submitting user be part of at least one group of the ACL. Note also that the ACL can contain user
+names as well. If there is an ACL defined and the submitting user isn't part of any group or user name present in the ACL, an
+`AuthorizationException` is thrown.
+
+**Example: A typical ACL setup**
+
+Detail of `job.properties` on workflow job submission:
+
+```
+user.name=joe
+group.name=marketing,admin,qa,root
+```
+
+HDFS group membership of HDFS user `joe` is `qa`. That is, the check to `org.apache.hadoop.security.Groups#getGroups("joe")` returns
+`qa`. Hence, ACL check will pass inside `AuthorizationService`, because the `user.name` provided belongs to at least of the ACL list
+elements provided as `group.name`.
+
+### Oozie System ID Configuration
+
+Oozie has a system ID that is is used to generate the Oozie temporary runtime directory, the workflow job IDs, and the
+workflow action IDs.
+
+Two Oozie systems running with the same ID will not have any conflict but in case of troubleshooting it will be easier
+to identify resources created/used by the different Oozie systems if they have different system IDs (default value
+shown):
+
+
+```
+ oozie.system.id=oozie-${user.name}
+```
+
+### Filesystem Configuration
+
+Oozie lets you to configure the allowed Filesystems by using the following configuration property in oozie-site.xml:
+
+```
+ <property>
+ <name>oozie.service.HadoopAccessorService.supported.filesystems</name>
+ <value>hdfs</value>
+ </property>
+```
+
+The above value, `hdfs`, which is the default, means that Oozie will only allow HDFS filesystems to be used. Examples of other
+filesystems that Oozie is compatible with are: hdfs, hftp, webhdfs, and viewfs. Multiple filesystems can be specified as
+comma-separated values. Putting a * will allow any filesystem type, effectively disabling this check.
+
+### HCatalog Configuration
+
+Refer to the [Oozie HCatalog Integration](DG_HCatalogIntegration.html) document for a overview of HCatalog and
+integration of Oozie with HCatalog. This section explains the various settings to be configured in oozie-site.xml on
+the Oozie server to enable Oozie to work with HCatalog.
+
+**Adding HCatalog jars to Oozie war:**
+
+ For Oozie server to talk to HCatalog server, HCatalog and hive jars need to be in the server classpath.
+hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath or specified by the
+following configuration property in oozie-site.xml:
+
+```
+ <property>
+ <name>oozie.service.HCatAccessorService.hcat.configuration</name>
+ <value>/local/filesystem/path/to/hive-site.xml</value>
+ </property>
+```
+The hive-site.xml can also be placed in a location on HDFS and the above property can have a value
+of `hdfs://HOST:PORT/path/to/hive-site.xml` to point there instead of the local file system.
+
+The oozie-[version]-hcataloglibs.tar.gz in the oozie distribution bundles the required hcatalog and hive jars that
+needs to be placed in the Oozie server classpath. If using a version of HCatalog bundled in
+Oozie hcataloglibs/, copy the corresponding HCatalog jars from hcataloglibs/ to the libext/ directory. If using a
+different version of HCatalog, copy the required HCatalog jars from such version in the libext/ directory.
+This needs to be done before running the `oozie-setup.sh` script so that these jars get added for Oozie.
+
+**Configure HCatalog URI Handling:**
+
+
+```
+ <property>
+ <name>oozie.service.URIHandlerService.uri.handlers</name>
+ <value>org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler</value>
+ <description>
+ Enlist the different uri handlers supported for data availability checks.
+ </description>
+ </property>
+```
+
+The above configuration defines the different uri handlers which check for existence of data dependencies defined in a
+Coordinator. The default value is `org.apache.oozie.dependency.FSURIHandler`. FSURIHandler supports uris with
+schemes defined in the configuration `oozie.service.HadoopAccessorService.supported.filesystems` which are hdfs, hftp
+and webhcat by default. HCatURIHandler supports uris with the scheme as hcat.
+
+**Configure HCatalog services:**
+
+
+```
+ <property>
+ <name>oozie.services.ext</name>
+ <value>
+ org.apache.oozie.service.JMSAccessorService,
+ org.apache.oozie.service.PartitionDependencyManagerService,
+ org.apache.oozie.service.HCatAccessorService
+ </value>
+ <description>
+ To add/replace services defined in 'oozie.services' with custom implementations.
+ Class names must be separated by commas.
+ </description>
+ </property>
+```
+
+PartitionDependencyManagerService and HCatAccessorService are required to work with HCatalog and support Coordinators
+having HCatalog uris as data dependency. If the HCatalog server is configured to publish partition availability
+notifications to a JMS compliant messaging provider like ActiveMQ, then JMSAccessorService needs to be added
+to `oozie.services.ext` to handle those notifications.
+
+**Configure JMS Provider JNDI connection mapping for HCatalog:**
+
+
+```
+ <property>
+ <name>oozie.service.HCatAccessorService.jmsconnections</name>
+ <value>
+ hcat://hcatserver.colo1.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.colo1.com:61616,
+ default=java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://broker.colo.com:61616;connectionFactoryNames#ConnectionFactory
+ </value>
+ <description>
+ Specify the map of endpoints to JMS configuration properties. In general, endpoint
+ identifies the HCatalog server URL. "default" is used if no endpoint is mentioned
+ in the query. If some JMS property is not defined, the system will use the property
+ defined jndi.properties. jndi.properties files is retrieved from the application classpath.
+ Mapping rules can also be provided for mapping Hcatalog servers to corresponding JMS providers.
+ hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616
+ </description>
+ </property>
+```
+
+ Currently HCatalog does not provide APIs to get the connection details to connect to the JMS Provider it publishes
+notifications to. It only has APIs which provide the topic name in the JMS Provider to which the notifications are
+published for a given database table. So the JMS Provider's connection properties needs to be manually configured
+in Oozie using the above setting. You can either provide a `default` JNDI configuration which will be used as the
+JMS Provider for all HCatalog servers, or can specify a configuration per HCatalog server URL or provide a
+configuration based on a rule matching multiple HCatalog server URLs. For example: With the configuration of
+`hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616`,
+request URL of `hcat://server1.colo1.com:8020 `will map to`tcp://broker.colo1.com:61616`, `hcat://server2.colo2.com:8020`
+will map to`tcp://broker.colo2.com:61616` and so on.
+
+**Configure HCatalog Polling Frequency:**
+
+
+```
+ <property>
+ <name>oozie.service.coord.push.check.requeue.interval
+ </name>
+ <value>600000</value>
+ <description>Command re-queue interval for push dependencies (in millisecond).
+ </description>
+ </property>
+```
+
+ If there is no JMS Provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined
+in `oozie.service.coord.input.check.requeue.interval`. This config also applies to HDFS polling.
+If there is a JMS provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined
+in `oozie.service.coord.push.check.requeue.interval` as a fallback.
+The defaults for `oozie.service.coord.input.check.requeue.interval` and `oozie.service.coord.push.check.requeue.interval`
+are 1 minute and 10 minutes respectively.
+
+### Notifications Configuration
+
+Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss events. For
+more information on the feature, refer [JMS Notifications](DG_JMSNotifications.html) documentation. Oozie can also send email
+notifications on SLA misses.
+
+ * **Message Broker Installation**: <br/>
+For Oozie to send/receive messages, a JMS-compliant broker should be installed. Apache ActiveMQ is a popular JMS-compliant
+broker usable for this purpose. See [here](http://activemq.apache.org/getting-started.html) for instructions on
+installing and running ActiveMQ.
+
+ * **Services**: <br/>
+Add/modify `oozie.services.ext` property in `oozie-site.xml` to include the following services.
+
+```
+ <property>
+ <name>oozie.services.ext</name>
+ <value>
+ org.apache.oozie.service.JMSAccessorService,
+ org.apache.oozie.service.JMSTopicService,
+ org.apache.oozie.service.EventHandlerService,
+ org.apache.oozie.sla.service.SLAService
+ </value>
+ </property>
+```
+
+ * **Event Handlers**: <br/>
+
+```
+ <property>
+ <name>oozie.service.EventHandlerService.event.listeners</name>
+ <value>
+ org.apache.oozie.jms.JMSJobEventListener,
+ org.apache.oozie.sla.listener.SLAJobEventListener,
+ org.apache.oozie.jms.JMSSLAEventListener,
+ org.apache.oozie.sla.listener.SLAEmailEventListener
+ </value>
+ </property>
+```
+It is also recommended to increase `oozie.service.SchedulerService.threads` to 15 for faster event processing and sending notifications. The services and their functions are as follows: <br/>
+ JMSJobEventListener - Sends JMS job notifications <br/>
+ JMSSLAEventListener - Sends JMS SLA notifications <br/>
+ SLAEmailEventListener - Sends Email SLA notifications <br/>
+ SLAJobEventListener - Processes job events and calculates SLA. Does not send any notifications
+
+ * **JMS properties**: <br/>
+Add `oozie.jms.producer.connection.properties` property in `oozie-site.xml`. Its value corresponds to an
+identifier (e.g. default) assigned to a semi-colon separated key#value list of properties from your JMS broker's
+`jndi.properties` file. The important properties are `java.naming.factory.initial` and `java.naming.provider.url`.
+
+As an example, if using ActiveMQ in local env, the property can be set to
+
+```
+ <property>
+ <name>oozie.jms.producer.connection.properties</name>
+ <value>
+ java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory
+ </value>
+ </property>
+```
+ * **JMS Topic name**: <br/>
+JMS consumers listen on a particular "topic". Hence Oozie needs to define a topic variable with which to publish messages
+about the various jobs.
+
+```
+ <property>
+ <name>oozie.service.JMSTopicService.topic.name</name>
+ <value>
+ default=${username}
+ </value>
+ <description>
+ Topic options are ${username}, ${jobId}, or a fixed string which can be specified as default or for a
+ particular job type.
+ For e.g To have a fixed string topic for workflows, coordinators and bundles,
+ specify in the following comma-separated format: {jobtype1}={some_string1}, {jobtype2}={some_string2}
+ where job type can be WORKFLOW, COORDINATOR or BUNDLE.
+ Following example defines topic for workflow job, workflow action, coordinator job, coordinator action,
+ bundle job and bundle action
+ WORKFLOW=workflow,
+ COORDINATOR=coordinator,
+ BUNDLE=bundle
+ For jobs with no defined topic, default topic will be ${username}
+ </description>
+ </property>
+```
+
+Another related property is the topic prefix.
+
+```
+ <property>
+ <name>oozie.service.JMSTopicService.topic.prefix</name>
+ <value></value>
+ <description>
+ This can be used to append a prefix to the topic in oozie.service.JMSTopicService.topic.name. For eg: oozie.
+ </description>
+ </property>
+```
+
+
+### Setting Up Oozie with HTTPS (SSL)
+
+**IMPORTANT**:
+The default HTTPS configuration will cause all Oozie URLs to use HTTPS except for the JobTracker callback URLs. This is to simplify
+configuration (no changes needed outside of Oozie), but this is okay because Oozie doesn't inherently trust the callbacks anyway;
+they are used as hints.
+
+The related environment variables are explained at [Environment Setup](AG_Install.html#Environment_Setup).
+
+You can use either a certificate from a Certificate Authority or a Self-Signed Certificate. Using a self-signed certificate
+requires some additional configuration on each Oozie client machine. If possible, a certificate from a Certificate Authority is
+recommended because it's simpler to configure.
+
+There's also some additional considerations when using Oozie HA with HTTPS.
+
+#### To use a Self-Signed Certificate
+There are many ways to create a Self-Signed Certificate, this is just one way. We will be using
+the [keytool](http://docs.oracle.com/javase/6/docs/technotes/tools/solaris/keytool.html) program, which is
+included with your JRE. If it's not on your path, you should be able to find it in $JAVA_HOME/bin.
+
+1. Run the following command (as the Oozie user) to create the keystore file, which will be named `.keystore` and located in the
+ Oozie user's home directory.
+
+ ```
+ keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=hostname" -storepass password -keypass password
+ ```
+ The `hostname` should be the host name of the Oozie Server or a wildcard on the subdomain it belongs to. Make sure to include
+ the "CN=" part. You can change `storepass` and `keypass` values, but they should be the same. If you do want to use something
+ other than password, you'll also need to change the value of the `oozie.https.keystore.pass` property in `oozie-site.xml` to
+ match; `password` is the default.
+
+ For example, if your Oozie server was at oozie.int.example.com, then you would do this:
+
+ ```
+ keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=oozie.int.example.com" -storepass password -keypass password
+ ```
+ If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie servers in the HA group can use.
+ To do that, you'll need to use a wildcard on the subdomain it belongs to:
+
+ ```
+ keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=*.int.example.com" -storepass password -keypass password
+ ```
+ The above would work on any server in the int.example.com domain.
+
+2. Run the following command (as the Oozie user) to export a certificate file from the keystore file:
+
+ ```
+ keytool -exportcert -alias jetty -file path/to/anywhere/certificate.cert -storepass password
+ ```
+
+3. Run the following command (as any user) to create a truststore containing the certificate we just exported:
+
+ ```
+ keytool -import -alias jetty -file path/to/certificate.cert -keystore /path/to/anywhere/oozie.truststore -storepass password2
+ ```
+ You'll need the `oozie.truststore` later if you're using the Oozie client (or other Java-based client); otherwise, you can skip
+ this step. The `storepass` value here is only used to verify or change the truststore and isn't typically required when only
+ reading from it; so it does not have to be given to users only using the client.
+
+#### To use a Certificate from a Certificate Authority
+
+1. You will need to make a request to a Certificate Authority in order to obtain a proper Certificate; please consult a Certificate
+ Authority on this procedure. If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie
+ servers in the HA group can use. To do that, you'll need to use a wild on the subdomain it belongs to (e.g. "*.int.example.com").
+
+2. Once you have your .cert file, run the following command (as the Oozie user) to create a keystore file from your certificate:
+
+ ```
+ keytool -import -alias jetty -file path/to/certificate.cert
+ ```
+ The keystore file will be named `.keystore` and located in the Oozie user's home directory.
+
+#### Configure the Oozie Server to use SSL (HTTPS)
+
+1. Make sure the Oozie server isn't running
+
+2. Configure settings necessary for enabling SSL/TLS support in `oozie-site.xml`.
+
+ 2a. Set `oozie.https.enabled` to `true`. To revert back to HTTP, set `oozie.https.enabled` to `false`.
+
+ 2b. Set location and password for the keystore and location for truststore by setting `oozie.https.keystore.file`,
+ `oozie.https.keystore.pass`, `oozie.https.truststore.file`.
+
+ **Note:** `oozie.https.truststore.file` can be overridden by setting `javax.net.ssl.trustStore` system property.
+
+ The default HTTPS port Oozie listens on for secure connections is 11443; it can be changed via `oozie.https.port`.
+
+ It is possible to specify other HTTPS settings via `oozie-site.xml`:
+ - To include / exclude cipher suites, set `oozie.https.include.cipher.suites` / `oozie.https.exclude.cipher.suites`.
+ - To include / exclude TLS protocols, set `oozie.https.include.protocols` / `oozie.https.exclude.protocols`.
+ **Note:** Exclude is always preferred over include (i.e. if you both include and exclude an entity, it will be excluded).
+
+3. Start the Oozie server
+
+ **Note:** If using Oozie HA, make sure that each Oozie server has a copy of the .keystore file.
+
+#### Configure the Oozie Client to connect using SSL (HTTPS)
+
+The first two steps are only necessary if you are using a Self-Signed Certificate; the third is required either way.
+Also, these steps must be done on every machine where you intend to use the Oozie Client.
+
+1. Copy or download the oozie.truststore file onto the client machine
+
+2. When using any Java-based program, you'll need to pass `-Djavax.net.ssl.trustStore` to the JVM. To
+ do this for the Oozie client:
+
+ ```
+ export OOZIE_CLIENT_OPTS='-Djavax.net.ssl.trustStore=/path/to/oozie.truststore'
+ ```
+
+3. When using the Oozie Client, you will need to use `https://oozie.server.hostname:11443/oozie` instead of
+ `http://oozie.server.hostname:11000/oozie` -- Java will not automatically redirect from the http address to the https address.
+
+#### Connect to the Oozie Web UI using SSL (HTTPS)
+
+1. Use `https://oozie.server.hostname:11443/oozie`
+ though most browsers should automatically redirect you if you use `http://oozie.server.hostname:11000/oozie`
+
+ **IMPORTANT**: If using a Self-Signed Certificate, your browser will warn you that it can't verify the certificate or something
+ similar. You will probably have to add your certificate as an exception.
+
+#### Additional considerations for Oozie HA with SSL
+
+You'll need to configure the load balancer to do SSL pass-through. This will allow the clients talking to Oozie to use the
+SSL certificate provided by the Oozie servers (so the load balancer does not need one). Please consult your load balancer's
+documentation on how to configure this. Make sure to point the load balancer at the `https://HOST:HTTPS_PORT` addresses for your
+Oozie servers. Clients can then connect to the load balancer at `https://LOAD_BALANCER_HOST:PORT`.
+
+**Important:** Callbacks from the ApplicationMaster are done via http or https depending on what you enter for the
+`OOZIE_BASE_URL` property. If you are using a Certificate from a Certificate Authority, you can simply put the https address here.
+If you are using a self-signed certificate, you have to do one of the following options (Option 1 is recommended):
+
+Option 1) You'll need to follow the steps in
+the [Configure the Oozie Client to connect using SSL (HTTPS)](AG_Install.html#Configure_the_Oozie_Client_to_connect_using_SSL_HTTPS)
+section, but on the host of the ApplicationMaster. You can then set `OOZIE_BASE_URL` to the load balancer https address.
+This will allow the ApplicationMaster to contact the Oozie server with https (like the Oozie client, they are also Java
+programs).
+
+Option 2) You'll need setup another load balancer, or another "pool" on the existing load balancer, with the http addresses of the
+Oozie servers. You can then set `OOZIE_BASE_URL` to the load balancer http address. Clients should use the https load balancer
+address. This will allow clients to use https while the ApplicationMaster uses http for callbacks.
+
+### Fine Tuning an Oozie Server
+
+Refer to the [oozie-default.xml](./oozie-default.xml) for details.
+
+### Using Instrumentation instead of Metrics
+
+As of version 4.1.0, Oozie includes a replacement for the Instrumentation based on Codahale's Metrics library. It includes a
+number of improvements over the original Instrumentation included in Oozie. They both report most of the same information, though
+the formatting is slightly different and there's some additional information in the Metrics version; the format of the output to the
+oozie-instrumentation log is also different.
+
+As of version 5.0.0, `MetricsInstrumentationService` is the default one, it's enlisted in `oozie.services`:
+
+```
+ <property>
+ <name>oozie.services</name>
+ <value>
+ ...
+ org.apache.oozie.service.MetricsInstrumentationService,
+ ...
+ </value>
+ </property>
+```
+
+The deprecated `InstrumentationService` can be enabled by adding `InstrumentationService` reference to the list of
+`oozie.services.ext`:
+
+```
+ <property>
+ <name>oozie.services.ext</name>
+ <value>
+ ...
+ org.apache.oozie.service.InstrumentationService,
+ ...
+ </value>
+ </property>
+```
+
+By default the `admin/instrumentation` REST endpoint is no longer be available and instead the `admin/metrics` endpoint can
+be used (see the [Web Services API](WebServicesAPI.html#Oozie_Metrics) documentation for more details); the Oozie Web UI also replaces
+the "Instrumentation" tab with a "Metrics" tab.
+
+If the deprecated `InstrumentationService` is used, the `admin/instrumentation` REST endpoint gets enabled, the `admin/metrics`
+REST endpoint is no longer available (see the [Web Services API](WebServicesAPI.html#Oozie_Metrics) documentation for more details);
+the Oozie Web UI also replaces the "Metrics" tab with the "Instrumentation" tab.
+
+We can also publish the instrumentation metrics to the external server graphite or ganglia. For this the following
+properties should be specified in oozie-site.xml :
+
+```
+ <property>
+ <name>oozie.external_monitoring.enable</name>
+ <value>false</value>
+ <description>
+ If the oozie functional metrics needs to be exposed to the metrics-server backend, set it to true
+ If set to true, the following properties has to be specified : oozie.metrics.server.name,
+ oozie.metrics.host, oozie.metrics.prefix, oozie.metrics.report.interval.sec, oozie.metrics.port
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.external_monitoring.type</name>
+ <value>graphite</value>
+ <description>
+ The name of the server to which we want to send the metrics, would be graphite or ganglia.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.external_monitoring.address</name>
+ <value>http://localhost:2020</value>
+ </property>
+
+ <property>
+ <name>oozie.external_monitoring.metricPrefix</name>
+ <value>oozie</value>
+ </property>
+
+ <property>
+ <name>oozie.external_monitoring.reporterIntervalSecs</name>
+ <value>60</value>
+ </property>
+```
+
+We can also publish the instrumentation metrics via JMX interface. For this the following property should be specified
+in oozie-site.xml :
+
+```
+ <property>
+ <name>oozie.jmx_monitoring.enable</name>
+ <value>false</value>
+ <description>
+ If the oozie functional metrics needs to be exposed via JMX interface, set it to true.
+ </description>
+ </property>>
+```
+
+<a name="HA"></a>
+### High Availability (HA)
+
+Multiple Oozie Servers can be configured against the same database to provide High Availability (HA) of the Oozie service.
+
+#### Pre-requisites
+
+1. A database that supports multiple concurrent connections. In order to have full HA, the database should also have HA support, or
+ it becomes a single point of failure.
+
+ **NOTE:** The default derby database does not support this
+
+2. A ZooKeeper ensemble.
+
+ Apache ZooKeeper is a distributed, open-source coordination service for distributed applications; the Oozie servers use it for
+ coordinating access to the database and communicating with each other. In order to have full HA, there should be at least 3
+ ZooKeeper servers.
+ More information on ZooKeeper can be found [here](http://zookeeper.apache.org).
+
+3. Multiple Oozie servers.
+
+ **IMPORTANT:** While not strictly required for all configuration properties, all of the servers should ideally have exactly the same
+ configuration for consistency's sake.
+
+4. A Loadbalancer, Virtual IP, or Round-Robin DNS.
+
+ This is used to provide a single entry-point for users and for callbacks from the JobTracker/ResourceManager. The load balancer
+ should be configured for round-robin between the Oozie servers to distribute the requests. Users (using either the Oozie client, a
+ web browser, or the REST API) should connect through the load balancer. In order to have full HA, the load balancer should also
+ have HA support, or it becomes a single point of failure.
+
+#### Installation/Configuration Steps
+
+1. Install identically configured Oozie servers normally. Make sure they are all configured against the same database and make sure
+ that you DO NOT start them yet.
+
+2. Add the following services to the extension services configuration property in oozie-site.xml in all Oozie servers. This will
+ make Oozie use the ZooKeeper versions of these services instead of the default implementations.
+
+
+ ```
+ <property>
+ <name>oozie.services.ext</name>
+ <value>
+ org.apache.oozie.service.ZKLocksService,
+ org.apache.oozie.service.ZKXLogStreamingService,
+ org.apache.oozie.service.ZKJobsConcurrencyService,
+ org.apache.oozie.service.ZKUUIDService
+ </value>
+ </property>
+ ```
+
+3. Add the following property to oozie-site.xml in all Oozie servers. It should be a comma-separated list of host:port pairs of the
+ ZooKeeper servers. The default value is shown below.
+
+
+ ```
+ <property>
+ <name>oozie.zookeeper.connection.string</name>
+ <value>localhost:2181</value>
+ </property>
+ ```
+
+4. (Optional) Add the following property to oozie-site.xml in all Oozie servers to specify the namespace to use. All of the Oozie
+ Servers that are planning on talking to each other should have the same namespace. If there are multiple Oozie setups each doing
+ their own HA, they should have their own namespace. The default value is shown below.
+
+
+ ```
+ <property>
+ <name>oozie.zookeeper.namespace</name>
+ <value>oozie</value>
+ </property>
+ ```
+
+5. Change the value of `OOZIE_BASE_URL` in oozie-site.xml to point to the loadbalancer or virtual IP, for example:
+
+
+ ```
+ <property>
+ <name>oozie.base.url</name>
+ <value>http://my.loadbalancer.hostname:11000/oozie</value>
+ </property>
+ ```
+
+6. (Optional) If using a secure cluster, see [Security](AG_Install.html#Security) below on configuring Kerberos with Oozie HA.
+
+7. Start the ZooKeeper servers.
+
+8. Start the Oozie servers.
+
+ Note: If one of the Oozie servers becomes unavailable, querying Oozie for the logs from a job in the Web UI, REST API, or client may
+ be missing information until that server comes back up.
+
+#### Security
+
+Oozie HA works with the existing Oozie security framework and settings. For HA features (log streaming, share lib, etc) to work
+properly in a secure setup, following property can be set on each server. If `oozie.server.authentication.type` is not set, then
+server-server authentication will fall back on `oozie.authentication.type`.
+
+
+```
+<property>
+ <name>oozie.server.authentication.type</name>
+ <value>kerberos</value>
+</property>
+```
+
+Below are some additional steps and information specific to Oozie HA:
+
+1. (Optional) To prevent unauthorized users or programs from interacting with or reading the znodes used by Oozie in ZooKeeper,
+ you can tell Oozie to use Kerberos-backed ACLs. To enforce this for all of the Oozie-related znodes, simply add the following
+ property to oozie-site.xml in all Oozie servers and set it to `true`. The default is `false`.
+
+
+ ```
+ <property>
+ <name>oozie.zookeeper.secure</name>
+ <value>true</value>
+ </property>
+ ```
+
+ Note: The Kerberos principals of each of the Oozie servers should have the same primary name (i.e. in `primary/instance@REALM`, each
+ server should have the same value for `primary`).
+
+ **Important:** Once this property is set to `true`, it will set the ACLs on all existing Oozie-related znodes to only allow Kerberos
+ authenticated users with a principal that has the same primary as described above (also for any subsequently created new znodes).
+ This means that if you ever want to turn this feature off, you will have to manually connect to ZooKeeper using a Kerberos principal
+ with the same primary and either delete all znodes under and including the namespace (i.e. if `oozie.zookeeper.namespace` ` `oozie=
+ then that would be `/oozie`); alternatively, instead of deleting them all, you can manually set all of their ACLs to `world:anyone`.
+ In either case, make sure that no Oozie servers are running while this is being done.
+
+ Also, in your zoo.cfg for ZooKeeper, make sure to set the following properties:
+
+ ```
+ authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
+ kerberos.removeHostFromPrincipal=true
+ kerberos.removeRealmFromPrincipal=true
+ ```
+
+2. Until Hadoop 2.5.0 and later, there is a known limitation where each Oozie server can only use one HTTP principal. However,
+ for Oozie HA, we need to use two HTTP principals: `HTTP/oozie-server-host@realm` and `HTTP/load-balancer-host@realm`. This
+ allows access to each Oozie server directly and through the load balancer. While users should always go through the load balancer,
+ certain features (e.g. log streaming) require the Oozie servers to talk to each other directly; it can also be helpful for an
+ administrator to talk directly to an Oozie server. So, if using a Hadoop version prior to 2.5.0, you will have to choose which
+ HTTP principal to use as you cannot use both; it is recommended to choose `HTTP/load-balancer-host@realm` so users can connect
+ through the load balancer. This will prevent Oozie servers from talking to each other directly, which will effectively disable
+ log streaming.
+
+ For Hadoop 2.5.0 and later:
+
+ 2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the two HTTP principals mentioned above.
+
+
+ 2b. Set `oozie.authentication.kerberos.principal` to * (that is, an asterisks) so it will use both HTTP principals.
+
+ For earlier versions of Hadoop:
+
+ 2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the load balancer HTTP principal
+
+
+ 2b. Set `oozie.authentication.kerberos.principal` to `HTTP/load-balancer-host@realm`.
+
+3. With Hadoop 2.6.0 and later, a rolling random secret that is synchronized across all Oozie servers will be used for signing the
+ Oozie auth tokens. This is done automatically when HA is enabled; no additional configuration is needed.
+
+ For earlier versions of Hadoop, each server will have a different random secret. This will still work but will likely result in
+ additional calls to the KDC to authenticate users to the Oozie server (because the auth tokens will not be accepted by other
+ servers, which will cause a fallback to Kerberos).
+
+4. If you'd like to use HTTPS (SSL) with Oozie HA, there's some additional considerations that need to be made.
+ See the [Setting Up Oozie with HTTPS (SSL)](AG_Install.html#Setting_Up_Oozie_with_HTTPS_SSL) section for more information.
+
+#### JobId sequence
+Oozie in HA mode, uses ZK to generate job id sequence. Job Ids are of following format.
+`<Id sequence>-<yyMMddHHmmss(server start time)>-<system_id>-<W/C/B>`
+
+Where, `<systemId>` is configured as `oozie.system.id` (default is "oozie-" + "user.name")
+W/C/B is suffix to job id indicating that generated job is a type of workflow or coordinator or bundle.
+
+Maximum allowed character for job id sequence is 40. "Id sequence" is stored in ZK and reset to 0 once maximum job id sequence is
+reached. Maximum job id sequence is configured as `oozie.service.ZKUUIDService.jobid.sequence.max`, default value is 99999999990.
+
+
+```
+<property>
+ <name>oozie.service.ZKUUIDService.jobid.sequence.max</name>
+ <value>99999999990</value>
+</property>
+```
+
+## Starting and Stopping Oozie
+
+Use the standard commands to start and stop Oozie.
+
+## Oozie Command Line Installation
+
+Copy and expand the `oozie-client` TAR.GZ file bundled with the distribution. Add the `bin/` directory to the `PATH`.
+
+Refer to the [Command Line Interface Utilities](DG_CommandLineTool.html) document for a full reference of the `oozie`
+command line tool.
+
+## Oozie Share Lib
+
+The Oozie sharelib TAR.GZ file bundled with the distribution contains the necessary files to run Oozie map-reduce streaming, pig,
+hive, sqooop, and distcp actions. There is also a sharelib for HCatalog. The sharelib is required for these actions to work; any
+other actions (mapreduce, shell, ssh, and java) do not require the sharelib to be installed.
+
+As of Oozie 4.0, the following property is included. If true, Oozie will create and ship a "launcher jar" to hdfs that contains
+classes necessary for the launcher job. If false, Oozie will not do this, and it is assumed that the necessary classes are in their
+respective sharelib jars or the "oozie" sharelib instead. When false, the sharelib is required for ALL actions; when true, the
+sharelib is only required for actions that need additional jars (the original list from above).
+
+
+```
+<property>
+ <name>oozie.action.ship.launcher.jar</name>
+ <value>true</value>
+</property>
+```
+
+Using sharelib CLI, sharelib files are copied to new lib_`<timestamped>` directory. At start, server picks the sharelib from latest
+time-stamp directory. While starting, server also purges sharelib directory which are older than sharelib retention days
+(defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).
+
+Sharelib mapping file can be also configured. Configured file is a key value mapping, where key will be the sharelib name for the
+action and value is a comma separated list of DFS or local filesystem directories or jar files. Local filesystem refers to the local
+filesystem of the node where the Oozie launcher is running. This can be configured in oozie-site.xml as :
+
+```
+ <!-- OOZIE -->
+ <property>
+ <name>oozie.service.ShareLibService.mapping.file</name>
+ <value></value>
+ <description>
+ Sharelib mapping files contains list of key=value,
+ where key will be the sharelib name for the action and value is a comma separated list of
+ DFS or local filesystem directories or jar files.
+ Example.
+ oozie.pig_10=hdfs:///share/lib/pig/pig-0.10.1/lib/
+ oozie.pig=hdfs:///share/lib/pig/pig-0.11.1/lib/
+ oozie.distcp=hdfs:///share/lib/hadoop-2.2.0/share/hadoop/tools/lib/hadoop-distcp-2.2.0.jar
+ oozie.spark=hdfs:///share/lib/spark/lib/,hdfs:///share/lib/spark/python/lib/pyspark.zip,hdfs:///share/lib/spark/python/lib/py4j-0-9-src.zip
+ oozie.hive=file:///usr/local/oozie/share/lib/hive/
+ </description>
+ </property>
+```
+
+Example mapping file with local filesystem resources:
+
+
+```
+ <property>
+ <name>oozie.service.ShareLibService.mapping.file</name>
+ <value>
+ oozie.distcp=file:///usr/local/oozie/share/lib/distcp
+ oozie.hcatalog=file:///usr/local/oozie/share/lib/hcatalog
+ oozie.hive=file:///usr/local/oozie/share/lib/hive
+ oozie.hive2=file:///usr/local/oozie/share/lib/hive2
+ oozie.mapreduce-streaming=file:///usr/local/oozie/share/lib/mapreduce-streaming
+ oozie.oozie=file://usr/local/oozie/share/lib/oozie
+ oozie.pig=file:///usr/local/oozie/share/lib/pig
+ oozie.spark=file:///usr/local/oozie/share/lib/spark
+ oozie.sqoop=file:///usr/localoozie/share/lib/sqoop
+ </value>
+ </property>
+```
+
+If you are using local filesystem resources in the mapping file, make sure corresponding jars are already deployed to
+all the nodes where Oozie launcher jobs will be executed, and the files are readable by the launchers. To do this, you
+can extract Oozie sharelib TAR.GZ file in the directory of your choice on the nodes, and set permission of the files.
+
+Oozie sharelib TAR.GZ file bundled with the distribution does not contain pyspark and py4j zip files since they vary
+with Apache Spark version. Therefore, to run pySpark using Spark Action, user need to specify pyspark and py4j zip
+files. These files can be added either to workflow's lib/ directory, to the sharelib or in sharelib mapping file.
+
+
+## Oozie Coordinators/Bundles Processing Timezone
+
+By default Oozie runs coordinator and bundle jobs using `UTC` timezone for datetime values specified in the application
+XML and in the job parameter properties. This includes coordinator applications start and end times of jobs, coordinator
+datasets initial-instance, and bundle applications kickoff times. In addition, coordinator dataset instance URI templates
+will be resolved using datetime values of the Oozie processing timezone.
+
+It is possible to set the Oozie processing timezone to a timezone that is an offset of UTC, alternate timezones must
+expressed in using a GMT offset ( `GMT+/-#### ` ). For example: `GMT+0530` (India timezone).
+
+To change the default `UTC` timezone, use the `oozie.processing.timezone` property in the `oozie-site.xml`. For example:
+
+
+```
+<configuration>
+ <property>
+ <name>oozie.processing.timezone</name>
+ <value>GMT+0530</value>
+ </property>
+</configuration>
+```
+
+**IMPORTANT:** If using a processing timezone other than `UTC`, all datetime values in coordinator and bundle jobs must
+be expressed in the corresponding timezone, for example `2012-08-08T12:42+0530`.
+
+**NOTE:** It is strongly encouraged to use `UTC`, the default Oozie processing timezone.
+
+For more details on using an alternate Oozie processing timezone, please refer to the
+[Coordinator Functional Specification, section '4. Datetime'](CoordinatorFunctionalSpec.html#datetime)
+
+<a name="UberJar"></a>
+## MapReduce Workflow Uber Jars
+For Map-Reduce jobs (not including streaming or pipes), additional jar files can also be included via an uber jar. An uber jar is a
+jar file that contains additional jar files within a "lib" folder (see
+[Workflow Functional Specification](WorkflowFunctionalSpec.html#AppDeployment) for more information). Submitting a workflow with an uber jar
+requires at least Hadoop 2.2.0 or 1.2.0. As such, using uber jars in a workflow is disabled by default. To enable this feature, use
+the `oozie.action.mapreduce.uber.jar.enable` property in the `oozie-site.xml` (and make sure to use a supported version of Hadoop).
+
+
+```
+<configuration>
+ <property>
+ <name>oozie.action.mapreduce.uber.jar.enable</name>
+ <value>true</value>
+ </property>
+</configuration>
+```
+
+## Advanced/Custom Environment Settings
+
+Oozie can be configured to use Unix standard filesystem hierarchy for its different files
+(configuration, logs, data and temporary files).
+
+These settings must be done in the `bin/oozie-env.sh` script.
+
+This script is sourced before the configuration `oozie-env.sh` and supports additional
+environment variables (shown with their default values):
+
+
+```
+export OOZIE_CONFIG=${OOZIE_HOME}/conf
+export OOZIE_DATA={OOZIE_HOME}/data
+export OOZIE_LOG={OOZIE_HOME}/logs
+export JETTY_OUT=${OOZIE_LOGS}/jetty.out
+export JETTY_PID=/tmp/oozie.pid
+```
+
+Sample values to make Oozie follow Unix standard filesystem hierarchy:
+
+
+```
+export OOZIE_CONFIG=/etc/oozie
+export OOZIE_DATA=/var/lib/oozie
+export OOZIE_LOG=/var/log/oozie
+export JETTY_PID=/tmp/oozie.pid
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
[16/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_CoordinatorRerun.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_CoordinatorRerun.md b/docs/src/site/markdown/DG_CoordinatorRerun.md
new file mode 100644
index 0000000..f535d16
--- /dev/null
+++ b/docs/src/site/markdown/DG_CoordinatorRerun.md
@@ -0,0 +1,53 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Coordinator Rerun
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Pre-Conditions
+
+ * Rerun coordinator action must be in TIMEDOUT/SUCCEEDED/KILLED/FAILED.
+ * Coordinator actions cannot be rerun if the coordinator job is in the PREP or IGNORED state.
+ * Rerun a PAUSED coordinator job, the status and pause time and pending flag will not be reset.
+ * Rerun a SUSPENDED coordinator job, the status will reset to RUNNING.
+ * All rerun actions must exist already.
+ * Coordinator Rerun will only use the original configs from first run.
+ * Coordinator Rerun will not re-read the coordinator.xml in hdfs.
+
+## Rerun Arguments
+
+
+```
+$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>]
+[-action 1, 3-4, 7-40] (-action or -date is required to rerun.)
+[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
+(if neither -action nor -date is given, the exception will be thrown.)
+```
+
+ * Either -action or -date should be given.
+ * If -action and -date both are given, an error will be thrown.
+ * Multiple ranges can be used in -action or -date. See the above examples.
+ * If one of action in the given list of -action does not exist or not in terminal state, the rerun throws an error.
+ * The dates specified in -date must be UTC.
+ * Single date specified in -date must be able to find an action with matched nominal time to be effective.
+ * If -nocleanup is given, coordinator directories will not be removed; otherwise the 'output-event' will be deleted, unless nocleanup attribute is explicitly set in coordinator.xml
+ * If -refresh is set, new dataset is re-evaluated for latest() and future().
+ * If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked.
+ * If -failed is set, re-runs the failed workflow actions of the coordinator actions.
+ * -config can be used to supply properties to workflow by job configuration file '.xml' or '.properties'.
+
+## Rerun coordinator actions
+
+ * Rerun terminated (timeout, succeeded, killed, failed) coordinator actions.
+ * By default, Oozie will delete the 'output-event' directories before changing actions' status and materializing actions.
+ * If coordinator job is RUNNING, rerun actions will be materialized and compete with current running actions.
+ * Rerun for job, user should use job's start date and end date in -date.
+ * If the user specifies a date range (say Jan 1 to May 1), the actions that will be re-run are the existing actions
+ within that range. If the existing actions are action #5....#40, which map to Jan 15 to Feb 15, then only those actions will run.
+ * The rerun action_id and nominal_time of the actions which are eligible to rerun will be returned.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_CustomActionExecutor.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_CustomActionExecutor.md b/docs/src/site/markdown/DG_CustomActionExecutor.md
new file mode 100644
index 0000000..5768b27
--- /dev/null
+++ b/docs/src/site/markdown/DG_CustomActionExecutor.md
@@ -0,0 +1,83 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Custom Action Nodes
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Introduction
+Oozie can be extended to support additional action types by writing a custom [Action Node](WorkflowFunctionalSpec.html#ActionNodes). Action Nodes can be synchronous or asynchronous.
+
+ * Synchronous Node - Sync nodes are executed inline by Oozie, which waits for completion of these nodes before proceeding. Hence, these nodes should almost never be used and are meant for lightweight tasks like FileSystem move, mkdir, delete.
+ * Asynchronous Nodes - Oozie starts asynchronous nodes, and then monitors the action being executed for completion. This is done via a callback from the action or Oozie polling for the action status.
+
+## Writing a custom Action Node
+Action Executors are configured in the oozie configuration file oozie-site.xml. These executors are loaded during Oozie startup. [Deploying a Custom Action Executor](DG_CustomActionExecutor.html#Deploying_a_custom_Action_Executor).
+
+Action Executors MUST extend the `ActionExecutor` class and override the required methods.
+
+Most methods take as argument the Execution Context and the actual Action object with various configuration properties resolved.
+### ActionExecutor.Context
+The Execution context gives Action Nodes access to configuration properties, methods to set the state of the action, methods to set variables which are to be made available later in the execution path.
+
+**The following methods from the ActionExecutor interface should be implemented.**
+### Constructor
+A no argument constructor should be implemented, which calls super(ACTION_TYPE). ACTION_TYPE is the name of the action which will be used in the workflow xml, and is used by Oozie to instantiate the correct type of Executor.
+
+### initActionType()
+This method is called once during initialization of the Action Executor during Oozie startup. Any common initialization code for the Action Node should go here.
+
+As an example, setting up of error handling for the Custom Action should be done here.
+
+This method must call super.initActionType() as it's first statement.
+
+### start(ActionExecutor.Context context, Action action)
+The action start up happens here.
+
+ * Async Actions - The action should be started and context.setStartData(externalId, trackerUri, consoleUrl) must be set. A check can be made for whether the action has completed, in which case context.setExecutionData(externalStatus, actionData) must be called.
+ * Sync Actions - The action should be started and should complete execution. context.setExecutionData(externalStatus, actionData) must be called.
+### check(ActionExecutor.Context context, Action action)
+`check(...)` is used by Oozie to poll for the status of the action. This method should interact with the action started previously, and update the status. If the action has completed, context.setExecutionData(externalStatus, actionData) must be called. Otherwise, the status can be updated using context.setExternalStatus(externalStatus).
+
+For sync actions, this method will not be called, and should throw an UnsupportedOperationException().
+### kill(ActionExecutor.Context context, Action action)
+`kill(...)` is called when there is an attempt to kill the running job or action. No workflow transition is made after this.
+
+The implementation for a custom action should interact with and kill the running action, and take care of any cleanup which may be required. context.setEndData(status, signalValue) should be called with both values set to Action.Status.KILLED.
+### end(ActionExecutor.Context context, Action action)
+`end(...)` is used for any cleanup or processing which may need to be done after completion of the action. After any processing, context.setEndData(status, signalValue) should be called to complete execution of the action and trigger the next workflow transition. signalValue can be Action.Status.OK or Action.Status.ERROR.
+### Registering Errors
+Oozie actions can generate different types of Errors.
+
+ * TRANSIENT - will be retried
+ * NON TRANSIENT - the job will be suspended and can be resumed later by human intervention, after fixing whatever problem caused this error.
+ * ERROR - causes the error transition to be taken.
+ * FAILED - the action and the job are set to FAILED state. No transitions are taken.
+registerError(exceptionClassName, errorType, errorMessage) can be used to register possible exceptions while executing the action, along with their type and error message. This will normally be done during initialization of the Action Executor.
+
+## Deploying a custom Action Executor
+Action Nodes can be registered in the oozie configuration file oozie-site.xml, by changing the property 'oozie.service.ActionService.executor.ext.classes'. For multiple Executors, the class name should be separated by commas.
+
+``` <property>
+ <name>oozie.service.ActionService.executor.ext.classes</name>
+ <value>
+ org.apache.oozie.wf.action.decision.CustomActionExecutor,
+ Custom_Action_Executr_2.class
+ </value>
+ </property>
+```
+Any configuration properties to be made available to this class should also be added to oozie-site.xml. The convention to be followed for naming these properties is 'oozie.action.[ActionName].property.name'
+
+The XML schema (XSD) for the new Actions should be added to oozie-site.xml, under the property 'oozie.service.WorkflowSchemaService.ext.schemas'. A comma separated list for multiple Action schemas.
+
+The XML schema (XSD) for the new action should be also added to Fluent Job API. Please refer to
+[Fluent Job API :: How To Extend](DG_FluentJobAPI.html#AE.C_Appendix_C_How_To_Extend) for details.
+
+The executor class should be placed along with the oozie webapp in the correct path. Once Oozie is restarted, the custom action node can be used in workflows.
+
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_DistCpActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_DistCpActionExtension.md b/docs/src/site/markdown/DG_DistCpActionExtension.md
new file mode 100644
index 0000000..13c2a0a
--- /dev/null
+++ b/docs/src/site/markdown/DG_DistCpActionExtension.md
@@ -0,0 +1,220 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie DistCp Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## DistCp Action
+
+The `DistCp` action uses Hadoop distributed copy to copy files from one cluster to another or within the same cluster.
+
+**IMPORTANT:** The DistCp action may not work properly with all configurations (secure, insecure) in all versions
+of Hadoop. For example, distcp between two secure clusters is tested and works well. Same is true with two insecure
+clusters. In cases where a secure and insecure clusters are involved, distcp will not work.
+
+Both Hadoop clusters have to be configured with proxyuser for the Oozie process as explained
+[here](DG_QuickStart.html#HadoopProxyUser) on the Quick Start page.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="distcp-example">
+ <distcp xmlns="uri:oozie:distcp-action:1.0">
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode1}</name-node>
+ <arg>${nameNode1}/path/to/input.txt</arg>
+ <arg>${nameNode2}/path/to/output.txt</arg>
+ </distcp>
+ <ok to="end"/>
+ <error to="fail"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The first `arg` indicates the input and the second `arg` indicates the output. In the above example, the input is on `namenode1`
+and the output is on `namenode2`.
+
+**IMPORTANT:** If using the DistCp action between 2 secure clusters, the following property must be added to the `configuration` of
+the action:
+
+```
+<property>
+ <name>oozie.launcher.mapreduce.job.hdfs-servers</name>
+ <value>${nameNode1},${nameNode2}</value>
+</property>
+```
+
+The `DistCp` action is also commonly used to copy files within the same cluster. Cases where copying files within
+a directory to another directory or directories to target directory is supported. Example below will illustrate a
+copy within a cluster, notice the source and target `nameNode` is the same and use of `*` syntax is supported to
+represent only child files or directories within a source directory. For the sake of the example, `jobTracker` and `resourceManager`
+are synonymous.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="copy-example">
+ <distcp xmlns="uri:oozie:distcp-action:1.0">
+ <resource-manager>${resourceManager}</resource-manager>
+ <name-node>${nameNode}</name-node>
+ <arg>${nameNode}/path/to/source/*</arg>
+ <arg>${nameNode}/path/to/target/</arg>
+ </distcp>
+ <ok to="end"/>
+ <error to="fail"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+## Appendix, DistCp XML-Schema
+
+### AE.A Appendix A, DistCp XML-Schema
+
+#### DistCp Action Schema Version 1.0
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:distcp="uri:oozie:distcp-action:1.0" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:distcp-action:1.0">
+.
+ <xs:include schemaLocation="oozie-common-1.0.xsd"/>
+.
+ <xs:element name="distcp" type="distcp:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:choice>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="launcher" type="distcp:LAUNCHER" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### DistCp Action Schema Version 0.2
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:distcp="uri:oozie:distcp-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:distcp-action:0.2">
+.
+ <xs:element name="distcp" type="distcp:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="distcp:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="distcp:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### DistCp Action Schema Version 0.1
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:distcp="uri:oozie:distcp-action:0.1" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:distcp-action:0.1">
+.
+ <xs:element name="distcp" type="distcp:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="distcp:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="distcp:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_EmailActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_EmailActionExtension.md b/docs/src/site/markdown/DG_EmailActionExtension.md
new file mode 100644
index 0000000..1afcbb4
--- /dev/null
+++ b/docs/src/site/markdown/DG_EmailActionExtension.md
@@ -0,0 +1,136 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Email Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+<a name="EmailAction"></a>
+## 3.2.4 Email action
+
+The `email` action allows sending emails in Oozie from a workflow application. An email action must provide `to`
+addresses, `cc` addresses (optional), `bcc` addresses (optional), a `subject` and a `body`.
+Multiple recipients of an email can be provided as comma separated addresses.
+
+The email action is executed synchronously, and the workflow job will wait until the specified
+emails are sent before continuing to the next action.
+
+All values specified in the `email` action can be parameterized (templatized) using EL expressions.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1">
+ ...
+ <action name="[NODE-NAME]">
+ <email xmlns="uri:oozie:email-action:0.2">
+ <to>[COMMA-SEPARATED-TO-ADDRESSES]</to>
+ <cc>[COMMA-SEPARATED-CC-ADDRESSES]</cc> <!-- cc is optional -->
+ <bcc>[COMMA-SEPARATED-BCC-ADDRESSES]</bcc> <!-- bcc is optional -->
+ <subject>[SUBJECT]</subject>
+ <body>[BODY]</body>
+ <content_type>[CONTENT-TYPE]</content_type> <!-- content_type is optional -->
+ <attachment>[COMMA-SEPARATED-HDFS-FILE-PATHS]</attachment> <!-- attachment is optional -->
+ </email>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `to` and `cc` and `bcc` commands are used to specify recipients who should get the mail. Multiple email recipients
+can be provided using comma-separated values. Providing a `to` command is necessary, while the `cc` or `bcc` may
+optionally be used along.
+
+The `subject` and `body` commands are used to specify subject and body of the mail.
+From uri:oozie:email-action:0.2 one can also specify mail content type as <content_type>text/html</content_type>.
+"text/plain" is default.
+
+The `attachment` is used to attach a file(s) on HDFS to the mail. Multiple attachment can be provided using comma-separated values.
+Non fully qualified path is considered as a file on default HDFS. A local file cannot be attached.
+
+**Configuration**
+
+The `email` action requires some SMTP server configuration to be present (in oozie-site.xml). The following are the values
+it looks for:
+
+ * `oozie.email.smtp.host` - The host where the email action may find the SMTP server (localhost by default).
+ * `oozie.email.smtp.port` - The port to connect to for the SMTP server (25 by default).
+ * `oozie.email.from.address` - The from address to be used for mailing all emails (oozie@localhost by default).
+ * `oozie.email.smtp.auth` - Boolean property that toggles if authentication is to be done or not. (false by default).
+ * `oozie.email.smtp.starttls.enable` - Boolean property that toggles if use TLS communication or not. (false by default).
+ * `oozie.email.smtp.username` - If authentication is enabled, the username to login as (empty by default).
+ * `oozie.email.smtp.password` - If authentication is enabled, the username's password (empty by default).
+ * `oozie.email.attachment.enabled` - Boolean property that toggles if configured attachments are to be placed into the emails.
+ (false by default).
+ * `oozie.email.smtp.socket.timeout.ms` - The timeout to apply over all SMTP server socket operations (10000ms by default).
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
+ ...
+ <action name="an-email">
+ <email xmlns="uri:oozie:email-action:0.1">
+ <to>bob@initech.com,the.other.bob@initech.com</to>
+ <cc>will@initech.com</cc>
+ <bcc>yet.another.bob@initech.com</bcc>
+ <subject>Email notifications for ${wf:id()}</subject>
+ <body>The wf ${wf:id()} successfully completed.</body>
+ </email>
+ <ok to="myotherjob"/>
+ <error to="errorcleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+In the above example, an email is sent to 'bob', 'the.other.bob', 'will' (cc), yet.another.bob (bcc)
+with the subject and body both containing the workflow ID after substitution.
+
+## AE.A Appendix A, Email XML-Schema
+
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:email="uri:oozie:email-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:email-action:0.2">
+.
+ <xs:element name="email" type="email:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="to" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="cc" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="bcc" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="subject" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="body" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="content_type" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="attachment" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:sequence>
+ </xs:complexType>
+</xs:schema>
+```
+
+**GMail example to oozie-site.xml**
+
+
+```
+oozie.email.smtp.host=smtp.gmail.com
+oozie.email.smtp.port=587
+oozie.email.from.address=<some email address>
+oozie.email.smtp.auth=true
+oozie.email.smtp.starttls.enable=true
+oozie.email.smtp.username=<Gmail Id>
+oozie.email.smtp.password=<Gmail Pass>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_Examples.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_Examples.md b/docs/src/site/markdown/DG_Examples.md
new file mode 100644
index 0000000..ff33506
--- /dev/null
+++ b/docs/src/site/markdown/DG_Examples.md
@@ -0,0 +1,204 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Examples
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Command Line Examples
+
+### Setting Up the Examples
+
+Oozie examples are bundled within the Oozie distribution in the `oozie-examples.tar.gz` file.
+
+Expanding this file will create an `examples/` directory in the local file system.
+
+The `examples/` directory must be copied to the user HOME directory in HDFS:
+
+
+```
+$ hadoop fs -put examples examples
+```
+
+**NOTE:** If an examples directory already exists in HDFS, it must be deleted before copying it again. Otherwise files may not be
+copied.
+
+### Running the Examples
+
+For the Streaming and Pig example, the [Oozie Share Library](DG_QuickStart.html#OozieShareLib) must be installed in HDFS.
+
+Add Oozie `bin/` to the environment PATH.
+
+The examples assume the ResourceManager is `localhost:8032` and the NameNode is `hdfs://localhost:8020`. If the actual
+values are different, the job properties files in the examples directory must be edited to the correct values.
+
+The example applications are under the examples/app directory, one directory per example. The directory contains the
+application XML file (workflow, or workflow and coordinator), the `job.properties` file to submit the job and any JAR
+files the example may need.
+
+The inputs for all examples are in the `examples/input-data/` directory.
+
+The examples create output under the `examples/output-data/${EXAMPLE_NAME}` directory.
+
+**Note**: The `job.properties` file needs to be a local file during submissions, and not a HDFS path.
+
+**How to run an example application:**
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
+.
+job: 14-20090525161321-oozie-tucu
+```
+
+Check the workflow job status:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-tucu
+.
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+Workflow Name : map-reduce-wf
+App Path : hdfs://localhost:8020/user/tucu/examples/apps/map-reduce
+Status : SUCCEEDED
+Run : 0
+User : tucu
+Group : users
+Created : 2009-05-26 05:01 +0000
+Started : 2009-05-26 05:01 +0000
+Ended : 2009-05-26 05:01 +0000
+Actions
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+Action Name Type Status Transition External Id External Status Error Code Start Time End Time
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+mr-node map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+```
+
+To check the workflow job status via the Oozie web console, with a browser go to `http://localhost:11000/oozie`.
+
+To avoid having to provide the `-oozie` option with the Oozie URL with every `oozie` command, set `OOZIE_URL` env
+variable to the Oozie URL in the shell environment. For example:
+
+
+```
+$ export OOZIE_URL="http://localhost:11000/oozie"
+$
+$ oozie job -info 14-20090525161321-oozie-tucu
+```
+
+## Java API Example
+
+Oozie provides a [Java Client API](./apidocs/org/org/apache/oozie/client/package-summary.html) that simplifies
+integrating Oozie with Java applications. This Java Client API is a convenience API to interact with Oozie Web-Services
+API.
+
+The following code snippet shows how to submit an Oozie job using the Java Client API.
+
+
+```
+import org.apache.oozie.client.OozieClient;
+import org.apache.oozie.client.WorkflowJob;
+.
+import java.util.Properties;
+.
+ ...
+.
+ // get a OozieClient for local Oozie
+ OozieClient wc = new OozieClient("http://bar:11000/oozie");
+.
+ // create a workflow job configuration and set the workflow application path
+ Properties conf = wc.createConfiguration();
+ conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app");
+.
+ // setting workflow parameters
+ conf.setProperty("resourceManager", "foo:8032");
+ conf.setProperty("inputDir", "/usr/tucu/inputdir");
+ conf.setProperty("outputDir", "/usr/tucu/outputdir");
+ ...
+.
+ // submit and start the workflow job
+ String jobId = wc.run(conf);
+ System.out.println("Workflow job submitted");
+.
+ // wait until the workflow job finishes printing the status every 10 seconds
+ while (wc.getJobInfo(jobId).getStatus() == Workflow.Status.RUNNING) {
+ System.out.println("Workflow job running ...");
+ Thread.sleep(10 * 1000);
+ }
+.
+ // print the final status of the workflow job
+ System.out.println("Workflow job completed ...");
+ System.out.println(wf.getJobInfo(jobId));
+ ...
+```
+
+## Local Oozie Example
+
+Oozie provides an embedded Oozie implementation, [LocalOozie](./apidocs/org/apache/oozie/local/LocalOozie.html) ,
+which is useful for development, debugging and testing of workflow applications within the convenience of an IDE.
+
+The code snippet below shows the usage of the `LocalOozie` class. All the interaction with Oozie is done using Oozie
+ `OozieClient` Java API, as shown in the previous section.
+
+The examples bundled with Oozie include the complete and running class, `LocalOozieExample` from where this snippet was
+taken.
+
+
+```
+import org.apache.oozie.local.LocalOozie;
+import org.apache.oozie.client.OozieClient;
+import org.apache.oozie.client.WorkflowJob;
+.
+import java.util.Properties;
+.
+ ...
+ // start local Oozie
+ LocalOozie.start();
+.
+ // get a OozieClient for local Oozie
+ OozieClient wc = LocalOozie.getClient();
+.
+ // create a workflow job configuration and set the workflow application path
+ Properties conf = wc.createConfiguration();
+ conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app");
+.
+ // setting workflow parameters
+ conf.setProperty("resourceManager", "foo:8032");
+ conf.setProperty("inputDir", "/usr/tucu/inputdir");
+ conf.setProperty("outputDir", "/usr/tucu/outputdir");
+ ...
+.
+ // submit and start the workflow job
+ String jobId = wc.run(conf);
+ System.out.println("Workflow job submitted");
+.
+ // wait until the workflow job finishes printing the status every 10 seconds
+ while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
+ System.out.println("Workflow job running ...");
+ Thread.sleep(10 * 1000);
+ }
+.
+ // print the final status of the workflow job
+ System.out.println("Workflow job completed ...");
+ System.out.println(wc.getJobInfo(jobId));
+.
+ // stop local Oozie
+ LocalOozie.stop();
+ ...
+```
+
+Also asynchronous actions like FS action can be used / tested using `LocalOozie` / `OozieClient` API. Please see the module
+`oozie-mini` for details like `fs-decision.xml` workflow example.
+
+
+## Fluent Job API Examples
+
+There are some elaborate examples how to use the [Fluent Job API](DG_FluentJobAPI.html), under `examples/fluentjob/`. There are two
+simple examples covered under [Fluent Job API :: A Simple Example](DG_FluentJobAPI.html#A_Simple_Example) and
+[Fluent Job API :: A More Verbose Example](DG_FluentJobAPI.html#A_More_Verbose_Example).
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_FluentJobAPI.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_FluentJobAPI.md b/docs/src/site/markdown/DG_FluentJobAPI.md
new file mode 100644
index 0000000..bd36517
--- /dev/null
+++ b/docs/src/site/markdown/DG_FluentJobAPI.md
@@ -0,0 +1,384 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Fluent Job API
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Introduction
+
+Oozie is a mature workflow scheduler system. XML is the standard way of defining workflow, coordinator, or bundle jobs. For users
+who prefer an alternative, the Fluent Job API provides a Java interface instead.
+
+### Motivation
+
+Prior to Oozie 5.1.0, the following ways were available to submit a workflow, coordinator, or bundle job: through Oozie CLI or via
+HTTP submit a generic workflow, coordinator, or bundle job, or submit a Pig, Hive, Sqoop, or MapReduce workflow job.
+
+As the generic way goes, the user has to have uploaded a workflow, coordinator, or bundle XML and all necessary dependencies like
+scripts, JAR or ZIP files, to HDFS beforehand, as well as have a `job.properties` file at command line and / or provide any
+missing parameters as part of the command.
+
+As the specific Pig, Hive, or Sqoop ways go, the user can provide all necessary parameters as part of the command issued. A
+ `workflow.xml` file will be generated with all the necessary details and stored to HDFS so that Oozie can grab it. Note that
+dependencies have to be uploaded to HDFS beforehand as well.
+
+There are some usability problems by using the XML job definition. XML is not an ideal way to express dependencies and a directed
+acyclic graph (DAG). We have to define a control flow, that is, which action follows the actual one. It's also necessary to build
+the whole control flow up front as XML is a declarative language that doesn't allow for dynamic evaluation. We have to define also
+boilerplate actions like start and end - those are present in every Oozie workflow, still need to explicitly define these.
+
+Apart from boilerplate actions, all the transitions between actions have also to be defined and taken care of. Furthermore, multiple
+similar actions cannot inherit common properties from each other. Again, the reason being workflows are defined in XML.
+
+Fork and join actions have to be defined in pairs, that is, there shouldn't be defined a join those incoming actions do not share
+the same ancestor fork. Such situations would result still in a DAG, but Oozie doesn't currently allow that. Note that with Fluent
+Job API new dependencies are introduced automatically when the DAG represented by API code couldn't have been expressed as
+fork / join pairs automatically.
+
+Either way, there were no programmatic ways to define workflow jobs. That doesn't mean users could not generate XML themselves -
+actually this is something HUE's Oozie UI also tries to target.
+
+### Goals
+
+Fluent Job API aims to solve following from the user's perspective. It provides a Java API instead of declarative XML to define
+workflows. It defines dependencies across actions as opposed to defining a control flow. This is how data engineers and data
+scientists think. It eliminates all boilerplate actions and transitions. Only the necessary bits should be defined.
+
+Multiple similar actions can inherit from each other. In fact, since Fluent Job API is programmatic, it's possible to generate
+actions or even workflows using conditional, iterative, or recursive structures.
+
+Fluent Job API is backwards compatible with workflows defined as XML. That is, it should also be possible to have a Fluent Job API
+workflow rendered as XML, as well as coexist XML based and Fluent Job API based workflows in the same Oozie installation at the same
+time all workflow action types. When XSDs change, as few manual steps are necessary as possible both on API internal and public
+side.
+
+### Non-goals
+
+The following points are not targeted for the initial release of Fluent Job API with Oozie 5.1.0. It doesn't provide API in any
+language other than Java. It doesn't provide a REPL. It doesn't allow for dynamic action instantiation depending on e.g. conditional
+logic. That is, using the API users still have to implement the whole workflow generation logic in advance.
+
+It has no support for programmatic coordinators and bundles, or even EL expressions created by API builders. Note that EL
+expressions for workflows can now be expressed the way these are used in XML workflow definitions, as strings in the right places.
+
+At the moment only the transformation from Fluent Job API to workflow definition is present. The other direction, from workflow
+definition to Fluent Job API JAR artifact, though sensible, is not supported.
+
+It's based only on latest XSDs. Older XSD versions, as well as conversion between XSD versions are not supported. Also no support
+for user-supplied custom actions / XSDs.
+
+Most of the non-goals may be targeted as enhancements of the Fluent Job API for future Oozie releases.
+
+### Approach
+
+When using the Fluent Job API, the following points are different from the XML jobs definition. Instead of control flow (successor)
+definition, the user can define dependencies (parents of an action).
+
+All boilerplate (start, end, ...) has been eliminated, only nodes having useful actions have to be defined.
+
+Control flow and necessary boilerplate are generated automatically by keeping user defined dependencies, and possibly introducing
+new dependencies to keep Oozie workflow format of nested fork / join pairs. Note that not every dependency DAG can be expressed in
+the Oozie workflow format. When this is not possible, user is notified at build time.
+
+## How To Use
+
+### A Simple Example
+
+The simplest thing to create using the Oozie Fluent Job API is a workflow consisting of only one action. Let's see how it goes, step
+by step.
+
+First, put the project `org.apache.oozie:oozie-fluent-job-api` to the build path. In case of a Maven managed build, create a new
+Maven project and declare a Maven dependency to `org.apache.oozie:oozie-fluent-job-api`.
+
+Then, create a class that `implements WorkflowFactory` and implement the method `WorkflowFactory#create()`. inside that method,
+create a `ShellAction` using `ShellActionBuilder`, fill in some attributes then create a `Workflow` using `WorkflowBuilder` using
+the `ShellAction` just built. Return the `Workflow`.
+
+Compile a Fluent Job API jar that has the `Main-Class` attribute set to the `WorkflowFactory` subclass just created,
+e.g. `shell-workflow.jar`.
+
+Moving on, [check via command line](DG_CommandLineTool.html#Checking_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file) that
+the compiled API JAR file is valid.
+
+As a finishing touch,
+[run via command line](DG_CommandLineTool.html#Running_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file) the Fluent Job API
+workflow.
+
+**For reference, a simplistic API JAR example consisting of a `Workflow` having only one `ShellAction`:**
+
+```
+public class MyFirstWorkflowFactory implements WorkflowFactory {
+.
+ @Override
+ public Workflow create() {
+ final ShellAction shellAction = ShellActionBuilder.create()
+ .withName("shell-action")
+ .withResourceManager("${resourceManager}")
+ .withNameNode("${nameNode}")
+ .withConfigProperty("mapred.job.queue.name", "${queueName}")
+ .withExecutable("echo")
+ .withArgument("my_output=Hello Oozie")
+ .withCaptureOutput(true)
+ .build();
+.
+ final Workflow shellWorkflow = new WorkflowBuilder()
+ .withName("shell-workflow")
+ .withDagContainingNode(shellAction).build();
+.
+ return shellWorkflow;
+ }
+}
+```
+
+**After check, the generated workflow XML looks like this:**
+
+```
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" xmlns:shell="uri:oozie:shell-action:1.0" name="shell-workflow">
+.
+ <workflow:start to="parent"/>
+.
+ <workflow:kill name="kill">
+ <workflow:message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</workflow:message>
+ </workflow:kill>
+.
+ <workflow:action name="shell-action">
+ <shell:shell>
+ <shell:resource-manager>${resourceManager}</shell:resource-manager>
+ <shell:name-node>${nameNode}</shell:name-node>
+ <shell:configuration>
+ <shell:property>
+ <shell:name>mapred.job.queue.name</shell:name>
+ <shell:value>${queueName}</shell:value>
+ </shell:property>
+ </shell:configuration>
+ <shell:exec>echo</shell:exec>
+ <shell:argument>my_output=Hello Oozie</shell:argument>
+ <shell:capture-output/>
+ </shell:shell>
+ <workflow:ok to="end"/>
+ <workflow:error to="kill"/>
+ </workflow:action>
+.
+ <workflow:end name="end"/>
+.
+</workflow:workflow-app>
+```
+
+
+### A More Verbose Example
+
+**Error handling**
+
+If you would like to provide some error handling in case of action failure, you should add an `ErrorHandler` to the `Node`
+representing the action. The error handler action will be added as the `"error-transition"` of the original action in the generated
+Oozie workflow XML. Both the `"ok-transition"` and the `"error-transition"` of the error handler action itself will lead to an
+autogenerated kill node.
+
+**Here you find an example consisting of a `Workflow` having three `ShellAction`s, an error handler `EmailAction`, and one `decision`
+to sort out which way to go:**
+
+```
+public class MySecondWorkflowFactory implements WorkflowFactory {
+.
+ @Override
+ public Workflow create() {
+ final ShellAction parent = ShellActionBuilder.create()
+ .withName("parent")
+ .withResourceManager("${resourceManager}")
+ .withNameNode("${nameNode}")
+ .withConfigProperty("mapred.job.queue.name", "${queueName}")
+ .withExecutable("echo")
+ .withArgument("my_output=Hello Oozie")
+ .withCaptureOutput(true)
+ .withErrorHandler(ErrorHandler.buildAsErrorHandler(EmailActionBuilder.create()
+ .withName("email-on-error")
+ .withRecipient("somebody@apache.org")
+ .withSubject("Workflow error")
+ .withBody("Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]")))
+ .build();
+.
+ ShellActionBuilder.createFromExistingAction(parent)
+ .withName("happy-path")
+ .withParentWithCondition(parent, "${wf:actionData('parent')['my_output'] eq 'Hello Oozie'}")
+ .withoutArgument("my_output=Hello Oozie")
+ .withArgument("Happy path")
+ .withCaptureOutput(null)
+ .build();
+.
+ ShellActionBuilder.createFromExistingAction(parent)
+ .withName("sad-path")
+ .withParentDefaultConditional(parent)
+ .withArgument("Sad path")
+ .build();
+.
+ final Workflow workflow = new WorkflowBuilder()
+ .withName("shell-example")
+ .withDagContainingNode(parent).build();
+.
+ return workflow;
+ }
+}
+```
+
+**After check, the generated workflow XML looks like this:**
+
+```
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<workflow:workflow-app ... name="shell-example">
+.
+ <workflow:start to="parent"/>
+.
+ <workflow:kill name="kill">
+ <workflow:message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</workflow:message>
+ </workflow:kill>
+.
+ <workflow:action name="email-on-error">
+ <email:email>
+ <email:to>somebody@apache.org</email:to>
+ <email:subject>Workflow error</email:subject>
+ <email:body>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</email:body>
+ </email:email>
+ <workflow:ok to="kill"/>
+ <workflow:error to="kill"/>
+ </workflow:action>
+.
+ <workflow:action name="parent">
+ <shell:shell>
+ <shell:resource-manager>${resourceManager}</shell:resource-manager>
+ <shell:name-node>${nameNode}</shell:name-node>
+ <shell:configuration>
+ <shell:property>
+ <shell:name>mapred.job.queue.name</shell:name>
+ <shell:value>${queueName}</shell:value>
+ </shell:property>
+ </shell:configuration>
+ <shell:exec>echo</shell:exec>
+ <shell:argument>my_output=Hello Oozie</shell:argument>
+ <shell:capture-output/>
+ </shell:shell>
+ <workflow:ok to="decision1"/>
+ <workflow:error to="email-on-error"/>
+ </workflow:action>
+.
+ <workflow:decision name="decision1">
+ <workflow:switch>
+ <workflow:case to="happy-path">${wf:actionData('parent')['my_output'] eq 'Hello Oozie'}</workflow:case>
+ <workflow:default to="sad-path"/>
+ </workflow:switch>
+ </workflow:decision>
+.
+ <workflow:action name="happy-path">
+ <shell:shell>
+ <shell:resource-manager>${resourceManager}</shell:resource-manager>
+ <shell:name-node>${nameNode}</shell:name-node>
+ <shell:configuration>
+ <shell:property>
+ <shell:name>mapred.job.queue.name</shell:name>
+ <shell:value>${queueName}</shell:value>
+ </shell:property>
+ </shell:configuration>
+ <shell:exec>echo</shell:exec>
+ <shell:argument>Happy path</shell:argument>
+ </shell:shell>
+ <workflow:ok to="end"/>
+ <workflow:error to="email-on-error"/>
+ </workflow:action>
+.
+ <workflow:action name="sad-path">
+ <shell:shell>
+ <shell:resource-manager>${resourceManager}</shell:resource-manager>
+ <shell:name-node>${nameNode}</shell:name-node>
+ <shell:configuration>
+ <shell:property>
+ <shell:name>mapred.job.queue.name</shell:name>
+ <shell:value>${queueName}</shell:value>
+ </shell:property>
+ </shell:configuration>
+ <shell:exec>echo</shell:exec>
+ <shell:argument>my_output=Hello Oozie</shell:argument>
+ <shell:argument>Sad path</shell:argument>
+ <shell:capture-output/>
+ </shell:shell>
+ <workflow:ok to="end"/>
+ <workflow:error to="email-on-error"/>
+ </workflow:action>
+.
+ <workflow:end name="end"/>
+.
+</workflow:workflow-app>
+```
+
+### Runtime Limitations
+
+Even if Fluent Job API tries to abstract away the task of assembly job descriptor XML files, there are some runtime
+limitations apart from the [non-goals section](DG_FluentJobAPI.html#Non-goals). All such limitations are based on the current
+implementations and subject to further improvements and fixes.
+
+There is only one `kill` possibility in every `workflow`. That is, there can be defined only one `action` to be executed just before
+any other `action` turns to be `kill`ed. Furthermore, `kill` goes to `end` directly. That means, there cannot be defined an
+intricate network of `kill` nodes, cascading sometimes to other `action` nodes, avoiding going to `end` in the first place.
+
+There are places where `decision` node generation fails, throwing an `Exception`. The problem is that during the transformation,
+Fluent Job API reaches a state where there is a `fork` that transitions to two `decision` nodes, which in turn split into two paths
+each. One of the paths from the first `decision` joins a path from the other `decision`, but the remaining conditional paths never
+meet. Therefore, not all paths originating from the `fork` converge to the same `join`.
+
+## Appendixes
+
+### AE.A Appendix A, API JAR format
+
+It's kept simple - all the necessary Java class files that are needed are packed into a JAR file, that has a `META-INF/MANIFEST.MF`
+with a single entry having the `Main-Class` attribute set to the fully qualified name of the entry class, the one that
+`implements WorkflowFactory`:
+
+```
+Main-Class: org.apache.oozie.jobs.api.factory.MyFirstWorkflowFactory
+```
+
+**An example of the command line assembly of such an API JAR:**
+
+```
+jar cfe simple-workflow.jar org.apache.oozie.fluentjob.api.factory.MyFirstWorkflowFactory \
+-C /Users/forsage/Workspace/oozie/fluent-job/fluent-job-api/target/classes \
+org/apache/oozie/jobs/api/factory/MyFirstWorkflowFactory.class
+```
+
+### AE.B Appendix B, Some Useful Builder classes
+
+For a complete list of `Builder` classes, please have a look at `oozie-fluent-job-api` artifact's following packages:
+
+ * `org.apache.oozie.fluentjob.api.action` - `ActionBuilder` classes
+ * `org.apache.oozie.fluentjob.api.factory` - the single entry point, `WorkflowFactory` is here
+ * `org.apache.oozie.fluentjob.api.workflow` - workflow related `Builder` classes
+
+On examples how to use these please see `oozie-examples` artifact's `org.apache.oozie.example.fluentjob` package.
+
+### AE.C Appendix C, How To Extend
+
+Sometimes there are new XSD versions of an existing custom or core workflow action, sometimes it's a new custom workflow action that
+gets introduced. In any case, Fluent Job API needs to keep up with the changes.
+
+Here are the steps needed:
+
+ * in `fluent-job-api/pom.xml` extend or modify `jaxb2-maven-plugin` section `sources` by a new `source`
+ * in `fluent-job-api/src/main/xjb/bindings.xml` extend by a new or modify an existing `jaxb:bindings`
+ * in `fluent-job-api`, `org.apache.oozie.fluentjob.api.mapping` package, introduce a new or modify an existing `DozerConverter`
+ * in `dozer_config.xml`, introduce a new or modify an existing `converter` inside `custom-converters`
+ * in `fluent-job-api`, `org.apache.oozie.fluentjob.api.action`, introduce a new `Action` and a new `Builder`
+ * write new / modify existing relevant unit and integration tests
+
+### AE.D Appendix D, API compatibility guarantees
+
+Fluent Job API is available beginning version 5.1.0. It's marked `@InterfaceAudience.Private` (intended for use in Oozie itself) and
+`@InterfaceStability.Unstable` (no stability guarantees are provided across any level of release granularity) to indicate that for
+the next few minor releases it's bound to change a lot.
+
+Beginning from around 5.4.0 planning the next phase, `@InterfaceStability.Evolving` (compatibility breaking only between minors),
+and a few minor releases later, `@InterfaceAudience.Public` (safe to use outside of Oozie).
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_HCatalogIntegration.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_HCatalogIntegration.md b/docs/src/site/markdown/DG_HCatalogIntegration.md
new file mode 100644
index 0000000..5c592e8
--- /dev/null
+++ b/docs/src/site/markdown/DG_HCatalogIntegration.md
@@ -0,0 +1,147 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# HCatalog Integration (Since Oozie 4.x)
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## HCatalog Overview
+HCatalog is a table and storage management layer for Hadoop that enables users with different data processing
+tools - Pig, MapReduce, and Hive - to more easily read and write data on the grid. HCatalog's table abstraction presents
+users with a relational view of data in the Hadoop distributed file system (HDFS).
+
+Read [HCatalog Documentation](http://incubator.apache.org/hcatalog/docs/r0.5.0/index.html) to know more about HCatalog.
+Working with HCatalog using pig is detailed in
+[HCatLoader and HCatStorer](http://incubator.apache.org/hcatalog/docs/r0.5.0/loadstore.html).
+Working with HCatalog using MapReduce directly is detailed in
+[HCatInputFormat and HCatOutputFormat](http://incubator.apache.org/hcatalog/docs/r0.5.0/inputoutput.html).
+
+### HCatalog notifications
+ HCatalog provides notifications through a JMS provider like ActiveMQ when a new partition is added to a table in the
+database. This allows applications to consume those events and schedule the work that depends on them. In case of Oozie,
+the notifications are used to determine the availability of HCatalog partitions defined as data dependencies in the
+Coordinator and trigger workflows.
+
+Read [HCatalog Notification](http://incubator.apache.org/hcatalog/docs/r0.5.0/notification.html) to know more about
+notifications in HCatalog.
+
+## Oozie HCatalog Integration
+ Oozie's Coordinators so far have been supporting HDFS directories as a input data dependency. When a HDFS URI
+template is specified as a dataset and input events are defined in Coordinator for the dataset, Oozie performs data
+availability checks by polling the HDFS directory URIs resolved based on the nominal time. When all the data
+dependencies are met, the Coordinator's workflow is triggered which then consumes the available HDFS data.
+
+With addition of HCatalog support, Coordinators also support specifying a set of HCatalog tables or table partitions as a dataset.
+The workflow is triggered when the HCatalog table partitions are available and the workflow actions can then read the
+partition data. A mix of HDFS and HCatalog dependencies can be specified as input data dependencies.
+Similar to HDFS directories, HCatalog table partitions can also be specified as output dataset events.
+
+With HDFS data dependencies, Oozie has to poll HDFS every time to determine the availability of a directory.
+If the HCatalog server is configured to publish partition availability notifications to a JMS provider, Oozie can be
+configured to subscribe to it and trigger jobs immediately. This pub-sub model reduces pressure on Namenode and also
+cuts down on delays caused by polling intervals.
+
+In the absence of a message bus in the deployment, Oozie will always
+poll the HCatalog server directly for partition availability with the same frequency as the HDFS polling. Even when
+subscribed to notifications, Oozie falls back to polling HCatalog server for partitions that were available before the
+coordinator action was materialized and to deal with missed notifications due to system downtimes. The frequency of the
+fallback polling is usually lower than the constant polling. Defaults are 10 minutes and 1 minute respectively.
+
+
+### Oozie Server Configuration
+ Refer to [HCatalog Configuration](AG_Install.html#HCatalog_Configuration) section of [Oozie Install](AG_Install.html)
+documentation for the Oozie server side configuration required to support HCatalog table partitions as a data dependency.
+
+### HCatalog URI Format
+
+Oozie supports specifying HCatalog partitions as a data dependency through a URI notation. The HCatalog partition URI is
+used to identify a set of table partitions: `hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US`
+
+The format to specify a HCatalog table URI is:
+
+hcat://[metastore server]:[port]/[database name]/[table name]
+
+The format to specify a HCatalog table partition URI is:
+
+hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...
+
+For example,
+
+```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hcat://myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA
+ </uri-template>
+ </dataset>
+```
+
+Post Oozie-4.3.0 release, Oozie also supports the multiple HCatalog servers in the URI. Each of the server needs to be
+separated by single comma (,).
+
+The format to specify a HCatalog table partition URI with multiple HCatalog server is:
+
+hcat://[metastore_server]:[port],[metastore_server]:[port]/[database_name]/[table_name]/[partkey1]=[value];[partkey2]=[value];...
+
+For example,
+
+```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hcat://myhcatmetastore:9080,myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA
+ </uri-template>
+ </dataset>
+```
+
+The regex for parsing the multiple HCatalog URI is exposed via oozie-site.xml, So Users can modify if there is any
+requirement. Key for the regex is: `oozie.hcat.uri.regex.pattern`
+
+For example, following has multiple HCatalog URI with multiple HCatalog servers. To understand this, Oozie will split them into
+two HCatalog URIs. For splitting the URIs, above mentioned regex is used.
+
+`hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us,scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us`
+
+After split: (This is internal Oozie mechanism)
+
+`hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us`
+
+`scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us`
+
+<a name="HCatalogLibraries"></a>
+### HCatalog Libraries
+
+A workflow action interacting with HCatalog requires the following jars in the classpath:
+hcatalog-core.jar, hcatalog-pig-adapter.jar, webhcat-java-client.jar, hive-common.jar, hive-exec.jar,
+hive-metastore.jar, hive-serde.jar and libfb303.jar.
+hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath. The correct
+version of HCatalog and hive jars should be placed in classpath based on the version of HCatalog installed on the cluster.
+
+The jars can be added to the classpath of the action using one of the below ways.
+
+ * You can place the jars and hive-site.xml in the system shared library. The shared library for a pig, hive or java action can be overridden to include hcatalog shared libraries along with the action's shared library. Refer to [Shared Libraries](WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3) for more information. The oozie-sharelib-[version].tar.gz in the oozie distribution bundles the required HCatalog jars in a hcatalog sharelib. If using a different version of HCatalog than the one bundled in the sharelib, copy the required HCatalog jars from such version into the sharelib.
+ * You can place the jars and hive-site.xml in the workflow application lib/ path.
+ * You can specify the location of the jar files in `archive` tag and the hive-site.xml in `file` tag in the corresponding pig, hive or java action.
+
+### Coordinator
+
+Refer to [Coordinator Functional Specification](CoordinatorFunctionalSpec.html) for more information about
+
+ * how to specify HCatalog partitions as a data dependency using input dataset events
+ * how to specify HCatalog partitions as output dataset events
+ * the various EL functions available to work with HCatalog dataset events and how to use them to access HCatalog partitions in pig, hive or java actions in a workflow.
+
+### Workflow
+Refer to [Workflow Functional Specification](WorkflowFunctionalSpec.html) for more information about
+
+ * how to drop HCatalog table/partitions in the prepare block of a action
+ * the HCatalog EL functions available to use in workflows
+
+Refer to [Action Authentication](DG_ActionAuthentication.html) for more information about
+
+ * how to access a secure HCatalog from any action (e.g. hive, pig, etc) in a workflow
+
+### Known Issues
+ * When rerunning a coordinator action without specifying -nocleanup option if the 'output-event' are hdfs directories, then they are deleted. But if the 'output-event' is a hcatalog partition, currently the partition is not dropped.
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_Hive2ActionExtension.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_Hive2ActionExtension.md b/docs/src/site/markdown/DG_Hive2ActionExtension.md
new file mode 100644
index 0000000..d81ed02
--- /dev/null
+++ b/docs/src/site/markdown/DG_Hive2ActionExtension.md
@@ -0,0 +1,315 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Hive 2 Action Extension
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Hive 2 Action
+
+The `hive2` action runs Beeline to connect to Hive Server 2.
+
+The workflow job will wait until the Hive Server 2 job completes before
+continuing to the next action.
+
+To run the Hive Server 2 job, you have to configure the `hive2` action with the `resource-manager`, `name-node`, `jdbc-url`,
+ `password` elements, and either Hive's `script` or `query` element, as well as the necessary parameters and configuration.
+
+A `hive2` action can be configured to create or delete HDFS directories
+before starting the Hive Server 2 job.
+
+Oozie EL expressions can be used in the inline configuration. Property
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
+
+As with Hadoop `map-reduce` jobs, it is possible to add files and
+archives in order to make them available to Beeline. Refer to the
+[Adding Files and Archives for the Job](WorkflowFunctionalSpec.html#FilesArchives)
+section for more information about this feature.
+
+Oozie Hive 2 action supports Hive scripts with parameter variables, their
+syntax is `${VARIABLES}`.
+
+**Syntax:**
+
+
+```
+<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="[NODE-NAME]">
+ <hive2 xmlns="uri:oozie:hive2-action:1.0">
+ <resource-manager>[RESOURCE-MANAGER]</resource-manager>
+ <name-node>[NAME-NODE]</name-node>
+ <prepare>
+ <delete path="[PATH]"/>
+ ...
+ <mkdir path="[PATH]"/>
+ ...
+ </prepare>
+ <job-xml>[HIVE SETTINGS FILE]</job-xml>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ <jdbc-url>[jdbc:hive2://HOST:10000/default]</jdbc-url>
+ <password>[PASS]</password>
+ <script>[HIVE-SCRIPT]</script>
+ <param>[PARAM-VALUE]</param>
+ ...
+ <param>[PARAM-VALUE]</param>
+ <argument>[ARG-VALUE]</argument>
+ ...
+ <argument>[ARG-VALUE]</argument>
+ <file>[FILE-PATH]</file>
+ ...
+ <archive>[FILE-PATH]</archive>
+ ...
+ </hive2>
+ <ok to="[NODE-NAME]"/>
+ <error to="[NODE-NAME]"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
+
+The `job-xml` element, if present, specifies a file containing configuration
+for Beeline. Multiple `job-xml` elements are allowed in order to specify multiple `job.xml` files.
+
+The `configuration` element, if present, contains configuration
+properties that are passed to the Beeline job.
+
+The `jdbc-url` element must contain the JDBC URL for the Hive Server 2. Beeline will use this to know where to connect to.
+
+The `password` element must contain the password of the current user. However, the `password` is only used if Hive Server 2 is
+backed by something requiring a password (e.g. LDAP); non-secured Hive Server 2 or Kerberized Hive Server 2 don't require a password
+so in those cases the `password` is ignored and can be omitted from the action XML. It is up to the user to ensure that a password
+is specified when required.
+
+The `script` element must contain the path of the Hive script to
+execute. The Hive script can be templatized with variables of the form
+`${VARIABLE}`. The values of these variables can then be specified
+using the `params` element.
+
+The `query` element available from uri:oozie:hive2-action:0.2, can be used instead of the `script` element. It allows for embedding
+queries within the `worklfow.xml` directly. Similar to the `script` element, it also allows for the templatization of variables
+in the form `${VARIABLE}`.
+
+The `params` element, if present, contains parameters to be passed to
+the Hive script.
+
+The `argument` element, if present, contains arguments to be passed as-is to Beeline.
+
+All the above elements can be parameterized (templatized) using EL
+expressions.
+
+**Example:**
+
+
+```
+<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
+ ...
+ <action name="my-hive2-action">
+ <hive2 xmlns="uri:oozie:hive2-action:1.0">
+ <resource-manager>foo:8032</resource-manager>
+ <name-node>bar:8020</name-node>
+ <prepare>
+ <delete path="${jobOutput}"/>
+ </prepare>
+ <configuration>
+ <property>
+ <name>mapred.compress.map.output</name>
+ <value>true</value>
+ </property>
+ </configuration>
+ <jdbc-url>jdbc:hive2://localhost:10000/default</jdbc-url>
+ <password>foo</password>
+ <script>myscript.q</script>
+ <param>InputDir=/home/rkanter/input-data</param>
+ <param>OutputDir=${jobOutput}</param>
+ </hive2>
+ <ok to="my-other-action"/>
+ <error to="error-cleanup"/>
+ </action>
+ ...
+</workflow-app>
+```
+
+
+### Security
+
+As mentioned above, `password` is only used in cases where Hive Server 2 is backed by something requiring a password (e.g. LDAP).
+Non-secured Hive Server 2 and Kerberized Hive Server 2 don't require a password so in these cases it can be omitted.
+
+## Appendix, Hive 2 XML-Schema
+
+### AE.A Appendix A, Hive 2 XML-Schema
+
+#### Hive 2 Action Schema Version 1.0
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive2="uri:oozie:hive2-action:1.0" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive2-action:1.0">
+.
+ <xs:include schemaLocation="oozie-common-1.0.xsd"/>
+.
+ <xs:element name="hive2" type="hive2:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:choice>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="launcher" type="hive2:LAUNCHER" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:choice minOccurs="1" maxOccurs="1">
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Hive 2 Action Schema Version 0.2
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive2="uri:oozie:hive2-action:0.2" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive2-action:0.2">
+.
+ <xs:element name="hive2" type="hive2:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:choice minOccurs="1" maxOccurs="1">
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ </xs:choice>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="hive2:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="hive2:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+#### Hive 2 Action Schema Version 0.1
+
+```
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:hive2="uri:oozie:hive2-action:0.1" elementFormDefault="qualified"
+ targetNamespace="uri:oozie:hive2-action:0.1">
+.
+ <xs:element name="hive2" type="hive2:ACTION"/>
+.
+ <xs:complexType name="ACTION">
+ <xs:sequence>
+ <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
+ <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="CONFIGURATION">
+ <xs:sequence>
+ <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
+ <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="PREPARE">
+ <xs:sequence>
+ <xs:element name="delete" type="hive2:DELETE" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element name="mkdir" type="hive2:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+.
+ <xs:complexType name="DELETE">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+ <xs:complexType name="MKDIR">
+ <xs:attribute name="path" type="xs:string" use="required"/>
+ </xs:complexType>
+.
+</xs:schema>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
[17/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/DG_CommandLineTool.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_CommandLineTool.md b/docs/src/site/markdown/DG_CommandLineTool.md
new file mode 100644
index 0000000..51382c7
--- /dev/null
+++ b/docs/src/site/markdown/DG_CommandLineTool.md
@@ -0,0 +1,2295 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Command Line Interface Utilities
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Introduction
+
+Oozie provides a command line utility, `oozie`, to perform job and admin tasks. All operations are done via
+sub-commands of the `oozie` CLI.
+
+The `oozie` CLI interacts with Oozie via its WS API.
+
+## Oozie Command Line Usage
+
+
+```
+usage:
+ the env variable 'OOZIE_URL' is used as default value for the '-oozie' option
+ the env variable 'OOZIE_TIMEZONE' is used as default value for the '-timezone' option
+ the env variable 'OOZIE_AUTH' is used as default value for the '-auth' option
+ custom headers for Oozie web services can be specified using '-Dheader:NAME=VALUE'
+```
+
+### Oozie basic commands
+
+```
+
+oozie help : display usage
+
+oozie version : show client version
+
+```
+
+### Oozie job operation commands
+
+```
+
+oozie job <OPTIONS> : job operations
+ -action <arg> coordinator rerun/kill on action ids (requires -rerun/-kill);
+ coordinator log retrieval on action ids (requires -log)
+ -allruns Get workflow jobs corresponding to a coordinator action
+ including all the reruns
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -change <arg> change a coordinator or bundle job
+ -config <arg> job configuration file '.xml' or '.properties'
+ -configcontent <arg> job configuration
+ -coordinator <arg> bundle rerun on coordinator names (requires -rerun)
+ -D <property=value> set/override value for given property
+ -date <arg> coordinator/bundle rerun on action dates (requires -rerun);
+ coordinator log retrieval on action dates (requires -log)
+ -debug Use debug mode to see debugging statements on stdout
+ -definition <arg> job definition
+ -diff <arg> Show diff of the new coord definition and properties with the
+ existing one (default true)
+ -doas <arg> doAs user, impersonates as the specified user
+ -dryrun Dryrun a coordinator (since 2.0), a workflow (since 3.3.2) or a bundle (since 5.1)
+ without actually executing it.
+ -failed re-runs the failed workflow actions of the coordinator actions (requires -rerun)
+ -filter <arg> <key><comparator><value>[;<key><comparator><value>]*
+ (All Coordinator actions satisfying the filters will be retrieved).
+ key: status or nominaltime
+ comparator: =, !=, <, <=, >, >=. = is used as OR and others as AND
+ status: values are valid status like SUCCEEDED, KILLED etc. Only = and != apply
+ for status.
+ nominaltime: time of format yyyy-MM-dd'T'HH:mm'Z'
+ -ignore <arg> change status of a coordinator job or action to IGNORED
+ (-action required to ignore coord actions)
+ -info <arg> info of a job
+ -interval <arg> polling interval in minutes (default is 5, requires -poll)
+ -kill <arg> kill a job (coordinator can mention -action or -date)
+ -len <arg> number of actions (default TOTAL ACTIONS, requires -info)
+ -localtime use local time (same as passing your time zone to -timezone).
+ Overrides -timezone option
+ -log <arg> job log
+ -errorlog <arg> job error log
+ -auditlog <arg> job audit log
+ -logfilter <arg> job log search parameter. Can be specified as -logfilter
+ opt1=val1;opt2=val1;opt3=val1. Supported options are recent,
+ start, end, loglevel, text, limit and debug
+ -nocleanup do not clean up output-events of the coordinator rerun
+ actions (requires -rerun)
+ -offset <arg> job info offset of actions (default '1', requires -info)
+ -oozie <arg> Oozie URL
+ -order <arg> order to show coord actions (default ascending order, 'desc'
+ for descending order, requires -info)
+ -poll <arg> poll Oozie until a job reaches a terminal state or a timeout
+ occurs
+ -refresh re-materialize the coordinator rerun actions (requires
+ -rerun)
+ -rerun <arg> rerun a job (coordinator requires -action or -date, bundle
+ requires -coordinator or -date)
+ -resume <arg> resume a job
+ -run run a job
+ -start <arg> start a job
+ -submit submit a job
+ -suspend <arg> suspend a job
+ -timeout <arg> timeout in minutes (default is 30, negative values indicate
+ no timeout, requires -poll)
+ -timezone <arg> use time zone with the specified ID (default GMT).
+ See 'oozie info -timezones' for a list
+ -update <arg> Update coord definition and properties
+ -value <arg> new endtime/concurrency/pausetime value for changing a
+ coordinator job
+ -verbose verbose mode
+ -sladisable disables sla alerts for the job and its children
+ -slaenable enables sla alerts for the job and its children
+ -slachange Update sla param for jobs, supported param are should-start, should-end and max-duration
+ -retries Get information of the retry attempts for a given workflow action.
+ -apijarcheck <jar> based on the supplied Fluent Job API jar, a workflow definition XML is generated and checked whether
+ it's a valid Oozie workflow. Output is whether the generated workflow is a valid one
+ -apijarsubmit <jar> based on the supplied Fluent Job API jar, a workflow definition XML is generated and submitted. When
+ the parameter =oozie.wf.application.path= isn't supplied on the command line, an HDFS location with
+ the prefix defined in
+ =oozie-site.xml#oozie.client.jobs.application.generated.path= is used. Output is the workflow ID
+ -apijarrun <jar> based on the supplied Fluent Job API jar, a workflow definition XML is generated and run. When the
+ parameter =oozie.wf.application.path= isn't supplied on the command line, an HDFS location with the
+ prefix defined in
+ =oozie-site.xml#oozie.client.jobs.application.generated.path= is used. Output is the workflow ID
+
+```
+
+### Oozie jobs operation commands
+
+```
+oozie jobs <OPTIONS> : jobs status
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -doas <arg> doAs user, impersonates as the specified user.
+ -filter <arg> user=<U>\;name=<N>\;group=<G>\;status=<S>\;frequency=<F>\;unit=<M>\;startcreatedtime=<SC>\;
+ endcreatedtime=<EC>\;sortby=<SB>
+ -jobtype <arg> job type ('Supported in Oozie-2.0 or later versions ONLY - coordinator' or 'wf' (default))
+ -len <arg> number of jobs (default '100')
+ -localtime use local time (same as passing your time zone to -timezone). Overrides -timezone option
+ -offset <arg> jobs offset (default '1')
+ -oozie <arg> Oozie URL
+ -timezone <arg> use time zone with the specified ID (default GMT). See 'oozie info -timezones' for a list
+ -kill kill all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without
+ other options, it will kill all the first 50 workflow jobs. Command will fail if one or more
+ of the jobs is in wrong state.
+ -suspend suspend all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without
+ other options, it will suspend all the first 50 workflow jobs. Command will fail if one or more
+ of the jobs is in wrong state.
+ -resume resume all jobs that satisfy the filter, len, offset, or/and jobtype options. If it's used without
+ other options, it will resume all the first 50 workflow jobs. Command will fail if one or more
+ of the jobs is in wrong state.
+ -verbose verbose mode
+```
+
+### Oozie admin operation commands
+
+```
+oozie admin <OPTIONS> : admin operations
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -configuration show Oozie system configuration
+ -doas <arg> doAs user, impersonates as the specified user
+ -instrumentation show Oozie system instrumentation
+ -javasysprops show Oozie Java system properties
+ -metrics show Oozie system metrics
+ -oozie <arg> Oozie URL
+ -osenv show Oozie system OS environment
+ -queuedump show Oozie server queue elements
+ -servers list available Oozie servers (more than one only if HA is enabled)
+ -shareliblist List available sharelib that can be specified in a workflow action
+ -sharelibupdate Update server to use a newer version of sharelib
+ -status show the current system status
+ -systemmode <arg> Supported in Oozie-2.0 or later versions ONLY. Change oozie
+ system mode [NORMAL|NOWEBSERVICE|SAFEMODE]
+ -version show Oozie server build version
+ -purge <arg> purge old oozie workflow, coordinator and bundle records from DB (parameter unit: day)
+ wf=<N>\;coord=<N>\;bundle=<N>\;limit=<N>\;oldcoordaction=<true/false>
+```
+
+### Oozie validate command
+
+```
+oozie validate <OPTIONS> <ARGS> : validate a workflow, coordinator, bundle XML file
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -oozie <arg> Oozie URL
+```
+
+### Oozie SLA operation commands
+
+```
+oozie sla <OPTIONS> : sla operations (Deprecated as of Oozie 4.0)
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -len <arg> number of results (default '100', max limited by oozie server setting which defaults to '1000')
+ -offset <arg> start offset (default '0')
+ -oozie <arg> Oozie URL
+ -filter <arg> jobid=<JobID/ActionID>\;appname=<Application Name>
+```
+
+### Oozie Pig submit command
+
+```
+oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass-through parameters to pig, any '-D' arguments
+ after '-X' are put in <configuration>
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -doas <arg> doAs user, impersonates as the specified user.
+ -config <arg> job configuration file '.properties'
+ -D <property=value> set/override value for given property
+ -file <arg> Pig script
+ -oozie <arg> Oozie URL
+ -P <property=value> set parameters for script
+```
+
+### Oozie Hive submit command
+
+```
+oozie hive <OPTIONS> -X<ARGS> : submit a hive job, everything after '-X' are pass-through parameters to hive, any '-D' arguments
+ after '-X' are put in <configuration>
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -config <arg> job configuration file '.properties'
+ -D <property=value> set/override value for given property
+ -doas <arg> doAs user, impersonates as the specified user
+ -file <arg> hive script
+ -oozie <arg> Oozie URL
+ -P <property=value> set parameters for script
+```
+
+### Oozie Sqoop submit command
+
+```
+oozie sqoop <OPTIONS> -X<ARGS> : submit a sqoop job, any '-D' arguments after '-X' are put in <configuration>
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -config <arg> job configuration file '.properties'
+ -D <property=value> set/override value for given property
+ -doas <arg> doAs user, impersonates as the specified user
+ -command <arg> sqoop command
+ -oozie <arg> Oozie URL
+```
+
+### Oozie info command
+
+```
+oozie info <OPTIONS> : get more detailed info about specific topics
+ -timezones display a list of available time zones
+```
+
+### Oozie MapReduce job command
+
+```
+oozie mapreduce <OPTIONS> : submit a mapreduce job
+ -auth <arg> select authentication type [SIMPLE|KERBEROS]
+ -config <arg> job configuration file '.properties'
+ -D <property=value> set/override value for given property
+ -doas <arg> doAs user, impersonates as the specified user
+ -oozie <arg> Oozie URL
+```
+
+## Common CLI Options
+
+### Authentication
+
+The `oozie` CLI automatically perform authentication if the Oozie server requests it. By default it supports both
+pseudo/simple authentication and Kerberos HTTP SPNEGO authentication.
+
+To perform a specific authentication, the `auth` option with authentication type requests Oozie client to run the
+specified authentication mechanism only. Oozie client provides two types `simple` and `kerberos` which supports `pseudo/simple` and `Kerberos`.
+
+For pseudo/simple authentication the `oozie` CLI uses the user name of the current OS user.
+
+For Kerberos HTTP SPNEGO authentication the `oozie` CLI uses the default principal for the OS Kerberos cache
+(normally the principal that did `kinit`).
+
+Oozie uses Apache Hadoop-Auth (Java HTTP SPNEGO) library for authentication.
+This library can be extended to support other authentication mechanisms.
+
+Once authentication is performed successfully the received authentication token is cached in the user home directory
+in the `.oozie-auth-token` file with owner-only permissions. Subsequent requests reuse the cached token while valid.
+
+The use of the cache file can be disabled by invoking the `oozie` CLI with the `-Doozie.auth.token.cache`false=
+option.
+
+To use an custom authentication mechanism, a Hadoop-Auth `Authenticator` implementation must be specified with the
+ `-Dauthenticator.class=CLASS` option.
+
+### Impersonation, doAs
+
+The `-doas` option allows the current user to impersonate other users when interacting with the Oozie
+system. The current user must be configured as a proxyuser in the Oozie system. The proxyuser configuration may
+restrict from which hosts a user may impersonate users, as well as users of which groups can be impersonated.
+
+### Oozie URL
+
+All `oozie` CLI sub-commands expect the `-oozie OOZIE_URL` option indicating the URL of the Oozie system
+to run the command against. If the OOZIE_URL environment variable has not been set, `oozie` will use the default
+URL specified in oozie-client-env.sh (equivalent to `!http://$(hostname -f):11000/oozie`).
+
+If the `-oozie` option is not specified, the `oozie` CLI will look for the `OOZIE_URL` environment variable
+and uses it if set.
+
+If the option is not provided and the environment variable is not set, the `oozie` CLI will fail.
+
+### Time zone
+
+The `-timezone TIME_ZONE_ID` option in the `job` and `jobs` sub-commands allows you to specify the time zone to use in
+the output of those sub-commands. The `TIME_ZONE_ID` should be one of the standard Java Time Zone IDs. You can get a
+list of the available time zones with the command `oozie info -timezones`.
+
+If the `-localtime` option is used, it will cause Oozie to use whatever the time zone is of the machine. If
+both `-localtime` and `-timezone TIME_ZONE_ID` are used, the `-localtime` option will override
+the `-timezone TIME_ZONE_ID` option. If neither option is given, Oozie will look for the `OOZIE_TIMEZONE` environment
+variable and uses it if set. If neither option is given and the environment variable is not set, or if Oozie is given an invalid
+time zone, it will use GMT.
+
+### Debug Mode
+
+If you export `OOZIE_DEBUG=1` then the Oozie CLI will output the Web Services API details used by any commands you
+execute. This is useful for debugging purposes to or see how the Oozie CLI works with the WS API.
+
+### CLI retry
+Oozie CLI retries connection to Oozie servers for transparent high availability failover when one of the Oozie servers go down.
+`Oozie` CLI command will retry for all commands in case of ConnectException.
+In case of SocketException, all commands except `PUT` and `POST` will have retry logic.
+All job submit are POST call, examples of PUT and POST commands can be find out from [WebServicesAPI](WebServicesAPI.html).
+Retry count can be configured with system property `oozie.connection.retry.count`. Default count is 4.
+
+
+## Job Operations
+
+### Submitting a Workflow, Coordinator or Bundle Job
+
+* Submitting bundle feature is only supported in Oozie 3.0 or later. Similarly, all bundle operation features below are only
+supported in Oozie 3.0 or later.
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -config job.properties -submit
+.
+job: 14-20090525161321-oozie-joe
+```
+
+The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
+Configuration file (.xml). This file must be specified with the `-config` option.
+
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
+coordinator application path must be specified in the file with the `oozie.coord.application.path` property.The
+bundle application path must be specified in the file with the `oozie.bundle.application.path` property.
+Specified path must be an HDFS path.
+
+The job will be created, but it will not be started, it will be in `PREP` status.
+
+### Starting a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -start 14-20090525161321-oozie-joe
+```
+
+The `start` option starts a previously submitted workflow job or bundle job that is in `PREP` status.
+
+After the command is executed the workflow job will be in `RUNNING` status and bundle job will be in `RUNNING`status.
+
+A coordinator job does not support the `start` action. It will show the following error message when trying to start it
+via the CLI:
+
+
+```
+Error: E0303 : E0303: Invalid parameter value, [action] = [start]
+```
+
+### Running a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -config job.properties -run
+.
+job: 15-20090525161321-oozie-joe
+```
+
+The `run` option creates and starts a workflow job, coordinator job or bundle job.
+
+The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
+Configuration file (.xml). This file must be specified with the `-config` option.
+
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
+coordinator application path must be specified in the file with the `oozie.coord.application.path` property. The
+bundle application path must be specified in the file with the `oozie.bundle.application.path` property.The
+specified path must be an HDFS path.
+
+The job will be created and it will started, the job will be in `RUNNING` status.
+
+### Suspending a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -suspend 14-20090525161321-oozie-joe
+```
+
+The `suspend` option suspends a workflow job in `RUNNING` status.
+After the command is executed the workflow job will be in `SUSPENDED` status.
+
+The `suspend` option suspends a coordinator/bundle job in `RUNNING`, `RUNNINGWITHERROR` or `PREP` status.
+When the coordinator job is suspended, running coordinator actions will stay in running and the workflows will be suspended. If the coordinator job is in `RUNNING`status, it will transit to `SUSPENDED`status; if it is in `RUNNINGWITHERROR`status, it will transit to `SUSPENDEDWITHERROR`; if it is in `PREP`status, it will transit to `PREPSUSPENDED`status.
+
+When the bundle job is suspended, running coordinators will be suspended. If the bundle job is in `RUNNING`status, it will transit to `SUSPENDED`status; if it is in `RUNNINGWITHERROR`status, it will transit to `SUSPENDEDWITHERROR`; if it is in `PREP`status, it will transit to `PREPSUSPENDED`status.
+
+### Resuming a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -resume 14-20090525161321-oozie-joe
+```
+
+The `resume` option resumes a workflow job in `SUSPENDED` status.
+
+After the command is executed the workflow job will be in `RUNNING` status.
+
+
+The `suspend` option suspends a coordinator/bundle job in `SUSPENDED`, `SUSPENDEDWITHERROR` or `PREPSUSPENDED` status.
+If the coordinator job is in `SUSPENDED`status, it will transit to `RUNNING`status; if it is in `SUSPENDEDWITHERROR`status, it will transit to `RUNNINGWITHERROR`; if it is in `PREPSUSPENDED`status, it will transit to `PREP`status.
+
+When the coordinator job is resumed it will create all the coordinator actions that should have been created during the time
+it was suspended, actions will not be lost, they will delayed.
+
+When the bundle job is resumed, suspended coordinators will resume running. If the bundle job is in `SUSPENDED`status, it will transit to `RUNNING`status; if it is in `SUSPENDEDWITHERROR`status, it will transit to `RUNNINGWITHERROR`; if it is in `PREPSUSPENDED`status, it will transit to `PREP`status.
+
+
+### Killing a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -kill 14-20090525161321-oozie-joe
+```
+
+The `kill` option kills a workflow job in `PREP`, `SUSPENDED` or `RUNNING` status and a coordinator/bundle job in
+`PREP`, `RUNNING`, `PREPSUSPENDED`, `SUSPENDED`, `PREPPAUSED`, or `PAUSED` status.
+
+After the command is executed the job will be in `KILLED` status.
+
+### Killing a Coordinator Action or Multiple Actions
+
+Example:
+
+
+```
+$oozie job -kill <coord_Job_id> [-action 1, 3-4, 7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
+```
+
+ * The `kill` option here for a range of coordinator actions kills a non-terminal (`RUNNING`, `WAITING`, `READY`, `SUSPENDED`) coordinator action when coordinator job is not in `FAILED` or `KILLED` state.
+ * Either -action or -date should be given.
+ * If neither -action nor -date is given, the exception will be thrown. Also if BOTH -action and -date are given, an error will be thrown.
+ * Multiple ranges can be used in -action or -date. See the above example.
+ * If one of the actions in the given list of -action is already in terminal state, the output of this command will only include the other actions.
+ * The dates specified in -date must be UTC.
+ * Single date specified in -date must be able to find an action with matched nominal time to be effective.
+ * After the command is executed the killed coordinator action will have `KILLED` status.
+
+### Changing endtime/concurrency/pausetime/status of a Coordinator Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value endtime=2011-12-01T05:00Z\;concurrency=100\;2011-10-01T05:00Z
+$ oozie job -oozie http://localhost:11000/oozie -change 0000001-140321155112907-oozie-puru-C -value status=RUNNING
+```
+
+The `endtime/concurrency/pausetime` option changes a coordinator job that is not in `KILLED` status.
+
+Valid value names are:
+
+
+ * endtime: the end time of the coordinator job.
+ * concurrency: the concurrency of the coordinator job.
+ * pausetime: the pause time of the coordinator job.
+ * status: new status for coordinator job.
+
+Conditions and usage:
+
+ * Repeated value names are not allowed.
+ * New end time should not be before job's start time and last action time.
+ * If end time is before job start time and if the job has not materialized any actions, then job status is changed to SUCCEEDED.
+ * Currently status only takes RUNNING and can be used to change the status of FAILED, KILLED, IGNORED coordinator job to RUNNING and resuming materialization. This status change command does not affect the status of already materialized actions in the coordinator. If there are FAILED, KILLED or IGNORED coordinator actions they have to be rerun separately.
+ * New concurrency value has to be a valid integer.
+ * All lookahead actions which are in WAITING/READY state will be revoked according to the new pause/end time. If any action after new pause/end time is not in WAITING/READY state, an exception will be thrown.
+ * Also empty string "" can be used to reset pause time to none.
+ * Endtime/concurrency/pausetime of IGNORED Job cannot be changed.
+
+After the command is executed the job's end time, concurrency or pause time should be changed. If an already-succeeded job changes its end time, its status will become running.
+
+### Changing endtime/pausetime of a Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value pausetime=2011-12-01T05:00Z
+```
+
+The `change` option changes a bundle job that is not in `KILLED` status.
+
+Valid value names are:
+
+ * pausetime: the pause time of the bundle job.
+ * endtime: the end time of the bundle job.
+
+Repeated value names are not allowed. An empty string "" can be used to reset pause time to none. New end time should not be before job's kickoff time.
+
+Bundle will execute pause/end date change command on each coordinator job. Refer conditions and usage section of coordinator change command for more details [Coordinator job change command](DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job).
+
+### Rerunning a Workflow Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -config job.properties -rerun 14-20090525161321-oozie-joe
+```
+
+The `rerun` option reruns a completed ( `SUCCEEDED`, `FAILED` or `KILLED` ) job skipping the specified nodes.
+
+The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
+Configuration file (.xml). This file must be specified with the `-config` option.
+
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
+specified path must be an HDFS path.
+
+The list of nodes to skipped must be provided in the `oozie.wf.rerun.skip.nodes` property separated by commas.
+
+After the command is executed the job will be in `RUNNING` status.
+
+Refer to the [Rerunning Workflow Jobs](DG_WorkflowReRun.html) for details on rerun.
+
+### Rerunning a Coordinator Action or Multiple Actions
+
+Example:
+
+
+```
+$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>]
+[-action 1, 3-4, 7-40] (-action or -date is required to rerun.)
+[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
+(if neither -action nor -date is given, the exception will be thrown.)
+```
+
+The `rerun` option reruns a terminated (`TIMEDOUT`, `SUCCEEDED`, `KILLED`, `FAILED`, `IGNORED`) coordinator action when coordinator job
+is not in `FAILED` or `KILLED` state.
+
+After the command is executed the rerun coordinator action will be in `WAITING` status.
+
+Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details on rerun.
+
+### Rerunning a Bundle Job
+
+Example:
+
+
+```
+$oozie job -rerun <bundle_Job_id> [-nocleanup] [-refresh]
+[-coordinator c1, c3, c4] (-coordinator or -date is required to rerun.)
+[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
+(if neither -coordinator nor -date is given, the exception will be thrown.)
+```
+
+The `rerun` option reruns coordinator actions belonging to specified coordinators within the specified date range.
+
+After the command is executed the rerun coordinator action will be in `WAITING` status.
+
+
+### Checking the Information and Status of a Workflow, Coordinator or Bundle Job or a Coordinator Action
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-joe
+.
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+Workflow Name : map-reduce-wf
+App Path : hdfs://localhost:8020/user/joe/workflows/map-reduce
+Status : SUCCEEDED
+Run : 0
+User : joe
+Group : users
+Created : 2009-05-26 05:01 +0000
+Started : 2009-05-26 05:01 +0000
+Ended : 2009-05-26 05:01 +0000
+Actions
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+Action Name Type Status Transition External Id External Status Error Code Start End
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+hadoop1 map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+```
+
+The `info` option can display information about a workflow job or coordinator job or coordinator action.
+The `info` option for a Coordinator job will retrieve the Coordinator actions ordered by nominal time. However, the `info` command may timeout if the number of Coordinator actions are very high. In that case, `info` should be used with `offset` and `len` option.
+
+The `offset` and `len` option should be used for pagination. offset determines the start offset of the action
+returned among all the actions that matched the filter criteria. len determines number of actions to be returned.
+
+The `localtime` option displays times in local time, if not specified times are displayed in GMT.
+
+The `filter` option can be used to filter coordinator actions based on some criteria.
+The filter option syntax is: `<key><comparator><value>[;<key><comparator><value>]*`.
+(Note escape `\` needed before semicolon to specify multiple names for filter in shell)
+key: status or nominalTime
+comparator: `, !`, <, <`, >, >`
+value: valid status like SUCCEEDED, KILLED, RUNNING etc. Only ` and !` apply for status
+value for nominalTime is valid date of the format yyyy-MM-dd'T'HH:mm'Z' (like 2014-06-01T00:00Z)
+
+Multiple values must be specified as different name value pairs. The query is formed by doing AND of all conditions,
+with the exception of = which uses OR if there are multiple values for the same key. For example,
+filter 'status`RUNNING;status`WAITING;nominalTime>`2014-06-01T00:00Z' maps to query (status ` RUNNING OR status =
+WAITING) AND nominalTime >` 2014-06-01T00:00Z which returns all waiting or running actions with nominalTime >`
+2014-06-01T00:00Z.
+
+Currently, the filter option can be used only with an `info` option on Coordinator job.
+
+The `verbose` option gives more detailed information for all the actions, if checking for workflow job or coordinator job.
+An example below shows how the `verbose` option can be used to gather action statistics information for a job:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -info 0000001-111219170928042-oozie-para-W@mr-node -verbose
+ID : 0000001-111219170928042-oozie-para-W@mr-node
+------------------------------------------------------------------------------------------------------------------------------------
+Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201112191708_0006
+Error Code : -
+Error Message : -
+External ID : job_201112191708_0006
+External Status : SUCCEEDED
+Name : mr-node
+Retries : 0
+Tracker URI : localhost:8021
+Type : map-reduce
+Started : 2011-12-20 01:12
+Status : OK
+Ended : 2011-12-20 01:12
+External Stats : {"org.apache.hadoop.mapred.JobInProgress$Counter":{"TOTAL_LAUNCHED_REDUCES":1,"TOTAL_LAUNCHED_MAPS":1,"DATA_LOCAL_MAPS":1},"ACTION_TYPE":"MAP_REDUCE","FileSystemCounters":{"FILE_BYTES_READ":1746,"HDFS_BYTES_READ":1409,"FILE_BYTES_WRITTEN":3524,"HDFS_BYTES_WRITTEN":1547},"org.apache.hadoop.mapred.Task$Counter":{"REDUCE_INPUT_GROUPS":33,"COMBINE_OUTPUT_RECORDS":0,"MAP_INPUT_RECORDS":33,"REDUCE_SHUFFLE_BYTES":0,"REDUCE_OUTPUT_RECORDS":33,"SPILLED_RECORDS":66,"MAP_OUTPUT_BYTES":1674,"MAP_INPUT_BYTES":1409,"MAP_OUTPUT_RECORDS":33,"COMBINE_INPUT_RECORDS":0,"REDUCE_INPUT_RECORDS":33}}
+External ChildIDs : null
+------------------------------------------------------------------------------------------------------------------------------------
+```
+
+The two fields External Stats and External ChildIDs display the action statistics information (that includes counter information in case of MR action and PigStats information in case of a pig action) and child ids of the given job.
+
+Note that the user can turn on/off External Stats by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in workflow.xml. By default, it is set to false (not to collect External Stats). External ChildIDs will always be stored.
+
+### Listing all the Workflows for a Coordinator Action
+
+A coordinator action kicks off different workflows for its original run and all subsequent reruns.
+Getting a list of those workflow ids is a useful tool to keep track of your actions' runs and
+to go debug the workflow job logs if required. Along with ids, it also lists their statuses,
+and start and end times for quick reference.
+
+This is achieved by using the Coordinator Action info command and specifying a flag **`allruns`**
+along with the `info` command.
+
+
+```
+$ oozie job -info 0000001-111219170928042-oozie-joe-C@1 -allruns -oozie http://localhost:11000/oozie
+.
+Job ID Status Started Ended
+.----------------------------------------------------------------------------------------------------
+0000001-140324163709596-oozie-joe-W SUCCEEDED 2014-03-24 23:40 GMT 2014-03-24 23:40 GMT
+.----------------------------------------------------------------------------------------------------
+0000000-140324164318985-oozie-joe-W SUCCEEDED 2014-03-24 23:44 GMT 2014-03-24 23:44 GMT
+.----------------------------------------------------------------------------------------------------
+0000001-140324164318985-oozie-joe-W SUCCEEDED 2014-03-24 23:44 GMT 2014-03-24 23:44 GMT
+.----------------------------------------------------------------------------------------------------
+```
+
+### Listing all retry attempts of a workflow action
+
+When retry-max is specified for an action in the workflow definition, and there is a failure, it will be retried till it succeeds or retry-max attempts are exhausted. To get information on all the retry attempts, `-retries` command can be used.
+
+
+```
+$ oozie job -retries 0000000-161212175234862-oozie-puru-W@pig-node -oozie http://localhost:11000/oozie
+
+ID : 0000000-161212175234862-oozie-puru-W@pig-node
+------------------------------------------------------------------------------------------------------------------------------------
+Attempt : 1
+Start Time : Tue, 13 Dec 2016 01:56:24 GMT
+End Time : Tue, 13 Dec 2016 01:56:27 GMT
+Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
+------------------------------------------------------------------------------------------------------------------------------------
+Attempt : 2
+Start Time : Tue, 13 Dec 2016 01:56:24 GMT
+End Time : Tue, 13 Dec 2016 01:56:27 GMT
+Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
+------------------------------------------------------------------------------------------------------------------------------------
+Attempt : 3
+Start Time : Tue, 13 Dec 2016 01:56:24 GMT
+End Time : Tue, 13 Dec 2016 01:56:27 GMT
+Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
+------------------------------------------------------------------------------------------------------------------------------------
+$
+```
+
+
+### Checking the xml definition of a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -definition 14-20090525161321-oozie-joe
+.
+<workflow-app xmlns="uri:oozie:workflow:0.2" name="sm3-segment-2413">
+ <start to="p0"/>
+ <action name="p0">
+ </action>
+ <end name="end"/>
+</workflow-app>
+
+```
+
+### Checking the server logs of a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+
+$ oozie job -oozie http://localhost:11000/oozie -log 14-20090525161321-oozie-joe
+
+```
+
+### Checking the server error logs of a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -errorlog 0000000-150121110331712-oozie-puru-B
+2015-01-21 11:33:29,090 WARN CoordSubmitXCommand:523 - SERVER[-] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150121110331712-oozie-puru-B] ACTION[] SAXException :
+org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-type.2.4.a: Invalid content was found starting with element 'concurrency'. One of '{"uri:oozie:coordinator:0.2":controls, "uri:oozie:coordinator:0.2":datasets, "uri:oozie:coordinator:0.2":input-events, "uri:oozie:coordinator:0.2":output-events, "uri:oozie:coordinator:0.2":action}' is expected.
+ at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
+ at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
+ at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
+ at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
+ at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
+ at org.apache.xerces.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(Unknown Source)
+ at org.apache.xerces.impl.xs.XMLSchemaValidator.reportSchemaError(Unknown Source)
+ at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source)
+ at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source)
+ at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
+ at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
+ at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
+ at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
+ at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
+ at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(Unknown Source)
+ at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(Unknown Source)
+```
+
+
+### Checking the audit logs of a Workflow, Coordinator or Bundle Job
+
+Example:
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -auditlog 0000000-150322000230582-oozie-puru-C
+2015-03-22 00:04:35,494 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 00:05:13,823 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 00:06:59,561 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 23:22:20,012 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+```
+
+### Checking the server logs for particular actions of a Coordinator Job
+
+Example:
+
+
+```
+
+$ oozie job -log <coord_job_id> [-action 1, 3-4, 7-40] (-action is optional.)
+
+```
+
+### Filtering the server logs with logfilter options
+
+User can provide multiple option to filter logs using -logfilter opt1=val1;opt2=val1;opt3=val1. This can be used to fetch only just logs of interest faster as fetching Oozie server logs is slow due to the overhead of pattern matching.
+Available options are:
+
+ * recent: Specify recent hours/min of logs to scan. The recent offset specified is taken relative to the `end` time specified, job end time or the current system time if the job is still running in that order of precedence. For eg: recent=3h or recent=30m will fetch logs starting 3 hours/30 minutes before the end time and up to the end time. H/h is used to denote hour and M/m is used to denote minutes. If no suffix is specified default is hours.
+ * start: Start time for scanning logs. Default is start time of the job. User can provide a valid date or offset similar to `recent` option. Valid date formats are "yyyy-MM-dd'T'HH:mm'Z'" and "yyyy-MM-dd HH:mm:ss,SSS". When an offset is specified, it is calculated relative to the start time of the job. For eg: start=2h will fetch logs starting 2 hours after the job was started.
+ * end: End time for scanning logs. Default is end time of the job or current system time if the job is still running. User can provide a valid date or offset similar to `start` option. When an offset is specified, it is calculated relative to start time i.e job start time . For eg: end=2h will fetch logs from start time and start time plus 2 hours.
+ * loglevel : Multiple log levels separated by "|" can be specified. Supported log levels are ALL, DEBUG, ERROR, INFO, TRACE, WARN, FATAL.
+ * text: String to search in logs.
+ * limit : Limit number of line to be searched. Log search will end when when n lines(excluding stack-trace) have been matched.
+ * debug : Prints debug information on the log files, time ranges and patterns being searched for. Can be used to debug if expected logs are not shown with the filter options provided.
+
+Examples.
+Searching log with log level ERROR or WARN will only give log with Error and Warning (with stack-trace) only.
+This will be useful if job has failed and user want to find error logs with exception.
+
+```
+
+$ ./oozie job -log 0000006-140319184715726-oozie-puru-W -logfilter loglevel=WARN\;limit=3 -oozie http://localhost:11000/oozie/
+2014-03-20 10:01:52,977 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@:start:] [***0000006-140319184715726-oozie-puru-W@:start:***]Action status=DONE
+2014-03-20 10:01:52,977 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@:start:] [***0000006-140319184715726-oozie-puru-W@:start:***]Action updated in DB!
+2014-03-20 10:01:53,189 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@mr-node-1] Error starting action [mr-node-1]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: java.io.IOException: java.io.IOException: Queue "aaadefault" does not exist
+ at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615)
+ at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
+ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+ at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
+ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
+ at java.lang.reflect.Method.invoke(Method.java:597)
+ at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
+ at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
+ at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
+ at java.security.AccessController.doPrivileged(Native Method)
+ at javax.security.auth.Subject.doAs(Subject.java:394)
+ at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
+ at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
+Caused by: java.io.IOException: Queue "aaadefault" does not exist
+ at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:433)
+ at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613)
+ ... 12 more
+$
+```
+
+Search with specific text and recent option.
+
+```
+$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter text=Missing\;limit=4\;recent=1h -oozie http://localhost:11000/oozie/
+2014-03-20 09:59:50,329 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/
+2014-03-20 09:59:50,330 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing.
+2014-03-20 10:02:19,087 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/
+2014-03-20 10:02:19,088 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing.
+$
+```
+
+Search example with specific date range.
+
+```
+$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter "start=2014-03-20 10:00:57,063;end=2014-03-20 10:10:57,063" -oozie http://localhost:11000/oozie/
+2014-03-20 10:00:57,063 INFO CoordActionUpdateXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] Updating Coordinator action id :0000003-140319184715726-oozie-puru-C@1 status to KILLED, pending = 0
+2014-03-20 10:02:18,967 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] materialize actions for tz=Coordinated Universal Time,
+ start=Thu Dec 31 18:00:00 PST 2009, end=Thu Dec 31 19:00:00 PST 2009,
+ timeUnit 12,
+ frequency :60:MINUTE,
+ lastActionNumber 1
+2014-03-20 10:02:18,971 WARN CoordELFunctions:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] If the initial instance of the dataset is later than the current-instance specified, such as coord:current(-200) in this case, an empty string is returned. This means that no data is available at the current-instance specified by the user and the user could try modifying his initial-instance to an earlier time.
+2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] [0000003-140319184715726-oozie-puru-C]: all actions have been materialized, set pending to true
+2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] Coord Job status updated to = RUNNING
+```
+
+### Dryrun of Coordinator Job
+
+* This feature is only supported in Oozie 2.0 or later.
+
+Example:
+
+
+```
+
+$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
+***coordJob after parsing: ***
+<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20"
+start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles">
+ <output-events>
+ <data-out name="Output" dataset="DayLogs">
+ <dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
+ <uri-template>hdfs://localhost:8020/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template>
+ </dataset>
+ <instance>${coord:current(0)}</instance>
+ </data-out>
+ </output-events>
+ <action>
+ </action>
+</coordinator-app>
+
+***actions for instance***
+***total coord actions is 1 ***
+------------------------------------------------------------------------------------------------------------------------------------
+coordAction instance: 1:
+<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20"
+start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles">
+ <output-events>
+ <data-out name="Output" dataset="DayLogs">
+ <uris>hdfs://localhost:8020/user/angeloh/coord_examples/2009/03/06</uris>
+ <dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
+ <uri-template>hdfs://localhost:8020/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template>
+ </dataset>
+ </data-out>
+ </output-events>
+ <action>
+ </action>
+</coordinator-app>
+------------------------------------------------------------------------------------------------------------------------------------
+
+```
+
+The `dryrun` option tests running a coordinator job with given job properties and does not create the job.
+
+The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
+Configuration file (.xml). This file must be specified with the `-config` option.
+
+The coordinator application path must be specified in the file with the `oozie.coord.application.path` property. The
+specified path must be an HDFS path.
+
+### Dryrun of Workflow Job
+
+* This feature is only supported in Oozie 3.3.2 or later.
+
+Example:
+
+
+```
+
+$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
+OK
+
+```
+
+The `dryrun` option tests running a workflow job with given job properties and does not create the job.
+
+The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
+Configuration file (.xml). This file must be specified with the `-config` option.
+
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
+specified path must be an HDFS path.
+
+If the workflow is accepted (i.e. Oozie is able to successfully read and parse it), it will return `"OK"`; otherwise, it will return
+an error message describing why it was rejected.
+
+### Dryrun of Bundle Job
+
+* This feature is only supported in Oozie 5.1 or later.
+
+Example:
+
+
+```
+
+$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
+***Bundle job after parsing: ***
+<bundle-app xmlns="uri:oozie:bundle:0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="My Bundle">
+ <parameters>
+ <property>
+ <name>oozie.use.system.libpath</name>
+ <value>true</value>
+ </property>
+ </parameters>
+ <controls>
+ <kick-off-time>2017-02-14T00:13Z</kick-off-time>
+ </controls>
+ <coordinator name="My Coordinator-0">
+ <app-path>hdfs://localhost:8020/user/hue/oozie/deployments/_admin_-oozie-24-1487060026.39</app-path>
+ <configuration>
+ <property>
+ <name>wf_application_path</name>
+ <value>hdfs://localhost:8020/user/hue/oozie/workspaces/hue-oozie-1486786607.01</value>
+ </property>
+ <property>
+ <name>start_date</name>
+ <value>2017-02-14T08:12Z</value>
+ </property>
+ <property>
+ <name>end_date</name>
+ <value>2017-02-21T08:12Z</value>
+ </property>
+ </configuration>
+ </coordinator>
+</bundle-app>
+
+```
+
+The `dryrun` option tests running a bundle job with given job properties and does not create the job.
+
+The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
+Configuration file (.xml). This file must be specified with the `-config` option.
+
+If the bundle job is accepted (i.e. Oozie is able to successfully read and parse it), it will return the parsed bundle job in xml;
+otherwise, it will return an error message describing why it was rejected.
+
+### Updating coordinator definition and properties
+Existing coordinator definition will be replaced by new definition. The refreshed coordinator would keep the same coordinator ID, state, and coordinator actions.
+All created coord action(including in WAITING) will use old configuration.
+One can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action
+
+Update command also verifies coordinator definition like submit command, if there is any issue with definition, update will fail.
+Update command with -dryrun will show coordinator definition and properties differences.
+Config option is optional, if not specified existing coordinator property is used to find coordinator path.
+
+Update command doesn't allow update of coordinator name, frequency, start time, end time and timezone and will fail on an attempt to change any of them. To change end time of coordinator use the `-change` command.
+
+To change the entire XML for a running coordinator, hdfs path for the new XML can be specified
+as `oozie.coord.application.path` in job.properties. Then, use `-config job.properties` in the update command.
+
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -config job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun
+.
+**********Job definition changes**********
+@@ -3,8 +3,8 @@
+ <concurrency>1</concurrency>
+ </controls>
+ <input-events>
+- <data-in name="input" dataset="raw-logs">
+- <dataset name="raw-logs" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
++ <data-in name="input" dataset="raw-logs-rename">
++ <dataset name="raw-logs-rename" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
+ <uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
+ <done-flag />
+ </dataset>
+**********************************
+**********Job conf changes**********
+@@ -8,10 +8,6 @@
+ <value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
+ </property>
+ <property>
+- <name>old</name>
+- <value>test</value>
+- </property>
+- <property>
+ <name>user.name</name>
+ <value>purushah</value>
+ </property>
+@@ -28,6 +24,10 @@
+ <value>hdfs://localhost:9000</value>
+ </property>
+ <property>
++ <name>adding</name>
++ <value>new</value>
++ </property>
++ <property>
+ <name>jobTracker</name>
+ <value>localhost:9001</value>
+ </property>
+**********************************
+```
+
+
+### Ignore a Coordinator Job
+
+Example:
+
+
+```
+$oozie job -ignore <coord_Job_id>
+```
+
+The `ignore` option changes a coordinator job in `KILLED`, `FAILED` to `IGNORED` state.
+When a coordinator job in a bundle is in `IGNORED` state, the coordinator job doesn't impact the state of the bundle job.
+For example, when a coordinator job in a bundle failed and afterwards ignored, the bundle job becomes `SUCCEEDED` instead of `DONEWITHERROR` as long as other coordinator jobs in the bundle succeeded.
+ A ignored coordinator job can be changed to `RUNNING` using -change command.
+ Refer to the [Coordinator job change command](DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job) for details.
+
+### Ignore a Coordinator Action or Multiple Coordinator Actions
+
+Example:
+
+
+```
+$oozie job -ignore <coord_Job_id> -action 1,3-4,7-40
+```
+The `ignore` option changes a coordinator action(s) in terminal state (`KILLED`, `FAILED`, `TIMEDOUT`) to `IGNORED` state, while not changing the state of the coordinator job.
+When a coordinator action is in `IGNORED` state, the action doesn't impact the state of a coordinator job.
+For example, when a coordinator action failed and afterwards ignored, a coordinator job becomes `SUCCEEDED` instead of `DONEWITHERROR` as long
+ as other coordinator actions succeeded.
+
+A ignored coordinator action can be rerun using -rerun command.
+Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details.
+When a workflow job of a ignored coordinator action is rerun, the coordinator action becomes `RUNNING` state.
+
+### Polling an Oozie job
+
+This command allows polling the Oozie server for an Oozie job until it reaches a completed status (e.g. `SUCCEEDED`, `KILLED`, etc).
+
+Example:
+
+
+```
+$ oozie job -poll <job_id> -interval 10 -timeout 60 -verbose
+.
+RUNNING
+RUNNING
+RUNNING
+SUCCEEDED
+```
+
+The `-poll` argument takes a valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.
+
+All other arguments are optional:
+
+ * `verbose` will cause the job status to be printed at each poll; otherwise, there will be no output
+ * `interval` allows specifying the polling interval in minutes (default is 5)
+ * `timeout` allows specifying the timeout in minutes (default is 30 minutes); negative values indicate no timeout
+
+### Changing job SLA definition and alerting
+ * slaenable command can be used to enable job sla alerts.
+ * sladisable command can be used to disable job sla alerts.
+ * slachange command can be used to change sla job definition.
+ * Supported parameters for sla change command are should-start, should-end and max-duration. Please specify the value in single quotes instead of double quotes in command line to avoid bash interpreting braces in EL functions and causing error.
+ * All sla commands takes -action or -date parameter. For bundle jobs additional -coordinator (coord_name/id) parameter can be passed. Sla change command need extra parameter -value to specify new sla definition.
+ * Sla commands without -action or -date parameter is applied to all non terminated actions and all future actions.
+ * Sla commands with -action or -date parameter will be applied to only non terminated actions.
+
+ Eg.
+
+```
+ $oozie job -slaenable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z]
+ $oozie job -sladisable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z]
+ $oozie job -slachange <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] -value 'sla-max-duration=${10 ** MINUTES};sla-should-end=${30 ** MINUTES};sla-max-duration=${30 * MINUTES}'
+ $oozie job -slaenable <bundle_job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] [-coordinator <List_of_coord_names/ids]
+```
+
+### Getting missing dependencies of coordinator action(s)
+ * Coordination action id can be specified directly for getting missing dependencies of a single action.
+ * To get information on multiple actions, either -action or -date option can be specified with the coordinator job id.
+ * missingdeps command doesn't recompute dependencies. It list missing dependencies which were last computed.
+ * Oozie checks missing dependencies sequentially, and it will stop on first missing dependency. `Blocked On` is the first missing dependency for action. So, there could be a chance that Oozie will report some missing dependencies, but it might be present. To resolve the waiting issue, one should fix the blockedOn missing dependency.
+ * For input logic, missingdeps command doesn't compute input-logic expression. It will report everything which is missing or not computed.
+
+```
+$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C -action 1
+$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C@1
+.
+CoordAction : 1
+Blocked on : hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS
+.
+Dataset : input-1
+Pending Dependencies :
+ hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/01/00/_SUCCESS
+.
+Dataset : input-2
+Pending Dependencies :
+ hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/01/00/_SUCCESS
+ hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/40/_SUCCESS
+ hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/20/_SUCCESS
+ hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS
+$
+```
+
+### Checking a workflow definition generated by a Fluent Job API jar file
+
+Since Oozie 5.1.0.
+
+Generate a workflow definition given the Fluent Job API jar file supplied at command line, and check for its correctness.
+
+**Preconditions:**
+
+ * the Fluent Job API jar file has to be present and readable by the current user at the local path provided
+ * the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
+ definition is written there
+
+If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
+
+For more information what an Fluent Job API jar file is, how to build it etc.,
+refer to [Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
+
+**Example:**
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -validatejar /tmp/workflow-api-jar.jar
+Valid workflow-app
+```
+
+**Example (verbose):**
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -validatejar /tmp/workflow-api-jar.jar -verbose
+Checking API jar:/tmp/workflow-api-jar.jar
+Loading API jar /tmp/workflow-api-jar.jar
+Workflow job definition generated from API jar:
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" ... name="shell-example">
+...
+</workflow:workflow-app>
+.
+API jar is written to /tmp/workflow1876390751841950810.xml
+Servlet response is:
+Valid workflow-app
+API jar is valid.
+```
+
+### Submitting a workflow definition generated by a Fluent Job API jar file
+
+Since Oozie 5.1.0.
+
+Generate a workflow definition given the Fluent Job API jar file supplied at command line, write it to a provided or generated HDFS
+location, and submit.
+
+**Preconditions:**
+
+ * all the parameters that are present in the workflow definition have to be supplied either as command line parameters or via
+ `job.properties` passed along with the `-config` option
+ * the Fluent Job API jar file has to be present and readable by the current user at the local path provided
+ * the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
+ definition is written there
+ * the HDFS folder either given by `-Doozie.wf.application.path` command line parameter, or in its absence contained by
+ `oozie-site.xml#oozie.client.jobs.application.generated.path` has to be writable by the current user
+
+If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
+
+For more information what an Fluent Job API jar file is, how to build it etc., refer to
+[Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
+
+**Example:**
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -submitjar /tmp/workflow-api-jar.jar -config /tmp/job.properties
+job: 0000009-180107110323219-oozie-oozi-W
+```
+
+**Example (verbose):**
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -submitjar /tmp/workflow-api-jar.jar -config /tmp/job.properties -verbose
+Submitting a job based on API jar: /tmp/workflow-api-jar.jar
+Loading API jar /tmp/workflow-api-jar.jar
+Workflow job definition generated from API jar:
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" ... name="shell-example">
+...
+</workflow:workflow-app>
+.
+job: 0000010-180107110323219-oozie-oozi-W
+Job based on API jar submitted successfully.
+```
+
+### Running a workflow definition generated by a Fluent Job API jar file
+
+Since Oozie 5.1.0.
+
+Generate a workflow definition given the Fluent Job API jar file supplied at command line, write it to a provided or generated HDFS
+location, submit and run.
+
+**Preconditions:**
+
+ * all the parameters that are present in the workflow definition have to be supplied either as command line parameters or via
+ `job.properties` passed along with the `-config` option
+ * the Fluent Job API jar file has to be present and readable by the current user at the local path provided
+ * the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
+ definition is written there
+ * the HDFS folder either given by `-Doozie.wf.application.path` command line parameter, or in its absence contained by
+ `oozie-site.xml#oozie.client.jobs.application.generated.path` has to be writable by the current user
+
+If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
+
+For more information what an Fluent Job API jar file is, how to build it etc., refer to
+[Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
+
+**Example:**
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -runjar /tmp/workflow-api-jar.jar -config /tmp/job.properties
+job: 0000011-180107110323219-oozie-oozi-W
+```
+
+**Example (verbose):**
+
+
+```
+$ oozie job -oozie http://localhost:11000/oozie -runjar /tmp/workflow-api-jar.jar -config /tmp/job.properties -verbose
+Submitting a job based on API jar: /tmp/workflow-api-jar.jar
+Loading API jar /tmp/workflow-api-jar.jar
+Workflow job definition generated from API jar:
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" ... name="shell-example">
+...
+</workflow:workflow-app>
+.
+job: 0000010-180107110323219-oozie-oozi-W
+Job based on API jar run successfully.
+```
+
+## Jobs Operations
+
+### Checking the Status of multiple Workflow Jobs
+
+Example:
+
+
+```
+$ oozie jobs -oozie http://localhost:11000/oozie -localtime -len 2 -filter status=RUNNING
+.
+Job Id Workflow Name Status Run User Group Created Started Ended
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+4-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:34 +0530 2009-05-27 15:34 +0530 -
+0-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:15 +0530 2009-05-27 15:15 +0530 -
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+```
+
+The `jobs` sub-command will display information about multiple jobs.
+
+The `offset` and `len` option specified the offset and number of jobs to display, default values are `1` and `100`
+respectively.
+
+The `localtime` option displays times in local time, if not specified times are displayed in GMT.
+
+The `verbose` option gives more detailed information for each job.
+
+A filter can be specified after all options.
+
+The `filter`option syntax is: `[NAME=VALUE][;NAME=VALUE]*`.
+
+Valid filter names are:
+
+ * text: any text that might be a part of application name or a part of user name or a complete job ID
+ * name: the workflow application name from the workflow definition.
+ * user: the user that submitted the job.
+ * group: the group for the job.
+ * status: the status of the job.
+ * startcreatedtime: the start of time window in specifying createdtime range filter.
+ * endcreatedtime: the end of time window in specifying createdtime range filter
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
+
+The query will do an AND among all the filter names. The query will do an OR among all the filter values for the same
+name. Multiple values must be specified as different name value pairs.
+
+startCreatedTime and endCreatedTime should be specified either in **ISO8601 (UTC)** format (**yyyy-MM-dd'T'HH:mm'Z'**) or a offset value in days or hours from the current time. For example, -2d means the current time - 2 days. -3h means the current time - 3 hours, -5m means the current time - 5 minutes
+
+### Checking the Status of multiple Coordinator Jobs
+
+* This feature is only supported in Oozie 2.0 or later.
+
+Example:
+
+
+```
+$ oozie jobs -oozie http://localhost:11000/oozie -jobtype coordinator
+.
+Job ID App Name Status Freq Unit Started Next Materialized
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+0004165-100531045722929-oozie-wrkf-C smaggs-xaggsptechno-coordinator SUCCEEDED 1440 MINUTE 2010-05-27 00:00 2010-05-29 00:00
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+0003823-100531045722929-oozie-wrkf-C coordcal2880minutescurrent SUCCEEDED 2880 MINUTE 2010-02-01 16:30 2010-02-05 16:30
+.----------------------------------------------------------------------------------------------------------------------------------------------------------------
+```
+
+The `jobtype` option specified the job type to display, default value is 'wf'. To see the coordinator jobs, value is 'coordinator'.
+
+Valid filter names are:
+
+ * name: the workflow application name from the workflow definition.
+ * user: the user that submitted the job.
+ * group: the group for the job.
+ * status: the status of the job.
+ * frequency: the frequency of the Coordinator job.
+ * unit: the time unit. It can take one of the following four values: months, days, hours or minutes. Time unit should be added only when frequency is specified.
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
+
+### Checking the Status of multiple Bundle Jobs
+
+* This feature is only supported in Oozie 3.0 or later.
+
+Example:
+
+
+```
+$ oozie jobs -oozie http://localhost:11000/oozie -jobtype bundle
+Job ID Bundle Name Status Kickoff Created User Group
+.------------------------------------------------------------------------------------------------------------------------------------
+0000027-110322105610515-oozie-chao-B BUNDLE-TEST RUNNING 2012-01-15 00:24 2011-03-22 18:07 joe users
+.------------------------------------------------------------------------------------------------------------------------------------
+0000001-110322105610515-oozie-chao-B BUNDLE-TEST RUNNING 2012-01-15 00:24 2011-03-22 18:06 joe users
+.------------------------------------------------------------------------------------------------------------------------------------
+0000000-110322105610515-oozie-chao-B BUNDLE-TEST DONEWITHERROR2012-01-15 00:24 2011-03-22 17:58 joe users
+.------------------------------------------------------------------------------------------------------------------------------------
+```
+
+The `jobtype` option specified the job type to display, default value is 'wf'. To see the bundle jobs, value is 'bundle'.
+
+### Bulk kill, suspend or resume multiple jobs
+
+Example:
+
+
+```
+$ oozie jobs -oozie http://localhost:11000/oozie -kill|-suspend|-resume -filter name=cron-coord -jobtype coordinator
+The following jobs have been killed|suspended|resumed
+Job ID App Name Status Freq Unit Started Next Materialized
+.------------------------------------------------------------------------------------------------------------------------------------
+0000005-150224141553231-oozie-bzha-C cron-coord KILLED 10 MINUTE 2015-02-24 22:05 GMT 2015-02-24 23:05 GMT
+.------------------------------------------------------------------------------------------------------------------------------------
+0000001-150224141553231-oozie-bzha-C cron-coord KILLED 10 MINUTE 2015-02-24 22:00 GMT 2015-02-24 23:00 GMT
+.------------------------------------------------------------------------------------------------------------------------------------
+0000000-150224141553231-oozie-bzha-C cron-coord KILLED 10 MINUTE 2015-02-25 22:00 GMT -
+.------------------------------------------------------------------------------------------------------------------------------------
+```
+
+The above command will kill, suspend, or resume all the coordinator jobs with name of "cron-coord" starting with offset 1
+to 50.
+The `jobs` sub-command will bulk modify all the jobs that satisfy the filter, len, offset, and jobtype options when adding
+a -kill|-suspend|-resume option. Another way to think about is the subcommand works to modify all the jobs that will be
+displayed if the write option(kill|suspend|resume) is not there.
+
+The `offset` and `len` option specified the offset and number of jobs to be modified, default values are `1` and `50`
+respectively.
+The `jobtype` option specifies the type of jobs to be modified, be it "wf", "coordinator" or "bundle". default value is "wf".
+
+A filter can be specified after all options.
+
+The `filter`option syntax is: `[NAME=VALUE][;NAME=VALUE]*`.
+
+Valid filter names are:
+
+ * name: the workflow application name from the workflow definition.
+ * user: the user that submitted the job.
+ * group: the group for the job.
+ * status: the status of the job.
+ * frequency: the frequency of the Coordinator job.
+ * unit: the time unit. It can take one of the following four values: months, days, hours or minutes. Time unit should be added only when frequency is specified.
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
+
+The query will do an AND among all the filter names. The query will do an OR among all the filter values for the same
+name. Multiple values must be specified as different name value pairs.
+
+The following example shows how to suspend the first 20 bundle jobs whose name is "bundle-app":
+
+
+```
+$ oozie jobs -oozie http://localhost:11000/oozie -suspend -filter name=bundle-app -jobtype bundle -len 20
+```
+
+### Bulk monitoring for jobs launched via Bundles
+
+* This command-line query helps to directly query for a bulk of jobs using a set of rich filters.
+The jobs need to have a parent **Bundle**, and this performs a deep query to provide results about all the workflows that satisfy your filters.
+These results display relevant job-ids, app-names, and error message (if any) and are most helpful when you need to monitor a bulk of jobs and get a gist,
+while avoiding typing multiple queries.
+
+This feature is only supported in Oozie 3.3 or later.
+
+Example 1:
+
+
+```
+$ oozie jobs -oozie http://localhost:11000/oozie -bulk bundle=bundle-app-1
+.
+Bundle Name Bundle ID Coord Name Coord Action ID External ID Status Created Time Error Message
+.-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+bundle-app-1 0000000-130408151805946-oozie-chit-B coord-1 0000001-130408151805946-oozie-chit-C@1 0000002-130408151805946-oozie-chit-W KILLED 2013-04-08 22:20 GMT null
+.-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+```
+
+Example 2: This example illustrates giving multiple arguments and -verbose option.
+
+_NOTE: The filter string after -bulk should be enclosed within quotes_
+
+
+```
+.
+$ oozie jobs -oozie http://localhost:11000/oozie -bulk 'bundle=bundle-app-2;actionstatus=SUCCEEDED' -verbose
+.
+Bundle Name : bundle-app-2
+.------------------------------------------------------------------------------------------------------------------------------------
+Bundle ID : 0000000-130422184245158-oozie-chit-B
+Coordinator Name : coord-1
+Coord Action ID : 0000001-130422184245158-oozie-chit-C@1
+Action Status : SUCCEEDED
+External ID : 0000002-130422184245158-oozie-chit-W
+Created Time : 2013-04-23 01:43 GMT
+User : user_xyz
+Error Message : -
+.------------------------------------------------------------------------------------------------------------------------------------
+```
+
+You can type 'help' to view usage for the CLI options and filters available. Namely:
+
+ * bundle: Bundle app-name (mandatory)
+ * coordinators: multiple, comma-separated Coordinator app-names. By default, if none specified, all coordinators pertaining to the bundle are included
+ * actionstatus: status of jobs (default job-type is coordinator action aka workflow job) you wish to filter on. Default value is (KILLED,FAILED)
+ * startcreatedtime/endcreatedtime: specify boundaries for the created-time. Only jobs created within this window are included.
+ * startscheduledtime/endscheduledtime: specify boundaries for the nominal-time. Only jobs scheduled to run within this window are included.
+
+
+```
+$ oozie jobs <OPTIONS> : jobs status
+ -bulk <arg> key-value pairs to filter bulk jobs response. e.g.
+ bundle=<B>\;coordinators=<C>\;actionstatus=<S>\;
+ startcreatedtime=<SC>\;endcreatedtime=<EC>\;
+ startscheduledtime=<SS>\;endscheduledtime=<ES>\;
+ coordinators and actionstatus can be multiple comma
+ separated values. "bundle" and "coordinators" are 'names' of those jobs.
+ Bundle name is mandatory, other params are optional
+```
+
+Similar to the usual jobs filter, different filter arguments here should be separated by semicolon (;).
+
+## Admin Operations
+
+### Checking the Status of the Oozie System
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -status
+.
+Safemode: OFF
+```
+
+It returns the current status of the Oozie system.
+
+### Changing the Status of the Oozie System
+
+* This feature is only supported in Oozie 2.0 or later.
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -systemmode [NORMAL|NOWEBSERVICE|SAFEMODE]
+.
+Safemode: ON
+```
+
+It returns the current status of the Oozie system.
+
+### Displaying the Build Version of the Oozie System
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -version
+.
+Oozie server build version: 2.0.2.1-0.20.1.3092118008--
+```
+
+It returns the Oozie server build version.
+
+### Displaying the queue dump of the Oozie System
+
+* This feature is for administrator debugging purpose
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -queuedump
+.
+[Server Queue Dump]:
+(coord_action_start,1),(coord_action_start,1),(coord_action_start,1)
+(coord_action_ready,1)
+(action.check,2)
+
+```
+
+It returns the Oozie server current queued commands.
+
+### Displaying the list of available Oozie Servers
+
+Example:
+
+
+```
+$ oozie admin -oozie http://hostA:11000/oozie -servers
+hostA : http://hostA:11000/oozie
+hostB : http://hostB:11000/oozie
+hostC : http://hostC:11000/oozie
+```
+
+It returns a list of available Oozie Servers. This is useful when Oozie is configured for [High Availability](AG_Install.html#HA); if
+not, it will simply return the one Oozie Server.
+
+### Displaying the Oozie server configuration
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -configuration
+local.realm : LOCALHOST
+oozie.JobCommand.job.console.url : http://localhost:11000/oozie?job=
+oozie.action.fs.glob.max : 1000
+oozie.action.jobinfo.enable : false
+oozie.action.launcher.mapreduce.job.ubertask.enable : true
+oozie.action.launcher.yarn.timeline-service.enabled : false
+oozie.action.mapreduce.uber.jar.enable : false
+oozie.action.max.output.data : 2048
+oozie.action.retries.max : 3
+...
+```
+
+It returns a list of the configuration properties and values from oozie-site.xml and oozie-default.xml being used by the Oozie
+server.
+
+### Displaying the Oozie server OS environment
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -osenv
+...
+JETTY_OPTS : -Doozie.home.dir=/Users/asasvari/dev/oozie -Doozie.config.dir=/Users/asasvari/dev/oozie/conf -Doozie.log.dir=/Users/asasvari/dev/oozie/logs -Doozie.data.dir=/Users/asasvari/dev/oozie/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Djava.library.path= -cp /Users/asasvari/dev/oozie/embedded-oozie-server/**:/Users/asasvari/dev/oozie/embedded-oozie-server/dependency/**:/Users/asasvari/dev/oozie/lib/**:/Users/asasvari/dev/oozie/libtools/**:/Users/asasvari/dev/oozie/embedded-oozie-server
+JETTY_OUT : /Users/asasvari/dev/oozie/logs/jetty.out
+JETTY_PID_FILE : /Users/asasvari/dev/oozie/embedded-oozie-server/oozie.pid
+OOZIE_CONFIG : /Users/asasvari/dev/oozie/conf
+OOZIE_CONFIG_FILE : oozie-site.xml
+OOZIE_DATA : /Users/asasvari/dev/oozie/data
+OOZIE_HOME : /Users/asasvari/dev/oozie
+OOZIE_LOG : /Users/asasvari/dev/oozie/logs
+OOZIE_LOG4J_FILE : oozie-log4j.properties
+OOZIE_LOG4J_RELOAD : 10
+...
+```
+
+It returns a list of OS environment variables in the Oozie server.
+
+### Displaying the Oozie server Java system properties
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -javasysprops
+...
+oozie.config.dir : /Users/asasvari/dev/oozie/conf
+oozie.config.file : oozie-site.xml
+oozie.data.dir : /Users/asasvari/dev/oozie/data
+oozie.home.dir : /Users/asasvari/dev/oozie
+oozie.log.dir : /Users/asasvari/dev/oozie/logs
+oozie.log4j.file : oozie-log4j.properties
+oozie.log4j.reload : 10
+...
+```
+
+It returns a list of java system properties in the Oozie server.
+
+### Displaying the Oozie server Instrumentation
+
+Deprecated and by default disabled since 5.0.0.
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -instrumentation
+COUNTERS
+--------
+callablequeue.executed : 1
+callablequeue.queued : 1
+commands.purge.executions : 1
+...
+
+VARIABLES
+---------
+configuration.action.types : [hive, shell, :START:, :FORK:, switch, spark, ssh, hive2, pig, :END:, email, distcp, :KILL:, sub-workflow, fs, java, :JOIN:, sqoop, map-reduce]
+configuration.config.dir : /Users/rkanter/dev/oozie/conf
+configuration.config.file : /Users/rkanter/dev/oozie/conf/oozie-site.xml
+...
+
+SAMPLERS
+---------
+callablequeue.queue.size : 0.0
+callablequeue.threads.active : 0.0
+jdbc.connections.active : 0.0
+...
+
+TIMERS
+---------
+callablequeue.time.in.queue
+ own time standard deviation : -1.0
+ own average time : 1
+ own max time : 1
+ own min time : 1
+ total time standard deviation : -1.0
+ total average time : 1
+ total max time : 1
+ total min time : 1
+ ticks : 1
+commands.purge.call
+ own time standard deviation : -1.0
+ own average time : 222
+ own max time : 222
+ own min time : 222
+ total time standard deviation : -1.0
+ total average time : 222
+ total max time : 222
+ total min time : 222
+ ticks : 1
+...
+```
+
+It returns the instrumentation from the Oozie server. Keep in mind that timers and counters that the Oozie server
+hasn't incremented yet will not show up.
+
+**Note:** If Instrumentation is enabled, then Metrics is unavailable.
+
+### Displaying the Oozie server Metrics
+
+By default enabled since 5.0.0.
+
+Example:
+
+
+```
+$ oozie admin -oozie http://localhost:11000/oozie -metrics
+COUNTERS
+--------
+callablequeue.executed : 1
+callablequeue.queued : 1
+commands.purge.executions : 1
+...
+
+GAUGES
+------
+configuration.action.types : [hive, shell, :START:, :FORK:, switch, spark, ssh, hive2, pig, :END:, email, distcp, :KILL:, sub-workflow, fs, java, :JOIN:, sqoop, map-reduce]
+configuration.config.dir : /Users/rkanter/dev/oozie/conf
+configuration.config.file : /Users/rkanter/dev/oozie/conf/oozie-site.xml
+...
+
+TIMERS
+------
+callablequeue.time.in.queue.timer
+ 999th percentile : 4.0
+ 99th percentile : 4.0
+ 98th percentile : 4.0
+ 95th percentile : 4.0
+ 75th percentile : 4.0
+ 50th percentile : 4.0
+ mean : 4.0
+ max : 4.0
+ min : 4.0
+ count : 1
+ standard deviation : 0.0
+ 15 minute rate : 0.0
+ 5 minute rate : 0.0
+ 1 minute rate : 0.0
+ mean rate : 0.0
+ duration units : milliseconds
+ rate units : calls/millisecond
+commands.purge.call.timer
+ 999th percentile : 260.0
+ 99th percentile : 260.0
+ 98th percentile : 260.0
+ 95th percentile : 260.0
+ 75th percentile : 260.0
+ 50th percentile : 260.0
+ mean : 260.0
+ max : 260.0
+ min : 260.0
+ count : 1
+ standard deviation : 0.0
+ 15 minute rate : 0.0
+ 5 minute
<TRUNCATED>
[06/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_CoordinatorRerun.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_CoordinatorRerun.twiki b/docs/src/site/twiki/DG_CoordinatorRerun.twiki
deleted file mode 100644
index f535d16..0000000
--- a/docs/src/site/twiki/DG_CoordinatorRerun.twiki
+++ /dev/null
@@ -1,53 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Coordinator Rerun
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Pre-Conditions
-
- * Rerun coordinator action must be in TIMEDOUT/SUCCEEDED/KILLED/FAILED.
- * Coordinator actions cannot be rerun if the coordinator job is in the PREP or IGNORED state.
- * Rerun a PAUSED coordinator job, the status and pause time and pending flag will not be reset.
- * Rerun a SUSPENDED coordinator job, the status will reset to RUNNING.
- * All rerun actions must exist already.
- * Coordinator Rerun will only use the original configs from first run.
- * Coordinator Rerun will not re-read the coordinator.xml in hdfs.
-
-## Rerun Arguments
-
-
-```
-$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>]
-[-action 1, 3-4, 7-40] (-action or -date is required to rerun.)
-[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
-(if neither -action nor -date is given, the exception will be thrown.)
-```
-
- * Either -action or -date should be given.
- * If -action and -date both are given, an error will be thrown.
- * Multiple ranges can be used in -action or -date. See the above examples.
- * If one of action in the given list of -action does not exist or not in terminal state, the rerun throws an error.
- * The dates specified in -date must be UTC.
- * Single date specified in -date must be able to find an action with matched nominal time to be effective.
- * If -nocleanup is given, coordinator directories will not be removed; otherwise the 'output-event' will be deleted, unless nocleanup attribute is explicitly set in coordinator.xml
- * If -refresh is set, new dataset is re-evaluated for latest() and future().
- * If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked.
- * If -failed is set, re-runs the failed workflow actions of the coordinator actions.
- * -config can be used to supply properties to workflow by job configuration file '.xml' or '.properties'.
-
-## Rerun coordinator actions
-
- * Rerun terminated (timeout, succeeded, killed, failed) coordinator actions.
- * By default, Oozie will delete the 'output-event' directories before changing actions' status and materializing actions.
- * If coordinator job is RUNNING, rerun actions will be materialized and compete with current running actions.
- * Rerun for job, user should use job's start date and end date in -date.
- * If the user specifies a date range (say Jan 1 to May 1), the actions that will be re-run are the existing actions
- within that range. If the existing actions are action #5....#40, which map to Jan 15 to Feb 15, then only those actions will run.
- * The rerun action_id and nominal_time of the actions which are eligible to rerun will be returned.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_CustomActionExecutor.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_CustomActionExecutor.twiki b/docs/src/site/twiki/DG_CustomActionExecutor.twiki
deleted file mode 100644
index 5768b27..0000000
--- a/docs/src/site/twiki/DG_CustomActionExecutor.twiki
+++ /dev/null
@@ -1,83 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Custom Action Nodes
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Introduction
-Oozie can be extended to support additional action types by writing a custom [Action Node](WorkflowFunctionalSpec.html#ActionNodes). Action Nodes can be synchronous or asynchronous.
-
- * Synchronous Node - Sync nodes are executed inline by Oozie, which waits for completion of these nodes before proceeding. Hence, these nodes should almost never be used and are meant for lightweight tasks like FileSystem move, mkdir, delete.
- * Asynchronous Nodes - Oozie starts asynchronous nodes, and then monitors the action being executed for completion. This is done via a callback from the action or Oozie polling for the action status.
-
-## Writing a custom Action Node
-Action Executors are configured in the oozie configuration file oozie-site.xml. These executors are loaded during Oozie startup. [Deploying a Custom Action Executor](DG_CustomActionExecutor.html#Deploying_a_custom_Action_Executor).
-
-Action Executors MUST extend the `ActionExecutor` class and override the required methods.
-
-Most methods take as argument the Execution Context and the actual Action object with various configuration properties resolved.
-### ActionExecutor.Context
-The Execution context gives Action Nodes access to configuration properties, methods to set the state of the action, methods to set variables which are to be made available later in the execution path.
-
-**The following methods from the ActionExecutor interface should be implemented.**
-### Constructor
-A no argument constructor should be implemented, which calls super(ACTION_TYPE). ACTION_TYPE is the name of the action which will be used in the workflow xml, and is used by Oozie to instantiate the correct type of Executor.
-
-### initActionType()
-This method is called once during initialization of the Action Executor during Oozie startup. Any common initialization code for the Action Node should go here.
-
-As an example, setting up of error handling for the Custom Action should be done here.
-
-This method must call super.initActionType() as it's first statement.
-
-### start(ActionExecutor.Context context, Action action)
-The action start up happens here.
-
- * Async Actions - The action should be started and context.setStartData(externalId, trackerUri, consoleUrl) must be set. A check can be made for whether the action has completed, in which case context.setExecutionData(externalStatus, actionData) must be called.
- * Sync Actions - The action should be started and should complete execution. context.setExecutionData(externalStatus, actionData) must be called.
-### check(ActionExecutor.Context context, Action action)
-`check(...)` is used by Oozie to poll for the status of the action. This method should interact with the action started previously, and update the status. If the action has completed, context.setExecutionData(externalStatus, actionData) must be called. Otherwise, the status can be updated using context.setExternalStatus(externalStatus).
-
-For sync actions, this method will not be called, and should throw an UnsupportedOperationException().
-### kill(ActionExecutor.Context context, Action action)
-`kill(...)` is called when there is an attempt to kill the running job or action. No workflow transition is made after this.
-
-The implementation for a custom action should interact with and kill the running action, and take care of any cleanup which may be required. context.setEndData(status, signalValue) should be called with both values set to Action.Status.KILLED.
-### end(ActionExecutor.Context context, Action action)
-`end(...)` is used for any cleanup or processing which may need to be done after completion of the action. After any processing, context.setEndData(status, signalValue) should be called to complete execution of the action and trigger the next workflow transition. signalValue can be Action.Status.OK or Action.Status.ERROR.
-### Registering Errors
-Oozie actions can generate different types of Errors.
-
- * TRANSIENT - will be retried
- * NON TRANSIENT - the job will be suspended and can be resumed later by human intervention, after fixing whatever problem caused this error.
- * ERROR - causes the error transition to be taken.
- * FAILED - the action and the job are set to FAILED state. No transitions are taken.
-registerError(exceptionClassName, errorType, errorMessage) can be used to register possible exceptions while executing the action, along with their type and error message. This will normally be done during initialization of the Action Executor.
-
-## Deploying a custom Action Executor
-Action Nodes can be registered in the oozie configuration file oozie-site.xml, by changing the property 'oozie.service.ActionService.executor.ext.classes'. For multiple Executors, the class name should be separated by commas.
-
-``` <property>
- <name>oozie.service.ActionService.executor.ext.classes</name>
- <value>
- org.apache.oozie.wf.action.decision.CustomActionExecutor,
- Custom_Action_Executr_2.class
- </value>
- </property>
-```
-Any configuration properties to be made available to this class should also be added to oozie-site.xml. The convention to be followed for naming these properties is 'oozie.action.[ActionName].property.name'
-
-The XML schema (XSD) for the new Actions should be added to oozie-site.xml, under the property 'oozie.service.WorkflowSchemaService.ext.schemas'. A comma separated list for multiple Action schemas.
-
-The XML schema (XSD) for the new action should be also added to Fluent Job API. Please refer to
-[Fluent Job API :: How To Extend](DG_FluentJobAPI.html#AE.C_Appendix_C_How_To_Extend) for details.
-
-The executor class should be placed along with the oozie webapp in the correct path. Once Oozie is restarted, the custom action node can be used in workflows.
-
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_DistCpActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_DistCpActionExtension.twiki b/docs/src/site/twiki/DG_DistCpActionExtension.twiki
deleted file mode 100644
index 13c2a0a..0000000
--- a/docs/src/site/twiki/DG_DistCpActionExtension.twiki
+++ /dev/null
@@ -1,220 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie DistCp Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## DistCp Action
-
-The `DistCp` action uses Hadoop distributed copy to copy files from one cluster to another or within the same cluster.
-
-**IMPORTANT:** The DistCp action may not work properly with all configurations (secure, insecure) in all versions
-of Hadoop. For example, distcp between two secure clusters is tested and works well. Same is true with two insecure
-clusters. In cases where a secure and insecure clusters are involved, distcp will not work.
-
-Both Hadoop clusters have to be configured with proxyuser for the Oozie process as explained
-[here](DG_QuickStart.html#HadoopProxyUser) on the Quick Start page.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="distcp-example">
- <distcp xmlns="uri:oozie:distcp-action:1.0">
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode1}</name-node>
- <arg>${nameNode1}/path/to/input.txt</arg>
- <arg>${nameNode2}/path/to/output.txt</arg>
- </distcp>
- <ok to="end"/>
- <error to="fail"/>
- </action>
- ...
-</workflow-app>
-```
-
-The first `arg` indicates the input and the second `arg` indicates the output. In the above example, the input is on `namenode1`
-and the output is on `namenode2`.
-
-**IMPORTANT:** If using the DistCp action between 2 secure clusters, the following property must be added to the `configuration` of
-the action:
-
-```
-<property>
- <name>oozie.launcher.mapreduce.job.hdfs-servers</name>
- <value>${nameNode1},${nameNode2}</value>
-</property>
-```
-
-The `DistCp` action is also commonly used to copy files within the same cluster. Cases where copying files within
-a directory to another directory or directories to target directory is supported. Example below will illustrate a
-copy within a cluster, notice the source and target `nameNode` is the same and use of `*` syntax is supported to
-represent only child files or directories within a source directory. For the sake of the example, `jobTracker` and `resourceManager`
-are synonymous.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="copy-example">
- <distcp xmlns="uri:oozie:distcp-action:1.0">
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode}</name-node>
- <arg>${nameNode}/path/to/source/*</arg>
- <arg>${nameNode}/path/to/target/</arg>
- </distcp>
- <ok to="end"/>
- <error to="fail"/>
- </action>
- ...
-</workflow-app>
-```
-
-## Appendix, DistCp XML-Schema
-
-### AE.A Appendix A, DistCp XML-Schema
-
-#### DistCp Action Schema Version 1.0
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:distcp="uri:oozie:distcp-action:1.0" elementFormDefault="qualified"
- targetNamespace="uri:oozie:distcp-action:1.0">
-.
- <xs:include schemaLocation="oozie-common-1.0.xsd"/>
-.
- <xs:element name="distcp" type="distcp:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:choice>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="launcher" type="distcp:LAUNCHER" minOccurs="0" maxOccurs="1"/>
- <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### DistCp Action Schema Version 0.2
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:distcp="uri:oozie:distcp-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:distcp-action:0.2">
-.
- <xs:element name="distcp" type="distcp:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="distcp:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="distcp:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### DistCp Action Schema Version 0.1
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:distcp="uri:oozie:distcp-action:0.1" elementFormDefault="qualified"
- targetNamespace="uri:oozie:distcp-action:0.1">
-.
- <xs:element name="distcp" type="distcp:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="distcp:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="configuration" type="distcp:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="java-opts" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="arg" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="distcp:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="distcp:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_EmailActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_EmailActionExtension.twiki b/docs/src/site/twiki/DG_EmailActionExtension.twiki
deleted file mode 100644
index 1afcbb4..0000000
--- a/docs/src/site/twiki/DG_EmailActionExtension.twiki
+++ /dev/null
@@ -1,136 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Email Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-<a name="EmailAction"></a>
-## 3.2.4 Email action
-
-The `email` action allows sending emails in Oozie from a workflow application. An email action must provide `to`
-addresses, `cc` addresses (optional), `bcc` addresses (optional), a `subject` and a `body`.
-Multiple recipients of an email can be provided as comma separated addresses.
-
-The email action is executed synchronously, and the workflow job will wait until the specified
-emails are sent before continuing to the next action.
-
-All values specified in the `email` action can be parameterized (templatized) using EL expressions.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1">
- ...
- <action name="[NODE-NAME]">
- <email xmlns="uri:oozie:email-action:0.2">
- <to>[COMMA-SEPARATED-TO-ADDRESSES]</to>
- <cc>[COMMA-SEPARATED-CC-ADDRESSES]</cc> <!-- cc is optional -->
- <bcc>[COMMA-SEPARATED-BCC-ADDRESSES]</bcc> <!-- bcc is optional -->
- <subject>[SUBJECT]</subject>
- <body>[BODY]</body>
- <content_type>[CONTENT-TYPE]</content_type> <!-- content_type is optional -->
- <attachment>[COMMA-SEPARATED-HDFS-FILE-PATHS]</attachment> <!-- attachment is optional -->
- </email>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `to` and `cc` and `bcc` commands are used to specify recipients who should get the mail. Multiple email recipients
-can be provided using comma-separated values. Providing a `to` command is necessary, while the `cc` or `bcc` may
-optionally be used along.
-
-The `subject` and `body` commands are used to specify subject and body of the mail.
-From uri:oozie:email-action:0.2 one can also specify mail content type as <content_type>text/html</content_type>.
-"text/plain" is default.
-
-The `attachment` is used to attach a file(s) on HDFS to the mail. Multiple attachment can be provided using comma-separated values.
-Non fully qualified path is considered as a file on default HDFS. A local file cannot be attached.
-
-**Configuration**
-
-The `email` action requires some SMTP server configuration to be present (in oozie-site.xml). The following are the values
-it looks for:
-
- * `oozie.email.smtp.host` - The host where the email action may find the SMTP server (localhost by default).
- * `oozie.email.smtp.port` - The port to connect to for the SMTP server (25 by default).
- * `oozie.email.from.address` - The from address to be used for mailing all emails (oozie@localhost by default).
- * `oozie.email.smtp.auth` - Boolean property that toggles if authentication is to be done or not. (false by default).
- * `oozie.email.smtp.starttls.enable` - Boolean property that toggles if use TLS communication or not. (false by default).
- * `oozie.email.smtp.username` - If authentication is enabled, the username to login as (empty by default).
- * `oozie.email.smtp.password` - If authentication is enabled, the username's password (empty by default).
- * `oozie.email.attachment.enabled` - Boolean property that toggles if configured attachments are to be placed into the emails.
- (false by default).
- * `oozie.email.smtp.socket.timeout.ms` - The timeout to apply over all SMTP server socket operations (10000ms by default).
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
- ...
- <action name="an-email">
- <email xmlns="uri:oozie:email-action:0.1">
- <to>bob@initech.com,the.other.bob@initech.com</to>
- <cc>will@initech.com</cc>
- <bcc>yet.another.bob@initech.com</bcc>
- <subject>Email notifications for ${wf:id()}</subject>
- <body>The wf ${wf:id()} successfully completed.</body>
- </email>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-In the above example, an email is sent to 'bob', 'the.other.bob', 'will' (cc), yet.another.bob (bcc)
-with the subject and body both containing the workflow ID after substitution.
-
-## AE.A Appendix A, Email XML-Schema
-
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:email="uri:oozie:email-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:email-action:0.2">
-.
- <xs:element name="email" type="email:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="to" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="cc" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="bcc" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="subject" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="body" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="content_type" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="attachment" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
-</xs:schema>
-```
-
-**GMail example to oozie-site.xml**
-
-
-```
-oozie.email.smtp.host=smtp.gmail.com
-oozie.email.smtp.port=587
-oozie.email.from.address=<some email address>
-oozie.email.smtp.auth=true
-oozie.email.smtp.starttls.enable=true
-oozie.email.smtp.username=<Gmail Id>
-oozie.email.smtp.password=<Gmail Pass>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_Examples.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_Examples.twiki b/docs/src/site/twiki/DG_Examples.twiki
deleted file mode 100644
index ff33506..0000000
--- a/docs/src/site/twiki/DG_Examples.twiki
+++ /dev/null
@@ -1,204 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Oozie Examples
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Command Line Examples
-
-### Setting Up the Examples
-
-Oozie examples are bundled within the Oozie distribution in the `oozie-examples.tar.gz` file.
-
-Expanding this file will create an `examples/` directory in the local file system.
-
-The `examples/` directory must be copied to the user HOME directory in HDFS:
-
-
-```
-$ hadoop fs -put examples examples
-```
-
-**NOTE:** If an examples directory already exists in HDFS, it must be deleted before copying it again. Otherwise files may not be
-copied.
-
-### Running the Examples
-
-For the Streaming and Pig example, the [Oozie Share Library](DG_QuickStart.html#OozieShareLib) must be installed in HDFS.
-
-Add Oozie `bin/` to the environment PATH.
-
-The examples assume the ResourceManager is `localhost:8032` and the NameNode is `hdfs://localhost:8020`. If the actual
-values are different, the job properties files in the examples directory must be edited to the correct values.
-
-The example applications are under the examples/app directory, one directory per example. The directory contains the
-application XML file (workflow, or workflow and coordinator), the `job.properties` file to submit the job and any JAR
-files the example may need.
-
-The inputs for all examples are in the `examples/input-data/` directory.
-
-The examples create output under the `examples/output-data/${EXAMPLE_NAME}` directory.
-
-**Note**: The `job.properties` file needs to be a local file during submissions, and not a HDFS path.
-
-**How to run an example application:**
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
-.
-job: 14-20090525161321-oozie-tucu
-```
-
-Check the workflow job status:
-
-
-```
-$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-tucu
-.
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-Workflow Name : map-reduce-wf
-App Path : hdfs://localhost:8020/user/tucu/examples/apps/map-reduce
-Status : SUCCEEDED
-Run : 0
-User : tucu
-Group : users
-Created : 2009-05-26 05:01 +0000
-Started : 2009-05-26 05:01 +0000
-Ended : 2009-05-26 05:01 +0000
-Actions
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-Action Name Type Status Transition External Id External Status Error Code Start Time End Time
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-mr-node map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000
-.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-```
-
-To check the workflow job status via the Oozie web console, with a browser go to `http://localhost:11000/oozie`.
-
-To avoid having to provide the `-oozie` option with the Oozie URL with every `oozie` command, set `OOZIE_URL` env
-variable to the Oozie URL in the shell environment. For example:
-
-
-```
-$ export OOZIE_URL="http://localhost:11000/oozie"
-$
-$ oozie job -info 14-20090525161321-oozie-tucu
-```
-
-## Java API Example
-
-Oozie provides a [Java Client API](./apidocs/org/org/apache/oozie/client/package-summary.html) that simplifies
-integrating Oozie with Java applications. This Java Client API is a convenience API to interact with Oozie Web-Services
-API.
-
-The following code snippet shows how to submit an Oozie job using the Java Client API.
-
-
-```
-import org.apache.oozie.client.OozieClient;
-import org.apache.oozie.client.WorkflowJob;
-.
-import java.util.Properties;
-.
- ...
-.
- // get a OozieClient for local Oozie
- OozieClient wc = new OozieClient("http://bar:11000/oozie");
-.
- // create a workflow job configuration and set the workflow application path
- Properties conf = wc.createConfiguration();
- conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app");
-.
- // setting workflow parameters
- conf.setProperty("resourceManager", "foo:8032");
- conf.setProperty("inputDir", "/usr/tucu/inputdir");
- conf.setProperty("outputDir", "/usr/tucu/outputdir");
- ...
-.
- // submit and start the workflow job
- String jobId = wc.run(conf);
- System.out.println("Workflow job submitted");
-.
- // wait until the workflow job finishes printing the status every 10 seconds
- while (wc.getJobInfo(jobId).getStatus() == Workflow.Status.RUNNING) {
- System.out.println("Workflow job running ...");
- Thread.sleep(10 * 1000);
- }
-.
- // print the final status of the workflow job
- System.out.println("Workflow job completed ...");
- System.out.println(wf.getJobInfo(jobId));
- ...
-```
-
-## Local Oozie Example
-
-Oozie provides an embedded Oozie implementation, [LocalOozie](./apidocs/org/apache/oozie/local/LocalOozie.html) ,
-which is useful for development, debugging and testing of workflow applications within the convenience of an IDE.
-
-The code snippet below shows the usage of the `LocalOozie` class. All the interaction with Oozie is done using Oozie
- `OozieClient` Java API, as shown in the previous section.
-
-The examples bundled with Oozie include the complete and running class, `LocalOozieExample` from where this snippet was
-taken.
-
-
-```
-import org.apache.oozie.local.LocalOozie;
-import org.apache.oozie.client.OozieClient;
-import org.apache.oozie.client.WorkflowJob;
-.
-import java.util.Properties;
-.
- ...
- // start local Oozie
- LocalOozie.start();
-.
- // get a OozieClient for local Oozie
- OozieClient wc = LocalOozie.getClient();
-.
- // create a workflow job configuration and set the workflow application path
- Properties conf = wc.createConfiguration();
- conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app");
-.
- // setting workflow parameters
- conf.setProperty("resourceManager", "foo:8032");
- conf.setProperty("inputDir", "/usr/tucu/inputdir");
- conf.setProperty("outputDir", "/usr/tucu/outputdir");
- ...
-.
- // submit and start the workflow job
- String jobId = wc.run(conf);
- System.out.println("Workflow job submitted");
-.
- // wait until the workflow job finishes printing the status every 10 seconds
- while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
- System.out.println("Workflow job running ...");
- Thread.sleep(10 * 1000);
- }
-.
- // print the final status of the workflow job
- System.out.println("Workflow job completed ...");
- System.out.println(wc.getJobInfo(jobId));
-.
- // stop local Oozie
- LocalOozie.stop();
- ...
-```
-
-Also asynchronous actions like FS action can be used / tested using `LocalOozie` / `OozieClient` API. Please see the module
-`oozie-mini` for details like `fs-decision.xml` workflow example.
-
-
-## Fluent Job API Examples
-
-There are some elaborate examples how to use the [Fluent Job API](DG_FluentJobAPI.html), under `examples/fluentjob/`. There are two
-simple examples covered under [Fluent Job API :: A Simple Example](DG_FluentJobAPI.html#A_Simple_Example) and
-[Fluent Job API :: A More Verbose Example](DG_FluentJobAPI.html#A_More_Verbose_Example).
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_FluentJobAPI.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_FluentJobAPI.twiki b/docs/src/site/twiki/DG_FluentJobAPI.twiki
deleted file mode 100644
index bd36517..0000000
--- a/docs/src/site/twiki/DG_FluentJobAPI.twiki
+++ /dev/null
@@ -1,384 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Fluent Job API
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Introduction
-
-Oozie is a mature workflow scheduler system. XML is the standard way of defining workflow, coordinator, or bundle jobs. For users
-who prefer an alternative, the Fluent Job API provides a Java interface instead.
-
-### Motivation
-
-Prior to Oozie 5.1.0, the following ways were available to submit a workflow, coordinator, or bundle job: through Oozie CLI or via
-HTTP submit a generic workflow, coordinator, or bundle job, or submit a Pig, Hive, Sqoop, or MapReduce workflow job.
-
-As the generic way goes, the user has to have uploaded a workflow, coordinator, or bundle XML and all necessary dependencies like
-scripts, JAR or ZIP files, to HDFS beforehand, as well as have a `job.properties` file at command line and / or provide any
-missing parameters as part of the command.
-
-As the specific Pig, Hive, or Sqoop ways go, the user can provide all necessary parameters as part of the command issued. A
- `workflow.xml` file will be generated with all the necessary details and stored to HDFS so that Oozie can grab it. Note that
-dependencies have to be uploaded to HDFS beforehand as well.
-
-There are some usability problems by using the XML job definition. XML is not an ideal way to express dependencies and a directed
-acyclic graph (DAG). We have to define a control flow, that is, which action follows the actual one. It's also necessary to build
-the whole control flow up front as XML is a declarative language that doesn't allow for dynamic evaluation. We have to define also
-boilerplate actions like start and end - those are present in every Oozie workflow, still need to explicitly define these.
-
-Apart from boilerplate actions, all the transitions between actions have also to be defined and taken care of. Furthermore, multiple
-similar actions cannot inherit common properties from each other. Again, the reason being workflows are defined in XML.
-
-Fork and join actions have to be defined in pairs, that is, there shouldn't be defined a join those incoming actions do not share
-the same ancestor fork. Such situations would result still in a DAG, but Oozie doesn't currently allow that. Note that with Fluent
-Job API new dependencies are introduced automatically when the DAG represented by API code couldn't have been expressed as
-fork / join pairs automatically.
-
-Either way, there were no programmatic ways to define workflow jobs. That doesn't mean users could not generate XML themselves -
-actually this is something HUE's Oozie UI also tries to target.
-
-### Goals
-
-Fluent Job API aims to solve following from the user's perspective. It provides a Java API instead of declarative XML to define
-workflows. It defines dependencies across actions as opposed to defining a control flow. This is how data engineers and data
-scientists think. It eliminates all boilerplate actions and transitions. Only the necessary bits should be defined.
-
-Multiple similar actions can inherit from each other. In fact, since Fluent Job API is programmatic, it's possible to generate
-actions or even workflows using conditional, iterative, or recursive structures.
-
-Fluent Job API is backwards compatible with workflows defined as XML. That is, it should also be possible to have a Fluent Job API
-workflow rendered as XML, as well as coexist XML based and Fluent Job API based workflows in the same Oozie installation at the same
-time all workflow action types. When XSDs change, as few manual steps are necessary as possible both on API internal and public
-side.
-
-### Non-goals
-
-The following points are not targeted for the initial release of Fluent Job API with Oozie 5.1.0. It doesn't provide API in any
-language other than Java. It doesn't provide a REPL. It doesn't allow for dynamic action instantiation depending on e.g. conditional
-logic. That is, using the API users still have to implement the whole workflow generation logic in advance.
-
-It has no support for programmatic coordinators and bundles, or even EL expressions created by API builders. Note that EL
-expressions for workflows can now be expressed the way these are used in XML workflow definitions, as strings in the right places.
-
-At the moment only the transformation from Fluent Job API to workflow definition is present. The other direction, from workflow
-definition to Fluent Job API JAR artifact, though sensible, is not supported.
-
-It's based only on latest XSDs. Older XSD versions, as well as conversion between XSD versions are not supported. Also no support
-for user-supplied custom actions / XSDs.
-
-Most of the non-goals may be targeted as enhancements of the Fluent Job API for future Oozie releases.
-
-### Approach
-
-When using the Fluent Job API, the following points are different from the XML jobs definition. Instead of control flow (successor)
-definition, the user can define dependencies (parents of an action).
-
-All boilerplate (start, end, ...) has been eliminated, only nodes having useful actions have to be defined.
-
-Control flow and necessary boilerplate are generated automatically by keeping user defined dependencies, and possibly introducing
-new dependencies to keep Oozie workflow format of nested fork / join pairs. Note that not every dependency DAG can be expressed in
-the Oozie workflow format. When this is not possible, user is notified at build time.
-
-## How To Use
-
-### A Simple Example
-
-The simplest thing to create using the Oozie Fluent Job API is a workflow consisting of only one action. Let's see how it goes, step
-by step.
-
-First, put the project `org.apache.oozie:oozie-fluent-job-api` to the build path. In case of a Maven managed build, create a new
-Maven project and declare a Maven dependency to `org.apache.oozie:oozie-fluent-job-api`.
-
-Then, create a class that `implements WorkflowFactory` and implement the method `WorkflowFactory#create()`. inside that method,
-create a `ShellAction` using `ShellActionBuilder`, fill in some attributes then create a `Workflow` using `WorkflowBuilder` using
-the `ShellAction` just built. Return the `Workflow`.
-
-Compile a Fluent Job API jar that has the `Main-Class` attribute set to the `WorkflowFactory` subclass just created,
-e.g. `shell-workflow.jar`.
-
-Moving on, [check via command line](DG_CommandLineTool.html#Checking_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file) that
-the compiled API JAR file is valid.
-
-As a finishing touch,
-[run via command line](DG_CommandLineTool.html#Running_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file) the Fluent Job API
-workflow.
-
-**For reference, a simplistic API JAR example consisting of a `Workflow` having only one `ShellAction`:**
-
-```
-public class MyFirstWorkflowFactory implements WorkflowFactory {
-.
- @Override
- public Workflow create() {
- final ShellAction shellAction = ShellActionBuilder.create()
- .withName("shell-action")
- .withResourceManager("${resourceManager}")
- .withNameNode("${nameNode}")
- .withConfigProperty("mapred.job.queue.name", "${queueName}")
- .withExecutable("echo")
- .withArgument("my_output=Hello Oozie")
- .withCaptureOutput(true)
- .build();
-.
- final Workflow shellWorkflow = new WorkflowBuilder()
- .withName("shell-workflow")
- .withDagContainingNode(shellAction).build();
-.
- return shellWorkflow;
- }
-}
-```
-
-**After check, the generated workflow XML looks like this:**
-
-```
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" xmlns:shell="uri:oozie:shell-action:1.0" name="shell-workflow">
-.
- <workflow:start to="parent"/>
-.
- <workflow:kill name="kill">
- <workflow:message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</workflow:message>
- </workflow:kill>
-.
- <workflow:action name="shell-action">
- <shell:shell>
- <shell:resource-manager>${resourceManager}</shell:resource-manager>
- <shell:name-node>${nameNode}</shell:name-node>
- <shell:configuration>
- <shell:property>
- <shell:name>mapred.job.queue.name</shell:name>
- <shell:value>${queueName}</shell:value>
- </shell:property>
- </shell:configuration>
- <shell:exec>echo</shell:exec>
- <shell:argument>my_output=Hello Oozie</shell:argument>
- <shell:capture-output/>
- </shell:shell>
- <workflow:ok to="end"/>
- <workflow:error to="kill"/>
- </workflow:action>
-.
- <workflow:end name="end"/>
-.
-</workflow:workflow-app>
-```
-
-
-### A More Verbose Example
-
-**Error handling**
-
-If you would like to provide some error handling in case of action failure, you should add an `ErrorHandler` to the `Node`
-representing the action. The error handler action will be added as the `"error-transition"` of the original action in the generated
-Oozie workflow XML. Both the `"ok-transition"` and the `"error-transition"` of the error handler action itself will lead to an
-autogenerated kill node.
-
-**Here you find an example consisting of a `Workflow` having three `ShellAction`s, an error handler `EmailAction`, and one `decision`
-to sort out which way to go:**
-
-```
-public class MySecondWorkflowFactory implements WorkflowFactory {
-.
- @Override
- public Workflow create() {
- final ShellAction parent = ShellActionBuilder.create()
- .withName("parent")
- .withResourceManager("${resourceManager}")
- .withNameNode("${nameNode}")
- .withConfigProperty("mapred.job.queue.name", "${queueName}")
- .withExecutable("echo")
- .withArgument("my_output=Hello Oozie")
- .withCaptureOutput(true)
- .withErrorHandler(ErrorHandler.buildAsErrorHandler(EmailActionBuilder.create()
- .withName("email-on-error")
- .withRecipient("somebody@apache.org")
- .withSubject("Workflow error")
- .withBody("Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]")))
- .build();
-.
- ShellActionBuilder.createFromExistingAction(parent)
- .withName("happy-path")
- .withParentWithCondition(parent, "${wf:actionData('parent')['my_output'] eq 'Hello Oozie'}")
- .withoutArgument("my_output=Hello Oozie")
- .withArgument("Happy path")
- .withCaptureOutput(null)
- .build();
-.
- ShellActionBuilder.createFromExistingAction(parent)
- .withName("sad-path")
- .withParentDefaultConditional(parent)
- .withArgument("Sad path")
- .build();
-.
- final Workflow workflow = new WorkflowBuilder()
- .withName("shell-example")
- .withDagContainingNode(parent).build();
-.
- return workflow;
- }
-}
-```
-
-**After check, the generated workflow XML looks like this:**
-
-```
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<workflow:workflow-app ... name="shell-example">
-.
- <workflow:start to="parent"/>
-.
- <workflow:kill name="kill">
- <workflow:message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</workflow:message>
- </workflow:kill>
-.
- <workflow:action name="email-on-error">
- <email:email>
- <email:to>somebody@apache.org</email:to>
- <email:subject>Workflow error</email:subject>
- <email:body>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</email:body>
- </email:email>
- <workflow:ok to="kill"/>
- <workflow:error to="kill"/>
- </workflow:action>
-.
- <workflow:action name="parent">
- <shell:shell>
- <shell:resource-manager>${resourceManager}</shell:resource-manager>
- <shell:name-node>${nameNode}</shell:name-node>
- <shell:configuration>
- <shell:property>
- <shell:name>mapred.job.queue.name</shell:name>
- <shell:value>${queueName}</shell:value>
- </shell:property>
- </shell:configuration>
- <shell:exec>echo</shell:exec>
- <shell:argument>my_output=Hello Oozie</shell:argument>
- <shell:capture-output/>
- </shell:shell>
- <workflow:ok to="decision1"/>
- <workflow:error to="email-on-error"/>
- </workflow:action>
-.
- <workflow:decision name="decision1">
- <workflow:switch>
- <workflow:case to="happy-path">${wf:actionData('parent')['my_output'] eq 'Hello Oozie'}</workflow:case>
- <workflow:default to="sad-path"/>
- </workflow:switch>
- </workflow:decision>
-.
- <workflow:action name="happy-path">
- <shell:shell>
- <shell:resource-manager>${resourceManager}</shell:resource-manager>
- <shell:name-node>${nameNode}</shell:name-node>
- <shell:configuration>
- <shell:property>
- <shell:name>mapred.job.queue.name</shell:name>
- <shell:value>${queueName}</shell:value>
- </shell:property>
- </shell:configuration>
- <shell:exec>echo</shell:exec>
- <shell:argument>Happy path</shell:argument>
- </shell:shell>
- <workflow:ok to="end"/>
- <workflow:error to="email-on-error"/>
- </workflow:action>
-.
- <workflow:action name="sad-path">
- <shell:shell>
- <shell:resource-manager>${resourceManager}</shell:resource-manager>
- <shell:name-node>${nameNode}</shell:name-node>
- <shell:configuration>
- <shell:property>
- <shell:name>mapred.job.queue.name</shell:name>
- <shell:value>${queueName}</shell:value>
- </shell:property>
- </shell:configuration>
- <shell:exec>echo</shell:exec>
- <shell:argument>my_output=Hello Oozie</shell:argument>
- <shell:argument>Sad path</shell:argument>
- <shell:capture-output/>
- </shell:shell>
- <workflow:ok to="end"/>
- <workflow:error to="email-on-error"/>
- </workflow:action>
-.
- <workflow:end name="end"/>
-.
-</workflow:workflow-app>
-```
-
-### Runtime Limitations
-
-Even if Fluent Job API tries to abstract away the task of assembly job descriptor XML files, there are some runtime
-limitations apart from the [non-goals section](DG_FluentJobAPI.html#Non-goals). All such limitations are based on the current
-implementations and subject to further improvements and fixes.
-
-There is only one `kill` possibility in every `workflow`. That is, there can be defined only one `action` to be executed just before
-any other `action` turns to be `kill`ed. Furthermore, `kill` goes to `end` directly. That means, there cannot be defined an
-intricate network of `kill` nodes, cascading sometimes to other `action` nodes, avoiding going to `end` in the first place.
-
-There are places where `decision` node generation fails, throwing an `Exception`. The problem is that during the transformation,
-Fluent Job API reaches a state where there is a `fork` that transitions to two `decision` nodes, which in turn split into two paths
-each. One of the paths from the first `decision` joins a path from the other `decision`, but the remaining conditional paths never
-meet. Therefore, not all paths originating from the `fork` converge to the same `join`.
-
-## Appendixes
-
-### AE.A Appendix A, API JAR format
-
-It's kept simple - all the necessary Java class files that are needed are packed into a JAR file, that has a `META-INF/MANIFEST.MF`
-with a single entry having the `Main-Class` attribute set to the fully qualified name of the entry class, the one that
-`implements WorkflowFactory`:
-
-```
-Main-Class: org.apache.oozie.jobs.api.factory.MyFirstWorkflowFactory
-```
-
-**An example of the command line assembly of such an API JAR:**
-
-```
-jar cfe simple-workflow.jar org.apache.oozie.fluentjob.api.factory.MyFirstWorkflowFactory \
--C /Users/forsage/Workspace/oozie/fluent-job/fluent-job-api/target/classes \
-org/apache/oozie/jobs/api/factory/MyFirstWorkflowFactory.class
-```
-
-### AE.B Appendix B, Some Useful Builder classes
-
-For a complete list of `Builder` classes, please have a look at `oozie-fluent-job-api` artifact's following packages:
-
- * `org.apache.oozie.fluentjob.api.action` - `ActionBuilder` classes
- * `org.apache.oozie.fluentjob.api.factory` - the single entry point, `WorkflowFactory` is here
- * `org.apache.oozie.fluentjob.api.workflow` - workflow related `Builder` classes
-
-On examples how to use these please see `oozie-examples` artifact's `org.apache.oozie.example.fluentjob` package.
-
-### AE.C Appendix C, How To Extend
-
-Sometimes there are new XSD versions of an existing custom or core workflow action, sometimes it's a new custom workflow action that
-gets introduced. In any case, Fluent Job API needs to keep up with the changes.
-
-Here are the steps needed:
-
- * in `fluent-job-api/pom.xml` extend or modify `jaxb2-maven-plugin` section `sources` by a new `source`
- * in `fluent-job-api/src/main/xjb/bindings.xml` extend by a new or modify an existing `jaxb:bindings`
- * in `fluent-job-api`, `org.apache.oozie.fluentjob.api.mapping` package, introduce a new or modify an existing `DozerConverter`
- * in `dozer_config.xml`, introduce a new or modify an existing `converter` inside `custom-converters`
- * in `fluent-job-api`, `org.apache.oozie.fluentjob.api.action`, introduce a new `Action` and a new `Builder`
- * write new / modify existing relevant unit and integration tests
-
-### AE.D Appendix D, API compatibility guarantees
-
-Fluent Job API is available beginning version 5.1.0. It's marked `@InterfaceAudience.Private` (intended for use in Oozie itself) and
-`@InterfaceStability.Unstable` (no stability guarantees are provided across any level of release granularity) to indicate that for
-the next few minor releases it's bound to change a lot.
-
-Beginning from around 5.4.0 planning the next phase, `@InterfaceStability.Evolving` (compatibility breaking only between minors),
-and a few minor releases later, `@InterfaceAudience.Public` (safe to use outside of Oozie).
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_HCatalogIntegration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_HCatalogIntegration.twiki b/docs/src/site/twiki/DG_HCatalogIntegration.twiki
deleted file mode 100644
index 5c592e8..0000000
--- a/docs/src/site/twiki/DG_HCatalogIntegration.twiki
+++ /dev/null
@@ -1,147 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# HCatalog Integration (Since Oozie 4.x)
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## HCatalog Overview
-HCatalog is a table and storage management layer for Hadoop that enables users with different data processing
-tools - Pig, MapReduce, and Hive - to more easily read and write data on the grid. HCatalog's table abstraction presents
-users with a relational view of data in the Hadoop distributed file system (HDFS).
-
-Read [HCatalog Documentation](http://incubator.apache.org/hcatalog/docs/r0.5.0/index.html) to know more about HCatalog.
-Working with HCatalog using pig is detailed in
-[HCatLoader and HCatStorer](http://incubator.apache.org/hcatalog/docs/r0.5.0/loadstore.html).
-Working with HCatalog using MapReduce directly is detailed in
-[HCatInputFormat and HCatOutputFormat](http://incubator.apache.org/hcatalog/docs/r0.5.0/inputoutput.html).
-
-### HCatalog notifications
- HCatalog provides notifications through a JMS provider like ActiveMQ when a new partition is added to a table in the
-database. This allows applications to consume those events and schedule the work that depends on them. In case of Oozie,
-the notifications are used to determine the availability of HCatalog partitions defined as data dependencies in the
-Coordinator and trigger workflows.
-
-Read [HCatalog Notification](http://incubator.apache.org/hcatalog/docs/r0.5.0/notification.html) to know more about
-notifications in HCatalog.
-
-## Oozie HCatalog Integration
- Oozie's Coordinators so far have been supporting HDFS directories as a input data dependency. When a HDFS URI
-template is specified as a dataset and input events are defined in Coordinator for the dataset, Oozie performs data
-availability checks by polling the HDFS directory URIs resolved based on the nominal time. When all the data
-dependencies are met, the Coordinator's workflow is triggered which then consumes the available HDFS data.
-
-With addition of HCatalog support, Coordinators also support specifying a set of HCatalog tables or table partitions as a dataset.
-The workflow is triggered when the HCatalog table partitions are available and the workflow actions can then read the
-partition data. A mix of HDFS and HCatalog dependencies can be specified as input data dependencies.
-Similar to HDFS directories, HCatalog table partitions can also be specified as output dataset events.
-
-With HDFS data dependencies, Oozie has to poll HDFS every time to determine the availability of a directory.
-If the HCatalog server is configured to publish partition availability notifications to a JMS provider, Oozie can be
-configured to subscribe to it and trigger jobs immediately. This pub-sub model reduces pressure on Namenode and also
-cuts down on delays caused by polling intervals.
-
-In the absence of a message bus in the deployment, Oozie will always
-poll the HCatalog server directly for partition availability with the same frequency as the HDFS polling. Even when
-subscribed to notifications, Oozie falls back to polling HCatalog server for partitions that were available before the
-coordinator action was materialized and to deal with missed notifications due to system downtimes. The frequency of the
-fallback polling is usually lower than the constant polling. Defaults are 10 minutes and 1 minute respectively.
-
-
-### Oozie Server Configuration
- Refer to [HCatalog Configuration](AG_Install.html#HCatalog_Configuration) section of [Oozie Install](AG_Install.html)
-documentation for the Oozie server side configuration required to support HCatalog table partitions as a data dependency.
-
-### HCatalog URI Format
-
-Oozie supports specifying HCatalog partitions as a data dependency through a URI notation. The HCatalog partition URI is
-used to identify a set of table partitions: `hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US`
-
-The format to specify a HCatalog table URI is:
-
-hcat://[metastore server]:[port]/[database name]/[table name]
-
-The format to specify a HCatalog table partition URI is:
-
-hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...
-
-For example,
-
-```
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
- <uri-template>
- hcat://myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA
- </uri-template>
- </dataset>
-```
-
-Post Oozie-4.3.0 release, Oozie also supports the multiple HCatalog servers in the URI. Each of the server needs to be
-separated by single comma (,).
-
-The format to specify a HCatalog table partition URI with multiple HCatalog server is:
-
-hcat://[metastore_server]:[port],[metastore_server]:[port]/[database_name]/[table_name]/[partkey1]=[value];[partkey2]=[value];...
-
-For example,
-
-```
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
- <uri-template>
- hcat://myhcatmetastore:9080,myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA
- </uri-template>
- </dataset>
-```
-
-The regex for parsing the multiple HCatalog URI is exposed via oozie-site.xml, So Users can modify if there is any
-requirement. Key for the regex is: `oozie.hcat.uri.regex.pattern`
-
-For example, following has multiple HCatalog URI with multiple HCatalog servers. To understand this, Oozie will split them into
-two HCatalog URIs. For splitting the URIs, above mentioned regex is used.
-
-`hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us,scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us`
-
-After split: (This is internal Oozie mechanism)
-
-`hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us`
-
-`scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us`
-
-<a name="HCatalogLibraries"></a>
-### HCatalog Libraries
-
-A workflow action interacting with HCatalog requires the following jars in the classpath:
-hcatalog-core.jar, hcatalog-pig-adapter.jar, webhcat-java-client.jar, hive-common.jar, hive-exec.jar,
-hive-metastore.jar, hive-serde.jar and libfb303.jar.
-hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath. The correct
-version of HCatalog and hive jars should be placed in classpath based on the version of HCatalog installed on the cluster.
-
-The jars can be added to the classpath of the action using one of the below ways.
-
- * You can place the jars and hive-site.xml in the system shared library. The shared library for a pig, hive or java action can be overridden to include hcatalog shared libraries along with the action's shared library. Refer to [Shared Libraries](WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3) for more information. The oozie-sharelib-[version].tar.gz in the oozie distribution bundles the required HCatalog jars in a hcatalog sharelib. If using a different version of HCatalog than the one bundled in the sharelib, copy the required HCatalog jars from such version into the sharelib.
- * You can place the jars and hive-site.xml in the workflow application lib/ path.
- * You can specify the location of the jar files in `archive` tag and the hive-site.xml in `file` tag in the corresponding pig, hive or java action.
-
-### Coordinator
-
-Refer to [Coordinator Functional Specification](CoordinatorFunctionalSpec.html) for more information about
-
- * how to specify HCatalog partitions as a data dependency using input dataset events
- * how to specify HCatalog partitions as output dataset events
- * the various EL functions available to work with HCatalog dataset events and how to use them to access HCatalog partitions in pig, hive or java actions in a workflow.
-
-### Workflow
-Refer to [Workflow Functional Specification](WorkflowFunctionalSpec.html) for more information about
-
- * how to drop HCatalog table/partitions in the prepare block of a action
- * the HCatalog EL functions available to use in workflows
-
-Refer to [Action Authentication](DG_ActionAuthentication.html) for more information about
-
- * how to access a secure HCatalog from any action (e.g. hive, pig, etc) in a workflow
-
-### Known Issues
- * When rerunning a coordinator action without specifying -nocleanup option if the 'output-event' are hdfs directories, then they are deleted. But if the 'output-event' is a hcatalog partition, currently the partition is not dropped.
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_Hive2ActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_Hive2ActionExtension.twiki b/docs/src/site/twiki/DG_Hive2ActionExtension.twiki
deleted file mode 100644
index d81ed02..0000000
--- a/docs/src/site/twiki/DG_Hive2ActionExtension.twiki
+++ /dev/null
@@ -1,315 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Hive 2 Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Hive 2 Action
-
-The `hive2` action runs Beeline to connect to Hive Server 2.
-
-The workflow job will wait until the Hive Server 2 job completes before
-continuing to the next action.
-
-To run the Hive Server 2 job, you have to configure the `hive2` action with the `resource-manager`, `name-node`, `jdbc-url`,
- `password` elements, and either Hive's `script` or `query` element, as well as the necessary parameters and configuration.
-
-A `hive2` action can be configured to create or delete HDFS directories
-before starting the Hive Server 2 job.
-
-Oozie EL expressions can be used in the inline configuration. Property
-values specified in the `configuration` element override values specified
-in the `job-xml` file.
-
-As with Hadoop `map-reduce` jobs, it is possible to add files and
-archives in order to make them available to Beeline. Refer to the
-[Adding Files and Archives for the Job](WorkflowFunctionalSpec.html#FilesArchives)
-section for more information about this feature.
-
-Oozie Hive 2 action supports Hive scripts with parameter variables, their
-syntax is `${VARIABLES}`.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <hive2 xmlns="uri:oozie:hive2-action:1.0">
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[HIVE SETTINGS FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <jdbc-url>[jdbc:hive2://HOST:10000/default]</jdbc-url>
- <password>[PASS]</password>
- <script>[HIVE-SCRIPT]</script>
- <param>[PARAM-VALUE]</param>
- ...
- <param>[PARAM-VALUE]</param>
- <argument>[ARG-VALUE]</argument>
- ...
- <argument>[ARG-VALUE]</argument>
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- </hive2>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-
-The `job-xml` element, if present, specifies a file containing configuration
-for Beeline. Multiple `job-xml` elements are allowed in order to specify multiple `job.xml` files.
-
-The `configuration` element, if present, contains configuration
-properties that are passed to the Beeline job.
-
-The `jdbc-url` element must contain the JDBC URL for the Hive Server 2. Beeline will use this to know where to connect to.
-
-The `password` element must contain the password of the current user. However, the `password` is only used if Hive Server 2 is
-backed by something requiring a password (e.g. LDAP); non-secured Hive Server 2 or Kerberized Hive Server 2 don't require a password
-so in those cases the `password` is ignored and can be omitted from the action XML. It is up to the user to ensure that a password
-is specified when required.
-
-The `script` element must contain the path of the Hive script to
-execute. The Hive script can be templatized with variables of the form
-`${VARIABLE}`. The values of these variables can then be specified
-using the `params` element.
-
-The `query` element available from uri:oozie:hive2-action:0.2, can be used instead of the `script` element. It allows for embedding
-queries within the `worklfow.xml` directly. Similar to the `script` element, it also allows for the templatization of variables
-in the form `${VARIABLE}`.
-
-The `params` element, if present, contains parameters to be passed to
-the Hive script.
-
-The `argument` element, if present, contains arguments to be passed as-is to Beeline.
-
-All the above elements can be parameterized (templatized) using EL
-expressions.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="my-hive2-action">
- <hive2 xmlns="uri:oozie:hive2-action:1.0">
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- </configuration>
- <jdbc-url>jdbc:hive2://localhost:10000/default</jdbc-url>
- <password>foo</password>
- <script>myscript.q</script>
- <param>InputDir=/home/rkanter/input-data</param>
- <param>OutputDir=${jobOutput}</param>
- </hive2>
- <ok to="my-other-action"/>
- <error to="error-cleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-
-### Security
-
-As mentioned above, `password` is only used in cases where Hive Server 2 is backed by something requiring a password (e.g. LDAP).
-Non-secured Hive Server 2 and Kerberized Hive Server 2 don't require a password so in these cases it can be omitted.
-
-## Appendix, Hive 2 XML-Schema
-
-### AE.A Appendix A, Hive 2 XML-Schema
-
-#### Hive 2 Action Schema Version 1.0
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive2="uri:oozie:hive2-action:1.0" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive2-action:1.0">
-.
- <xs:include schemaLocation="oozie-common-1.0.xsd"/>
-.
- <xs:element name="hive2" type="hive2:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:choice>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="launcher" type="hive2:LAUNCHER" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:choice minOccurs="1" maxOccurs="1">
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Hive 2 Action Schema Version 0.2
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive2="uri:oozie:hive2-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive2-action:0.2">
-.
- <xs:element name="hive2" type="hive2:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:choice minOccurs="1" maxOccurs="1">
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="hive2:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="hive2:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Hive 2 Action Schema Version 0.1
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive2="uri:oozie:hive2-action:0.1" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive2-action:0.1">
-.
- <xs:element name="hive2" type="hive2:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="hive2:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive2:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="jdbc-url" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="password" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="hive2:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="hive2:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
[05/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_HiveActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_HiveActionExtension.twiki b/docs/src/site/twiki/DG_HiveActionExtension.twiki
deleted file mode 100644
index 99a73c6..0000000
--- a/docs/src/site/twiki/DG_HiveActionExtension.twiki
+++ /dev/null
@@ -1,476 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Hive Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-<a name="HiveAction"></a>
-## Hive Action
-
-The `hive` action runs a Hive job.
-
-The workflow job will wait until the Hive job completes before
-continuing to the next action.
-
-To run the Hive job, you have to configure the `hive` action with the `resource-manager`, `name-node` and Hive `script`
-(or Hive `query`) elements as well as the necessary parameters and configuration.
-
-A `hive` action can be configured to create or delete HDFS directories
-before starting the Hive job.
-
-Hive configuration can be specified with a file, using the `job-xml`
-element, and inline, using the `configuration` elements.
-
-Oozie EL expressions can be used in the inline configuration. Property
-values specified in the `configuration` element override values specified
-in the `job-xml` file.
-
-Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties
-must not be present in the inline configuration.
-
-As with Hadoop `map-reduce` jobs, it is possible to add files and
-archives in order to make them available to the Hive job. Refer to the
-[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
-section for more information about this feature.
-
-Oozie Hive action supports Hive scripts with parameter variables, their
-syntax is `${VARIABLES}`.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <hive xmlns="uri:oozie:hive-action:1.0">
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[HIVE SETTINGS FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <script>[HIVE-SCRIPT]</script>
- <param>[PARAM-VALUE]</param>
- ...
- <param>[PARAM-VALUE]</param>
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- </hive>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-
-The `job-xml` element, if present, specifies a file containing configuration
-for the Hive job. As of schema 0.3, multiple `job-xml` elements are allowed in order to
-specify multiple `job.xml` files.
-
-The `configuration` element, if present, contains configuration
-properties that are passed to the Hive job.
-
-The `script` element must contain the path of the Hive script to
-execute. The Hive script can be templatized with variables of the form
-`${VARIABLE}`. The values of these variables can then be specified
-using the `params` element.
-
-The `query` element available from uri:oozie:hive-action:0.6, can be used instead of the
-`script` element. It allows for embedding queries within the `worklfow.xml` directly.
-Similar to the `script` element, it also allows for the templatization of variables in the
-form `${VARIABLE}`.
-
-The `params` element, if present, contains parameters to be passed to
-the Hive script.
-
-All the above elements can be parameterized (templatized) using EL
-expressions.
-
-**Example:**
-
-
-```
-<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="myfirsthivejob">
- <hive xmlns="uri:oozie:hive-action:1.0">
- <resource-manager>foo:8032</resource-manager>
- <name-node>bar:8020</name-node>
- <prepare>
- <delete path="${jobOutput}"/>
- </prepare>
- <configuration>
- <property>
- <name>mapred.compress.map.output</name>
- <value>true</value>
- </property>
- </configuration>
- <script>myscript.q</script>
- <param>InputDir=/home/tucu/input-data</param>
- <param>OutputDir=${jobOutput}</param>
- </hive>
- <ok to="myotherjob"/>
- <error to="errorcleanup"/>
- </action>
- ...
-</workflow-app>
-```
-
-### Hive Default and Site Configuration Files
-
-Hive (as of Hive 0.8) ignores a `hive-default.xml` file. As a result, Oozie (as of Oozie 3.4) ignores the `oozie.hive.defaults`
-property that was previously required by earlier versions of Oozie for the Hive action.
-
-### Hive Action Logging
-
-Hive action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Hive.
-
-From Oozie web-console, from the Hive action pop up using the 'Console URL' link, it is possible
-to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
-
-The logging level of the Hive action can set in the Hive action configuration using the
-property `oozie.hive.log.level`. The default value is `INFO`.
-
-## Appendix, Hive XML-Schema
-
-### AE.A Appendix A, Hive XML-Schema
-
-#### Hive Action Schema Version 1.0
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive="uri:oozie:hive-action:1.0"
- elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive-action:1.0">
-.
- <xs:include schemaLocation="oozie-common-1.0.xsd"/>
-.
- <xs:element name="hive" type="hive:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:choice>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:choice minOccurs="1" maxOccurs="1">
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Hive Action Schema Version 0.6
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive="uri:oozie:hive-action:0.6" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive-action:0.6">
-.
- <xs:element name="hive" type="hive:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:choice minOccurs="1" maxOccurs="1">
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="query" type="xs:string" minOccurs="1" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-#### Hive Action Schema Version 0.5
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive="uri:oozie:hive-action:0.5" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive-action:0.5">
-.
- <xs:element name="hive" type="hive:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Hive Action Schema Version 0.4
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive="uri:oozie:hive-action:0.4" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive-action:0.4">
-.
- <xs:element name="hive" type="hive:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Hive Action Schema Version 0.3
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive="uri:oozie:hive-action:0.3" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive-action:0.3">
-.
- <xs:element name="hive" type="hive:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Hive Action Schema Version 0.2
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:hive="uri:oozie:hive-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:hive-action:0.2">
-.
- <xs:element name="hive" type="hive:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="param" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="hive:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="hive:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-.
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_JMSNotifications.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_JMSNotifications.twiki b/docs/src/site/twiki/DG_JMSNotifications.twiki
deleted file mode 100644
index e8f8a76..0000000
--- a/docs/src/site/twiki/DG_JMSNotifications.twiki
+++ /dev/null
@@ -1,236 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# JMS Notifications
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Overview
-Since Oozie 4.0, Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss
-events. This provides an alternative to polling Oozie for Job or SLA related information and getting events as they
-happen without any delay. Clients can be written to consume these notifications and integrate with different monitoring
-and alerting systems.
-
-## Oozie Server Configuration
-Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) section of [Oozie Install](AG_Install.html)
-documentation for the Oozie server side configuration required to support publishing notifications to a JMS Provider.
-The JNDI properties for the JMS provider, the topics to publish to and the notification types to publish (Job and/or SLA)
-need to be configured.
-
-## Consuming Notifications
-
-### Notification types
-Job and SLA notifications are published to the configured JMS Provider on the configured topics.
-
-Job status change notifications include job start, success, failure, suspended, etc. Currently only workflow job and
-coordinator action status change notifications are published.
-
-SLA notifications include START_MET, END_MET, DURATION_MET, START_MISS, END_MISS, DURATION_MISS events and are published
-for a workflow job, workflow action or coordinator action for which SLA information is configured in the job xml. Refer
-to [SLA Configuration](DG_SLAMonitoring.html#Configuring_SLA_in_Applications) for information on configuring SLA for a workflow or
-coordinator.
-
-### JMS Topic
-Consumers interested in notification on events will require to know the JNDI properties to connect to the JMS provider.
-They will also need to know the JMS topic on which notifications for a particular job are published.
-
-Oozie Client provides the following APIs :
-
-```
-public JMSConnectionInfo getJMSConnectionInfo()
-public String getJMSTopicName(String jobId)
-```
-
-The JMSConnectionInfo exposes 3 methods:
-
-
-```
-Properties getJNDIProperties();
-String getTopicPattern(AppType appType);
-String getTopicPrefix();
-```
-
-The topic is obtained by concatenating topic prefix and the substituted value for topic pattern. The topic pattern
-can be a constant value like workflow or coordinator which the administrator has configured or a variable (either ${username}
-or ${jobId}). If ${username}, it has to be substituted with the name of the user who has submitted the job; and if ${jobId} it has
-to be substituted with the job Id. Administrators can chose to publish messages to topics containing user names to avoid having one
-topic containing all messages and all users having to apply selectors to filter the message they are interested in.
-
-The getJMSTopicName API can be used if the job id is already known and will give the exact topic name to which the
-notifications for that job are published.
-
-
-### JMS Message Format
-JMS messages published are `javax.jms.TextMessage`. The body contains JSON and the header contains multiple properties
-that can be used as selectors. The header properties are not repeated in the body of the message to keep the messages
-small.
-
-<b>Message Header:</b> <br/>
-The different header properties are:
-
- * msgType - Value can be JOB or SLA.
- * user - The user who submitted the job.
- * appName - Application name of the job.
- * appType - Type of the job. One of WORKFLOW_JOB, WORKFLOW_ACTION, COORDINATOR_ACTION
- * slaStatus - Applicable only to SLA messages. Value is one of NOT_STARTED, IN_PROCESS, MET, MISS
- * eventStatus - It takes one of the following values- START_MET, START_MISS, DURATION_MET, DURATION_MISS, END_MET, END_MISS for SLA notifications. It can take any of the following values- WAITING, STARTED, SUCCESS, SUSPEND, FAILURE for job notifications. Note that event status is different from the Job status. It is included in the header to provide better filtering. Below is the mapping of event status to the actual status of workflow job or coordinator action.
-
-WAITING = When a Coordinator Action is waiting for input dependency
-
-STARTED = When the Workflow Job or Coordinator Action is in RUNNING state
-
-SUCCESS = When the Workflow Job or Coordinator Action is in SUCCEEDED state
-
-SUSPEND = When the Workflow Job or Coordinator Action is in SUSPENDED state
-
-FAILURE = When the Workflow Job or Coordinator Action is in terminal state other than SUCCEEDED.
-
-<b>Message Body for Job Notifications:</b> <br/>
- Sample JSON response for different job and sla events as below.
-
-
-```
-Workflow Job in RUNNING state:
-{"status":"RUNNING","id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000}
-```
-
-
-```
-Workflow Job in FAILED state:
-{"status":"FAILED","errorCode":"EL_ERROR","errorMessage":"variable [dummyvalue] cannot be resolved",
-"id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000,"endTime":1366672183543}
-```
-
-
-```
-Workflow Job in SUCCEEDED state:
-{"status":"SUCCEEDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
-"parentId":"0000025-130618221729631-oozie-oozi-C@1","endTime":1366676224154}
-```
-
-
-```
-Workflow Job in SUSPENDED state:
-{"status":"SUSPENDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
-"parentId":"0000025-130618221729631-oozie-oozi-C@1"}
-```
-
-
-```
-Coordinator Action in WAITING state:
-{"status":"WAITING","nominalTime":1310342400000,"missingDependency":"hdfs://gsbl90107.blue.com:8020/user/john/dir1/file1",
-"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
-```
-
-
-```
-Coordinator Action in RUNNING state:
-{"status":"RUNNING","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
-"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
-```
-
-
-```
-Coordinator Action in SUCCEEDED state:
-{"status":"SUCCEEDED","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
-"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677082799}
-```
-
-
-```
-Coordinator Action in FAILED state:
-{"status":"FAILED","errorCode":"E0101","errorMessage":"dummyError","nominalTime":1310342400000,
-"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,
-"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677140818}
-```
-
-<b>Message Body for SLA Notifications:</b> <br/>
-
-
-```
-Workflow Job in sla END_MISS state:
-{"id":"0000000-000000000000001-oozie-wrkf-C@1","parentId":"0000000-000000000000001-oozie-wrkf-C",
-"expectedStartTime":1356998400000,"notificationMessage":"notification of start miss","actualStartTime":1357002000000,
-"expectedDuration":-1, "actualDuration":3600,"expectedEndTime":1356998400000,"actualEndTime":1357002000000}
-```
-
-### JMS Client
-
-Oozie provides a helper class JMSMessagingUtils for consumers to deserialize the JMS messages back to Java objects.
-The below method getEventMessage() expects a sub type of EventMessage.
-There are different implementations of EventMessage - WorkflowJobMessage, CoordinatorActionMessage and SLAMessage.
-
-
-```
-<T extends EventMessage> T JMSMessagingUtils.getEventMessage(Message jmsMessage)
-```
-#### Example
-Below is a sample code to consume notifications.
-
-First, create the Oozie client and retrieve the JNDI properties to make a connection to the JMS server.
-
-```
- OozieClient oc = new OozieClient("http://localhost:11000/oozie");
- JMSConnectionInfo jmsInfo = oc.getJMSConnectionInfo();
- Properties jndiProperties = jmsInfo.getJNDIProperties();
- Context jndiContext = new InitialContext(jndiProperties);
- String connectionFactoryName = (String) jndiContext.getEnvironment().get("connectionFactoryNames");
- ConnectionFactory connectionFactory = (ConnectionFactory) jndiContext.lookup(connectionFactoryName);
- Connection connection = connectionFactory.createConnection();
- Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
- String topicPrefix = jmsInfo.getTopicPrefix();
- String topicPattern = jmsInfo.getTopicPattern(AppType.WORKFLOW_JOB);
- // Following code checks if the topic pattern is
- //'username', then the topic name is set to the actual user submitting the job
- String topicName = null;
- if (topicPattern.equals("${username}")) {
- topicName = "john";
- // Following code checks if the topic pattern is
- //'jobId', then the topic name is set to the job id
- } else if (topicPattern.equals("${jobId}")) {
- topicName = "0000004-140328125200198-oozie-oozi-W";
- }
- Destination topic = session.createTopic(topicPrefix+topicName);
- MessageConsumer consumer = session.createConsumer(topic);
- consumer.setMessageListener(this);
- connection.start();
-```
-
-To start receiving messages, the JMS [MessageListener](http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html)
-interface needs to be implemented. Also, its onMessage() method needs to be implemented.
-This method will be called whenever a message is available on the JMS bus.
-
-
-```
- public void onMessage(Message message) {
- if (message.getStringProperty(JMSHeaderConstants.MESSAGE_TYPE).equals(MessageType.SLA.name())){
- SLAMessage slaMessage = JMSMessagingUtils.getEventMessage(message);
- // Further processing
- }
- else if (message.getStringProperty(JMSHeaderConstants.APP_TYPE).equals(AppType.WORKFLOW_JOB.name())){
- WorkflowJobMessage wfJobMessage = JMSMessagingUtils.getEventMessage(message);
- // Further processing
- }
- }
-```
-
-#### Applying Selectors
-
-Below is a sample ActiveMQ text message header properties section.
-
-```
-ActiveMQTextMessage
-{properties = {appName = map-reduce-wf, msgType=JOB, appType=WORKFLOW_JOB, user=john, msgFormat=json, eventStatus=STARTED} ...}
-```
-
-On the header properties, consumers can apply JMS selectors to filter messages from JMS provider.
-They are listed at [JMSHeaderConstants](../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html)
-
-Sample use of selector to filter events related to Job which have failed and has a particular app-name
-
-
-```
-String selector=JMSHeaderConstants.EVENT_STATUS + "='FAILURE' AND " + JMSHeaderConstants.APP_NAME + "='app-name'";
-MessageConsumer consumer = session.createConsumer(topic, selector);
-```
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_Overview.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_Overview.twiki b/docs/src/site/twiki/DG_Overview.twiki
deleted file mode 100644
index 6a2b9d2..0000000
--- a/docs/src/site/twiki/DG_Overview.twiki
+++ /dev/null
@@ -1,86 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Oozie Workflow Overview
-
-Oozie is a server based _Workflow Engine_ specialized in running workflow jobs with actions that run Hadoop Map/Reduce
-and Pig jobs.
-
-Oozie is a Java Web-Application that runs in a Java servlet-container.
-
-For the purposes of Oozie, a workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in
-a control dependency DAG (Directed Acyclic Graph). "control dependency" from one action to another means that the second
-action can't run until the first action has completed.
-
-Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to
-[JBOSS JBPM](http://www.jboss.org/jbossjbpm/) jPDL).
-
-Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems
-callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.
-
-Oozie uses a custom SecurityManager inside it's launcher to catch exit() calls from the user code. Make sure to delegate checkExit()
-calls to Oozie's SecurityManager if the user code uses its own SecurityManager. The Launcher also grants java.security.AllPermission
-by default to the user code.
-
-Oozie workflows contain control flow nodes and action nodes.
-
-Control flow nodes define the beginning and the end of a workflow ( `start`, `end` and `fail` nodes) and provide a
-mechanism to control the workflow execution path ( `decision`, `fork` and `join` nodes).
-
-Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie
-provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and
-Oozie sub-workflow. Oozie can be extended to support additional type of actions.
-
-Oozie workflows can be parameterized (using variables like `${inputDir}` within the workflow definition). When
-submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different
-output directories) several identical workflow jobs can concurrently.
-
-## WordCount Workflow Example
-
-**Workflow Diagram:**
-
-<img src="./DG_Overview.png"/>
-
-**hPDL Workflow Definition:**
-
-
-```
-<workflow-app name='wordcount-wf' xmlns="uri:oozie:workflow:0.1">
- <start to='wordcount'/>
- <action name='wordcount'>
- <map-reduce>
- <job-tracker>${jobTracker}</job-tracker>
- <name-node>${nameNode}</name-node>
- <configuration>
- <property>
- <name>mapred.mapper.class</name>
- <value>org.myorg.WordCount.Map</value>
- </property>
- <property>
- <name>mapred.reducer.class</name>
- <value>org.myorg.WordCount.Reduce</value>
- </property>
- <property>
- <name>mapred.input.dir</name>
- <value>${inputDir}</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>${outputDir}</value>
- </property>
- </configuration>
- </map-reduce>
- <ok to='end'/>
- <error to='end'/>
- </action>
- <kill name='kill'>
- <message>Something went wrong: ${wf:errorCode('wordcount')}</message>
- </kill/>
- <end name='end'/>
-</workflow-app>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_SLAMonitoring.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SLAMonitoring.twiki b/docs/src/site/twiki/DG_SLAMonitoring.twiki
deleted file mode 100644
index 0831b93..0000000
--- a/docs/src/site/twiki/DG_SLAMonitoring.twiki
+++ /dev/null
@@ -1,459 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-# Oozie SLA Monitoring
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-## Overview
-
-Critical jobs can have certain SLA requirements associated with them. This SLA can be in terms of time
-i.e. a maximum allowed time limit associated with when the job should start, by when should it end,
-and its duration of run. Oozie workflows and coordinators allow defining such SLA limits in the application definition xml.
-
-With the addition of SLA Monitoring, Oozie can now actively monitor the state of these SLA-sensitive jobs
-and send out notifications for SLA mets and misses.
-
-In versions earlier than 4.x, this was a passive feature where users needed to query the Oozie client SLA API
-to fetch the records regarding job status changes, and use their own custom calculation engine to compute
-whether SLA was met or missed, based on initial definition of time limits.
-
-Oozie now also has a SLA tab in the Oozie UI, where users can query for SLA information and have a summarized view
-of how their jobs fared against their SLAs.
-
-
-## Oozie Server Configuration
-
-Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) for configuring Oozie server to track
-SLA for jobs and send notifications.
-
-## SLA Tracking
-
-Oozie allows tracking SLA for meeting the following criteria:
-
- * Start time
- * End time
- * Job Duration
-
-### Event Status
-Corresponding to each of these 3 criteria, your jobs are processed for whether Met or Miss i.e.
-
- * START_MET, START_MISS
- * END_MET, END_MISS
- * DURATION_MET, DURATION_MISS
-
-### SLA Status
-Expected end-time is the most important criterion for majority of users while deciding overall SLA Met or Miss.
-Hence the _"SLA_Status"_ for a job will transition through these four stages
-
- * Not_Started <-- Job not yet begun
- * In_Process <-- Job started and is running, and SLAs are being tracked
- * Met <-- caused by an END_MET
- * Miss <-- caused by an END_MISS
-
-In addition to overshooting expected end-time, and END_MISS (and so an eventual SLA MISS) also occurs when the
-job does not end successfully e.g. goes to error state - Failed/Killed/Error/Timedout.
-
-## Configuring SLA in Applications
-
-To make your jobs trackable for SLA, you simply need to add the `<sla:info>` tag to your workflow application definition.
-If you were already using the existing SLA schema
-in your workflows (Schema xmlns:sla="uri:oozie:sla:0.1"), you don't need to
-do anything extra to receive SLA notifications via JMS messages. This new SLA monitoring framework is backward-compatible -
-no need to change application XML for now and you can continue to fetch old records via the [command line API](DG_CommandLineTool.html#SLAOperations).
-However, usage of old schema
-and API is deprecated and we strongly recommend using new schema.
-
- * New SLA schema
-is 'uri:oozie:sla:0.2'
- * In order to use new SLA schema,
- you will need to upgrade your workflow/coordinator schema
-to 0.5 i.e. 'uri:oozie:workflow:0.5'
-
-### SLA Definition in Workflow
-Example:
-
-```
-<workflow-app name="test-wf-job-sla"
- xmlns="uri:oozie:workflow:0.5"
- xmlns:sla="uri:oozie:sla:0.2">
- <start to="grouper"/>
- <action name="grouper">
- <map-reduce>
- <job-tracker>jt</job-tracker>
- <name-node>nn</name-node>
- <configuration>
- <property>
- <name>mapred.input.dir</name>
- <value>input</value>
- </property>
- <property>
- <name>mapred.output.dir</name>
- <value>output</value>
- </property>
- </configuration>
- </map-reduce>
- <ok to="end"/>
- <error to="end"/>
- </action>
- <end name="end"/>
- <sla:info>
- <sla:nominal-time>${nominal_time}</sla:nominal-time>
- <sla:should-start>${10 * MINUTES}</sla:should-start>
- <sla:should-end>${30 * MINUTES}</sla:should-end>
- <sla:max-duration>${30 * MINUTES}</sla:max-duration>
- <sla:alert-events>start_miss,end_miss,duration_miss</sla:alert-events>
- <sla:alert-contact>joe@example.com</sla:alert-contact>
- </sla:info>
-</workflow-app>
-```
-
-For the list of tags usable under `<sla:info>`, refer to [Schemas Appendix](WorkflowFunctionalSpec.html#SLASchema).
-This new schema
-is much more compact and meaningful, getting rid of redundant and unused tags.
-
- * `nominal-time`: As the name suggests, this is the time relative to which your jobs' SLAs will be calculated. Generally since Oozie workflows are aligned with synchronous data dependencies, this nominal time can be parameterized to be passed the value of your coordinator nominal time. Nominal time is also required in case of independent workflows and you can specify the time in which you expect the workflow to be run if you don't have a synchronous dataset associated with it.
- * `should-start`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *start running* to meet SLA. This is optional.
- * `should-end`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *finish* to meet SLA.
- * `max-duration`: This is the maximum amount of time (along with time-unit - MINUTES, HOURS, DAYS) your job is expected to run. This is optional.
- * `alert-events`: Specify the types of events for which **Email** alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not necessary. However, note that this setting is only for alerts via *email* alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured.
- * `alert-contact`: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts.
-
-NOTE: All tags can be parameterized as a EL function or a fixed value.
-
-Same schema
-can be applied to and embedded under Workflow-Action as well as Coordinator-Action XML.
-
-### SLA Definition in Workflow Action
-
-
-```
-<workflow-app name="test-wf-action-sla" xmlns="uri:oozie:workflow:0.5" xmlns:sla="uri:oozie:sla:0.2">
- <start to="grouper"/>
- <action name="grouper">
- ...
- <ok to="end"/>
- <error to="end"/>
- <sla:info>
- <sla:nominal-time>${nominal_time}</sla:nominal-time>
- <sla:should-start>${10 * MINUTES}</sla:should-start>
- ...
- </sla:info>
- </action>
- <end name="end"/>
-</workflow-app>
-```
-
-### SLA Definition in Coordinator Action
-
-```
-<coordinator-app name="test-coord-sla" frequency="${coord:days(1)}" freq_timeunit="DAY"
- end_of_duration="NONE" start="2013-06-20T08:01Z" end="2013-12-01T08:01Z"
- timezone="America/Los_Angeles" xmlns="uri:oozie:coordinator:0.4" xmlns:sla="uri:oozie:sla:0.2">
- <action>
- <workflow>
- <app-path>${wfAppPath}</app-path>
- </workflow>
- <sla:info>
- <sla:nominal-time>${nominal_time}</sla:nominal-time>
- ...
- </sla:info>
- </action>
-</coordinator-app>
-```
-
-## Accessing SLA Information
-
-SLA information is accessible via the following ways:
-
- * Through the SLA tab of the Oozie Web UI.
- * JMS messages sent to a configured JMS provider for instantaneous tracking.
- * RESTful API to query for SLA summary.
- * As an `Instrumentation.Counter` entry that is accessible via RESTful API and reflects to the number of all SLA tracked external
- entities. Name of this counter is `sla-calculator.sla-map`.
-
-For JMS Notifications, you have to have a message broker in place, on which Oozie publishes messages and you can
-hook on a subscriber to receive those messages. For more info on setting up and consuming JMS messages, refer
-[JMS Notifications](DG_JMSNotifications.html) documentation.
-
-In the REST API, the following filters can be applied while fetching SLA information:
-
- * app_name - Application name
- * id - id of the workflow job, workflow action or coordinator action
- * parent_id - Parent id of the workflow job, workflow action or coordinator action
- * nominal_start and nominal_end - Start and End range for nominal time of the workflow or coordinator.
- * bundle - Bundle Job ID or Bundle App Name. Fetches SLA information for actions of all coordinators in that bundle.
- * event_status - event status such as START_MET/START_MISS/DURATION_MET/DURATION_MISS/END_MET/END_MISS
- * sla_status - sla status such as NOT_STARTED/IN_PROCESS/MET/MISS
-
-multiple event_status and sla_status can be specified with comma separation. When multiple statuses are specified, they are considered as OR.
-For example, event_status=START_MET;END_MISS list the coordinator actions where event status is either START_MET OR END_MISS.
-
-When timezone query parameter is specified, the expected and actual start/end time returned is formatted. If not specified,
-the number of milliseconds that have elapsed since January 1, 1970 00:00:00.000 GMT is returned.
-
-The examples below demonstrate the use of REST API and explains the JSON response.
-
-### Scenario 1: Workflow Job Start_Miss
-**Request:**
-
-```
-GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=nominal_start=2013-06-18T00:01Z;nominal_end=2013-06-23T00:01Z;app_name=my-sla-app
-```
-
-**JSON Response**
-
-```
-{
-
- id : "000056-1238791320234-oozie-joe-W"
- parentId : "000001-1238791320234-oozie-joe-C@8"
- appType : "WORKFLOW_JOB"
- msgType : "SLA"
- appName : "my-sla-app"
- slaStatus : "IN_PROCESS"
- jobStatus : "RUNNING"
- user: "joe"
- nominalTime: "2013-16-22T05:00Z"
- expectedStartTime: "2013-16-22T05:10Z" <-- (should start by this time)
- actualStartTime: "2013-16-22T05:30Z" <-- (20 min late relative to expected start)
- expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time)
- actualEndTime: null
- expectedDuration: 900000 <-- (expected duration in milliseconds)
- actualDuration: 120000 <-- (actual duration in milliseconds)
- notificationMessage: "My Job has encountered an SLA event!"
- upstreamApps: "dependent-app-1, dependent-app-2"
-
-}
-```
-
-### Scenario 2: Workflow Action End_Miss
-**Request:**
-
-```
-GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=parent_id=000056-1238791320234-oozie-joe-W
-```
-
-**JSON Response**
-
-```
-{
-
- id : "000056-1238791320234-oozie-joe-W@map-reduce-action"
- parentId : "000056-1238791320234-oozie-joe-W"
- appType : "WORKFLOW_ACTION"
- msgType : "SLA"
- appName : "map-reduce-action"
- slaStatus : "MISS"
- jobStatus : "SUCCEEDED"
- user: "joe"
- nominalTime: "2013-16-22T05:00Z"
- expectedStartTime: "2013-16-22T05:10Z"
- actualStartTime: "2013-16-22T05:05Z"
- expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time)
- actualEndTime: "2013-16-22T06:00Z" <-- (20 min late relative to expected end)
- expectedDuration: 3600000 <-- (expected duration in milliseconds)
- actualDuration: 3300000 <-- (actual duration in milliseconds)
- notificationMessage: "My Job has encountered an SLA event!"
- upstreamApps: "dependent-app-1, dependent-app-2"
-
-}
-```
-
-### Scenario 3: Coordinator Action Duration_Miss
-**Request:**
-
-```
-GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=id=000001-1238791320234-oozie-joe-C
-```
-
-**JSON Response**
-
-```
-{
-
- id : "000001-1238791320234-oozie-joe-C@2"
- parentId : "000001-1238791320234-oozie-joe-C"
- appType : "COORDINATOR_ACTION"
- msgType : "SLA"
- appName : "my-coord-app"
- slaStatus : "MET"
- jobStatus : "SUCCEEDED"
- user: "joe"
- nominalTime: "2013-16-22T05:00Z"
- expectedStartTime: "2013-16-22T05:10Z"
- actualStartTime: "2013-16-22T05:05Z"
- expectedEndTime: "2013-16-22T05:40Z"
- actualEndTime: "2013-16-22T05:30Z"
- expectedDuration: 900000 <-- (expected duration in milliseconds)
- actualDuration: 1500000 <- (actual duration in milliseconds)
- notificationMessage: "My Job has encountered an SLA event!"
- upstreamApps: "dependent-app-1, dependent-app-2"
-
-}
-```
-
-Scenario #3 is particularly interesting because it is an overall "MET" because it met its expected End-time,
-but it is "Duration_Miss" because the actual run (between actual start and actual end) exceeded expected duration.
-
-### Scenario 4: All Coordinator actions in a Bundle
-**Request:**
-
-```
-GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=bundle=1234567-150130225116604-oozie-B;event_status=END_MISS
-```
-
-**JSON Response**
-
-```
-{
- id : "000001-1238791320234-oozie-joe-C@1"
- parentId : "000001-1238791320234-oozie-joe-C"
- appType : "COORDINATOR_ACTION"
- msgType : "SLA"
- appName : "my-coord-app"
- slaStatus : "MET"
- eventStatus : "START_MET,DURATION_MISS,END_MISS"
- user: "joe"
- nominalTime: "2014-01-10T12:00Z"
- expectedStartTime: "2014-01-10T12:00Z"
- actualStartTime: "2014-01-10T11:59Z"
- startDelay: -1
- expectedEndTime: "2014-01-10T13:00Z"
- actualEndTime: "2014-01-10T13:05Z"
- endDelay: 5
- expectedDuration: 3600000 <-- (expected duration in milliseconds)
- actualDuration: 3960000 <-- (actual duration in milliseconds)
- durationDelay: 6 <-- (duration delay in minutes)
-}
-{
- id : "000001-1238791320234-oozie-joe-C@2"
- parentId : "000001-1238791320234-oozie-joe-C"
- appType : "COORDINATOR_ACTION"
- msgType : "SLA"
- appName : "my-coord-app"
- slaStatus : "MET"
- eventStatus : "START_MISS,DURATION_MET,END_MISS"
- user: "joe"
- nominalTime: "2014-01-11T12:00Z"
- expectedStartTime: "2014-01-11T12:00Z"
- actualStartTime: "2014-01-11T12:05Z"
- startDelay: 5
- expectedEndTime: "2014-01-11T13:00Z"
- actualEndTime: "2014-01-11T13:01Z"
- endDelay: 1
- expectedDuration: 3600000 <-- (expected duration in milliseconds)
- actualDuration: 3360000 <-- (actual duration in milliseconds)
- durationDelay: -4 <-- (duration delay in minutes)
-}
-```
-
-Scenario #4 (All Coordinator actions in a Bundle) is to get SLA information of all coordinator actions under bundle job in one call.
-startDelay/durationDelay/endDelay values returned indicate how much delay compared to expected time (positive values in case of MISS, and negative values in case of MET).
-
-### Sample Email Alert
-
-```
-Subject: OOZIE - SLA END_MISS (AppName=wf-sla-job, JobID=0000004-130610225200680-oozie-oozi-W)
-
-
-Status:
- SLA Status - END_MISS
- Job Status - RUNNING
- Notification Message - Missed SLA for Data Pipeline job
-Job Details:
- App Name - wf-sla-job
- App Type - WORKFLOW_JOB
- User - strat_ci
- Job ID - 0000004-130610225200680-oozie-oozi-W
- Job URL - http://host.domain.com:4080/oozie//?job=0000004-130610225200680-oozie-oozi-W
- Parent Job ID - N/A
- Parent Job URL - N/A
- Upstream Apps - wf-sla-up-app
-SLA Details:
- Nominal Time - Mon Jun 10 23:33:00 UTC 2013
- Expected Start Time - Mon Jun 10 23:35:00 UTC 2013
- Actual Start Time - Mon Jun 10 23:34:04 UTC 2013
- Expected End Time - Mon Jun 10 23:38:00 UTC 2013
- Expected Duration (in mins) - 5
- Actual Duration (in mins) - -1
-```
-
-### Changing job SLA definition and alerting
-Following are ways to enable/disable SLA alerts for coordinator actions.
-
-#### 1. Specify in Bundle XML during submission.
-Following properties can be specified in bundle xml as properties for coordinator.
-
-`oozie.sla.disable.alerts.older.than` this property can be specified in hours, the SLA notification for
-coord actions will be disabled whose nominal is time older then this value. Default is 48 hours.
-
-```
-<property>
- <name>oozie.sla.disable.alerts.older.than</name>
- <value>12</value>
-</property>
-```
-
-`oozie.sla.disable.alerts` List of coord actions to be disabled. Value can be specified as list of coord actions or date range.
-
-```
-<property>
- <name>oozie.sla.disable.alerts</name>
- <value>1,3-4,7-10</value>
-</property>
-```
-Will disable alert for coord actions 1,3,5,7,8,9,10
-
-`oozie.sla.enable.alerts` List of coord actions to be enabled. Value can be specified as list of coord actions or date range.
-
-```
-<property>
- <name>oozie.sla.enable.alerts</name>
- <value>2009-01-01T01:00Z::2009-05-31T23:59Z</value>
-</property>
-```
-This will enable SLA alert for coord actions whose nominal time is in between (inclusive) 2009-01-01T01:00Z and 2009-05-31T23:59Z.
-
-ALL keyword can be specified to specify all actions. Below property will disable SLA notifications for all coord actions.
-
-```
-<property>
- <name>oozie.sla.disable.alerts</name>
- <value>ALL</value>
-</property>
-```
-
-#### 2. Specify during Coordinator job submission or update
-Above properties can be specified in job.properties in
-[Coord job update command](DG_CommandLineTool.html#Updating_coordinator_definition_and_properties),
-in [Coord job submit command](DG_CommandLineTool.html#Submitting_a_Workflow_Coordinator_or_Bundle_Job)
-or in [Coord job run command](DG_CommandLineTool.html#Running_a_Workflow_Coordinator_or_Bundle_Job)
-
-#### 3. Change using command line
-Refer [Changing job SLA definition and alerting](DG_CommandLineTool.html#Changing_job_SLA_definition_and_alerting) for commandline usage.
-
-#### 4. Change using REST API
-Refer the REST API [Changing job SLA definition and alerting](WebServicesAPI.html#Changing_job_SLA_definition_and_alerting).
-
-## In-memory SLA entries and database content
-
-There are special circumstances when the in-memory `SLACalcStatus` entries can exist without the workflow or coordinator job or
-action instances in database. For example:
-
- * SLA tracked database content may already have been deleted, and `SLA_SUMMARY` entry is not present anymore in database
- * SLA tracked database content and `SLA_SUMMARY` entry aren't yet present in database
-
-By the time `SLAService` scheduled job will be running, SLA map contents are checked. When the `SLA_SUMMARY` entry for the in-memory
-SLA entry is missing, a counter is increased. When this counter reaches the server-wide preconfigured value
-`oozie.sla.service.SLAService.maximum.retry.count` (by default `3`), in-memory SLA entry will get purged.
-
-## Known issues
-There are two known issues when you define SLA for a workflow action.
- * If there are decision nodes and SLA is defined for a workflow action not in the execution path because of the decision node, you will still get an SLA_MISS notification.
- * If you have dangling action nodes in your workflow definition and SLA is defined for it, you will still get an SLA_MISS notification.
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/twiki/DG_ShellActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_ShellActionExtension.twiki b/docs/src/site/twiki/DG_ShellActionExtension.twiki
deleted file mode 100644
index eff4b08..0000000
--- a/docs/src/site/twiki/DG_ShellActionExtension.twiki
+++ /dev/null
@@ -1,494 +0,0 @@
-
-
-[::Go back to Oozie Documentation Index::](index.html)
-
------
-
-# Oozie Shell Action Extension
-
-<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-
-<a name="ShellAction"></a>
-## Shell Action
-
-The `shell` action runs a Shell command.
-
-The workflow job will wait until the Shell command completes before
-continuing to the next action.
-
-To run the Shell job, you have to configure the `shell` action with the
-`job-tracker`, `name-node` and Shell `exec` elements as
-well as the necessary arguments and configuration.
-
-A `shell` action can be configured to create or delete HDFS directories
-before starting the Shell job.
-
-Shell _launcher_ configuration can be specified with a file, using the `job-xml`
-element, and inline, using the `configuration` elements.
-
-Oozie EL expressions can be used in the inline configuration. Property
-values specified in the `configuration` element override values specified
-in the `job-xml` file.
-
-Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties
-must not be present in the inline configuration.
-
-As with Hadoop `map-reduce` jobs, it is possible to add files and
-archives in order to make them available to the Shell job. Refer to the
-[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
-section for more information about this feature.
-
-The output (STDOUT) of the Shell job can be made available to the workflow job after the Shell job ends. This information
-could be used from within decision nodes. If the output of the Shell job is made available to the workflow job the shell
-command must follow the following requirements:
-
- * The format of the output must be a valid Java Properties file.
- * The size of the output must not exceed 2KB.
-
-**Syntax:**
-
-
-```
-<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
- ...
- <action name="[NODE-NAME]">
- <shell xmlns="uri:oozie:shell-action:1.0">
- <resource-manager>[RESOURCE-MANAGER]</resource-manager>
- <name-node>[NAME-NODE]</name-node>
- <prepare>
- <delete path="[PATH]"/>
- ...
- <mkdir path="[PATH]"/>
- ...
- </prepare>
- <job-xml>[SHELL SETTINGS FILE]</job-xml>
- <configuration>
- <property>
- <name>[PROPERTY-NAME]</name>
- <value>[PROPERTY-VALUE]</value>
- </property>
- ...
- </configuration>
- <exec>[SHELL-COMMAND]</exec>
- <argument>[ARG-VALUE]</argument>
- ...
- <argument>[ARG-VALUE]</argument>
- <env-var>[VAR1=VALUE1]</env-var>
- ...
- <env-var>[VARN=VALUEN]</env-var>
- <file>[FILE-PATH]</file>
- ...
- <archive>[FILE-PATH]</archive>
- ...
- <capture-output/>
- </shell>
- <ok to="[NODE-NAME]"/>
- <error to="[NODE-NAME]"/>
- </action>
- ...
-</workflow-app>
-```
-
-The `prepare` element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-
-The `job-xml` element, if present, specifies a file containing configuration
-for the Shell job. As of schema 0.2, multiple `job-xml` elements are allowed in order to
-specify multiple `job.xml` files.
-
-The `configuration` element, if present, contains configuration
-properties that are passed to the Shell job.
-
-The `exec` element must contain the path of the Shell command to
-execute. The arguments of Shell command can then be specified
-using one or more `argument` element.
-
-The `argument` element, if present, contains argument to be passed to
-the Shell command.
-
-The `env-var` element, if present, contains the environment to be passed
-to the Shell command. `env-var` should contain only one pair of environment variable
-and value. If the pair contains the variable such as $PATH, it should follow the
-Unix convention such as PATH=$PATH:mypath. Don't use ${PATH} which will be
-substituted by Oozie's EL evaluator.
-
-A `shell` action creates a Hadoop configuration. The Hadoop configuration is made available as a local file to the
-Shell application in its running directory. The exact file path is exposed to the spawned shell using the environment
-variable called `OOZIE_ACTION_CONF_XML`.The Shell application can access the environment variable to read the action
-configuration XML file path.
-
-If the `capture-output` element is present, it indicates Oozie to capture output of the STDOUT of the shell command
-execution. The Shell command output must be in Java Properties file format and it must not exceed 2KB. From within the
-workflow definition, the output of an Shell action node is accessible via the `String action:output(String node,
-String key)` function (Refer to section '4.2.6 Action EL Functions').
-
-All the above elements can be parameterized (templatized) using EL
-expressions.
-
-**Example:**
-
-How to run any shell script or perl script or CPP executable
-
-
-```
-<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'>
- <start to='shell1' />
- <action name='shell1'>
- <shell xmlns="uri:oozie:shell-action:1.0">
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode}</name-node>
- <configuration>
- <property>
- <name>mapred.job.queue.name</name>
- <value>${queueName}</value>
- </property>
- </configuration>
- <exec>${EXEC}</exec>
- <argument>A</argument>
- <argument>B</argument>
- <file>${EXEC}#${EXEC}</file> <!--Copy the executable to compute node's current working directory -->
- </shell>
- <ok to="end" />
- <error to="fail" />
- </action>
- <kill name="fail">
- <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
- </kill>
- <end name='end' />
-</workflow-app>
-```
-
-The corresponding job properties file used to submit Oozie job could be as follows:
-
-
-```
-oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script
-
-#Execute is expected to be in the Workflow directory.
-#Shell Script to run
-EXEC=script.sh
-#CPP executable. Executable should be binary compatible to the compute node OS.
-#EXEC=hello
-#Perl script
-#EXEC=script.pl
-
-resourceManager=localhost:8032
-nameNode=hdfs://localhost:8020
-queueName=default
-
-```
-
-How to run any java program bundles in a jar.
-
-
-```
-<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'>
- <start to='shell1' />
- <action name='shell1'>
- <shell xmlns="uri:oozie:shell-action:1.0">
- <resource-manager>${resourceManager}</resource-manager>
- <name-node>${nameNode}</name-node>
- <configuration>
- <property>
- <name>mapred.job.queue.name</name>
- <value>${queueName}</value>
- </property>
- </configuration>
- <exec>java</exec>
- <argument>-classpath</argument>
- <argument>./${EXEC}:$CLASSPATH</argument>
- <argument>Hello</argument>
- <file>${EXEC}#${EXEC}</file> <!--Copy the jar to compute node current working directory -->
- </shell>
- <ok to="end" />
- <error to="fail" />
- </action>
- <kill name="fail">
- <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
- </kill>
- <end name='end' />
-</workflow-app>
-```
-
-The corresponding job properties file used to submit Oozie job could be as follows:
-
-
-```
-oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script
-
-#Hello.jar file is expected to be in the Workflow directory.
-EXEC=Hello.jar
-
-resourceManager=localhost:8032
-nameNode=hdfs://localhost:8020
-queueName=default
-```
-
-### Shell Action Configuration
-
- * `oozie.action.shell.setup.hadoop.conf.dir` - Generates a config directory with various core/hdfs/yarn/mapred-site.xml files and points `HADOOP_CONF_DIR` and `YARN_CONF_DIR` env-vars to it, before the Script is invoked. XML is sourced from the action configuration. Useful when the Shell script passed uses various `hadoop` commands. Default is false.
- * `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` - When `oozie.action.shell.setup.hadoop.conf.dir` is enabled, toggle if a log4j.properties file should also be written under the configuration files directory. Default is true.
- * `oozie.action.shell.setup.hadoop.conf.dir.log4j.content` - When `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` is enabled, the content to write into the log4j.properties file under the configuration files directory. Default is a simple console based stderr logger, as presented below:
-
-```
-log4j.rootLogger=${hadoop.root.logger}
-hadoop.root.logger=INFO,console
-log4j.appender.console=org.apache.log4j.ConsoleAppender
-log4j.appender.console.target=System.err
-log4j.appender.console.layout=org.apache.log4j.PatternLayout
-log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
-```
-
-### Shell Action Logging
-
-Shell action's stdout and stderr output are redirected to the Oozie Launcher map-reduce job task STDOUT that runs the shell command.
-
-From Oozie web-console, from the Shell action pop up using the 'Console URL' link, it is possible
-to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
-
-### Shell Action Limitations
-Although Shell action can execute any shell command, there are some limitations.
-
- * No interactive command is supported.
- * Command can't be executed as different user using sudo.
- * User has to explicitly upload the required 3rd party packages (such as jar, so lib, executable etc). Oozie provides a way using \<file\> and \<archive\> tag through Hadoop's Distributed Cache to upload.
- * Since Oozie will execute the shell command into a Hadoop compute node, the default installation of utility in the compute node might not be fixed. However, the most common unix utilities are usually installed on all compute nodes. It is important to note that Oozie could only support the commands that are installed into the compute nodes or that are uploaded through Distributed Cache.
-
-## Appendix, Shell XML-Schema
-
-### AE.A Appendix A, Shell XML-Schema
-
-#### Shell Action Schema Version 1.0
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:shell="uri:oozie:shell-action:1.0"
- elementFormDefault="qualified"
- targetNamespace="uri:oozie:shell-action:1.0">
-.
- <xs:include schemaLocation="oozie-common-1.0.xsd"/>
-.
- <xs:element name="shell" type="shell:ACTION"/>
-.
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:choice>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="resource-manager" type="xs:string" minOccurs="0" maxOccurs="1"/>
- </xs:choice>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="launcher" type="shell:LAUNCHER" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
-.
-</xs:schema>
-```
-
-#### Shell Action Schema Version 0.3
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:shell="uri:oozie:shell-action:0.3" elementFormDefault="qualified"
- targetNamespace="uri:oozie:shell-action:0.3">
-
- <xs:element name="shell" type="shell:ACTION"/>
-
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="FLAG"/>
-
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
-</xs:schema>
-```
-
-#### Shell Action Schema Version 0.2
-
-```
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:shell="uri:oozie:shell-action:0.2" elementFormDefault="qualified"
- targetNamespace="uri:oozie:shell-action:0.2">
-
- <xs:element name="shell" type="shell:ACTION"/>
-
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="FLAG"/>
-
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
-</xs:schema>
-```
-
-#### Shell Action Schema Version 0.1
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:shell="uri:oozie:shell-action:0.1" elementFormDefault="qualified"
- targetNamespace="uri:oozie:shell-action:0.1">
-
- <xs:element name="shell" type="shell:ACTION"/>
-
- <xs:complexType name="ACTION">
- <xs:sequence>
- <xs:element name="job-tracker" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="name-node" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="prepare" type="shell:PREPARE" minOccurs="0" maxOccurs="1"/>
- <xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="1"/>
- <xs:element name="configuration" type="shell:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
- <xs:element name="exec" type="xs:string" minOccurs="1" maxOccurs="1"/>
- <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="env-var" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="file" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="archive" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="capture-output" type="shell:FLAG" minOccurs="0" maxOccurs="1"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="FLAG"/>
-
- <xs:complexType name="CONFIGURATION">
- <xs:sequence>
- <xs:element name="property" minOccurs="1" maxOccurs="unbounded">
- <xs:complexType>
- <xs:sequence>
- <xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
- <xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
- </xs:sequence>
- </xs:complexType>
- </xs:element>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="PREPARE">
- <xs:sequence>
- <xs:element name="delete" type="shell:DELETE" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element name="mkdir" type="shell:MKDIR" minOccurs="0" maxOccurs="unbounded"/>
- </xs:sequence>
- </xs:complexType>
-
- <xs:complexType name="DELETE">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
- <xs:complexType name="MKDIR">
- <xs:attribute name="path" type="xs:string" use="required"/>
- </xs:complexType>
-
-</xs:schema>
-```
-
-[::Go back to Oozie Documentation Index::](index.html)
-
-
[19/21] oozie git commit: OOZIE-2734 amend [docs] Switch from TWiki
to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/6a6f2199/docs/src/site/markdown/CoordinatorFunctionalSpec.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/CoordinatorFunctionalSpec.md b/docs/src/site/markdown/CoordinatorFunctionalSpec.md
new file mode 100644
index 0000000..d31d1aa
--- /dev/null
+++ b/docs/src/site/markdown/CoordinatorFunctionalSpec.md
@@ -0,0 +1,4890 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+-----
+
+# Oozie Coordinator Specification
+
+The goal of this document is to define a coordinator engine system specialized in submitting workflows based on time and data triggers.
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Changelog
+
+**03/JUL/2013**
+
+ * Appendix A, Added new coordinator schema 0.4, sla schema 0.2 and changed schemas ordering to newest first
+
+**07/JAN/2013**
+
+ * 6.8 Added section on new EL functions for datasets defined with HCatalog
+
+**26/JUL/2012**
+
+ * Appendix A, updated XML schema 0.4 to include `parameters` element
+ * 6.5 Updated to mention about `parameters` element as of schema 0.4
+
+**23/NOV/2011:**
+
+ * Update execution order typo
+
+**05/MAY/2011:**
+
+ * Update coordinator schema 0.2
+
+**09/MAR/2011:**
+
+ * Update coordinator status
+
+**02/DEC/2010:**
+
+ * Update coordinator done-flag
+
+**26/AUG/2010:**
+
+ * Update coordinator rerun
+
+**09/JUN/2010:**
+
+ * Clean up unsupported functions
+
+**02/JUN/2010:**
+
+ * Update all EL functions in CoordFunctionalSpec with "coord:" prefix
+
+**02/OCT/2009:**
+
+ * Added Appendix A, Oozie Coordinator XML-Schema
+ * Change #5.3., Datasets definition supports 'include' element
+
+**29/SEP/2009:**
+
+ * Change #4.4.1, added `${coord:endOfDays(int n)}` EL function
+ * Change #4.4.2, added `${coord:endOfMonths(int n)}` EL function
+
+**11/SEP/2009:**
+
+ * Change #6.6.4. `${coord:tzOffset()}` EL function now returns offset in minutes. Added more explanation on behavior
+ * Removed 'oozie' URL from action workflow invocation, per arch review feedback coord&wf run on the same instance
+
+**07/SEP/2009:**
+
+ * Full rewrite of sections #4 and #7
+ * Added sections #6.1.7, #6.6.2, #6.6.3 & #6.6.4
+ * Rewording through the spec definitions
+ * Updated all examples and syntax to latest changes
+
+**03/SEP/2009:**
+
+ * Change #2. Definitions. Some rewording in the definitions
+ * Change #6.6.4. Replaced `${coord:next(int n)}` with `${coord:version(int n)}` EL Function
+ * Added #6.6.5. Dataset Instance Resolution for Instances Before the Initial Instance
+
+## 1. Coordinator Overview
+
+Users typically run map-reduce, hadoop-streaming, hdfs and/or Pig jobs on the grid. Multiple of these jobs can be combined to form a workflow job. [Oozie, Hadoop Workflow System](https://issues.apache.org/jira/browse/HADOOP-5303) defines a workflow system that runs such jobs.
+
+Commonly, workflow jobs are run based on regular time intervals and/or data availability. And, in some cases, they can be triggered by an external event.
+
+Expressing the condition(s) that trigger a workflow job can be modeled as a predicate that has to be satisfied. The workflow job is started after the predicate is satisfied. A predicate can reference to data, time and/or external events. In the future, the model can be extended to support additional event types.
+
+It is also necessary to connect workflow jobs that run regularly, but at different time intervals. The outputs of multiple subsequent runs of a workflow become the input to the next workflow. For example, the outputs of last 4 runs of a workflow that runs every 15 minutes become the input of another workflow that runs every 60 minutes. Chaining together these workflows result it is referred as a data application pipeline.
+
+The Oozie **Coordinator** system allows the user to define and execute recurrent and interdependent workflow jobs (data application pipelines).
+
+Real world data application pipelines have to account for reprocessing, late processing, catchup, partial processing, monitoring, notification and SLAs.
+
+This document defines the functional specification for the Oozie Coordinator system.
+
+## 2. Definitions
+
+**Actual time:** The actual time indicates the time when something actually happens.
+
+**Nominal time:** The nominal time specifies the time when something should happen. In theory the nominal time and the actual time should match, however, in practice due to delays the actual time may occur later than the nominal time.
+
+**Dataset:** Collection of data referred to by a logical name. A dataset normally has several instances of data and each
+one of them can be referred individually. Each dataset instance is represented by a unique set of URIs.
+
+**Synchronous Dataset:** Synchronous datasets instances are generated at fixed time intervals and there is a dataset
+instance associated with each time interval. Synchronous dataset instances are identified by their nominal time.
+For example, in the case of a HDFS based dataset, the nominal time would be somewhere in the file path of the
+dataset instance: `hdfs://foo:8020/usr/logs/2009/04/15/23/30`. In the case of HCatalog table partitions, the nominal time
+would be part of some partition values: `hcat://bar:8020/mydb/mytable/year=2009;month=04;dt=15;region=us`.
+
+**Coordinator Action:** A coordinator action is a workflow job that is started when a set of conditions are met (input dataset instances are available).
+
+**Coordinator Application:** A coordinator application defines the conditions under which coordinator actions should be created (the frequency) and when the actions can be started. The coordinator application also defines a start and an end time. Normally, coordinator applications are parameterized. A Coordinator application is written in XML.
+
+**Coordinator Job:** A coordinator job is an executable instance of a coordination definition. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
+
+**Data pipeline:** A data pipeline is a connected set of coordinator applications that consume and produce interdependent datasets.
+
+**Coordinator Definition Language:** The language used to describe datasets and coordinator applications.
+
+**Coordinator Engine:** A system that executes coordinator jobs.
+
+## 3. Expression Language for Parameterization
+
+Coordinator application definitions can be parameterized with variables, built-in constants and built-in functions.
+
+At execution time all the parameters are resolved into concrete values.
+
+The parameterization of workflow definitions it done using JSP Expression Language syntax from the [JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to support variables as parameters but also functions and complex expressions.
+
+EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.
+
+Refer to section #6.5 'Parameterization of Coordinator Applications' for more details.
+
+## 4. Datetime, Frequency and Time-Period Representation
+
+Oozie processes coordinator jobs in a fixed timezone with no DST (typically `UTC`), this timezone is referred as 'Oozie
+processing timezone'.
+
+The Oozie processing timezone is used to resolve coordinator jobs start/end times, job pause times and the initial-instance
+of datasets. Also, all coordinator dataset instance URI templates are resolved to a datetime in the Oozie processing
+time-zone.
+
+All the datetimes used in coordinator applications and job parameters to coordinator applications must be specified
+in the Oozie processing timezone. If Oozie processing timezone is `UTC`, the qualifier is **Z**. If Oozie processing
+time zone is other than `UTC`, the qualifier must be the GMT offset, `(+/-)####`.
+
+For example, a datetime in `UTC` is `2012-08-12T00:00Z`, the same datetime in `GMT+5:30` is `2012-08-12T05:30+0530`.
+
+For simplicity, the rest of this specification uses `UTC` datetimes.
+
+<a name="datetime"></a>
+### 4.1. Datetime
+
+If the Oozie processing timezone is `UTC`, all datetime values are always in
+[UTC](http://en.wikipedia.org/wiki/Coordinated_Universal_Time) down to a minute precision, 'YYYY-MM-DDTHH:mmZ'.
+
+For example `2009-08-10T13:10Z` is August 10th 2009 at 13:10 UTC.
+
+If the Oozie processing timezone is a GMT offset `GMT(+/-)####`, all datetime values are always in
+[ISO 8601](http://en.wikipedia.org/wiki/ISO_8601) in the corresponding GMT offset down to a minute precision,
+'YYYY-MM-DDTHH:mmGMT(+/-)####'.
+
+For example `2009-08-10T13:10+0530` is August 10th 2009 at 13:10 GMT+0530, India timezone.
+
+#### 4.1.1 End of the day in Datetime Values
+
+It is valid to express the end of day as a '24:00' hour (i.e. `2009-08-10T24:00Z`).
+
+However, for all calculations and display, Oozie resolves such dates as the zero hour of the following day
+(i.e. `2009-08-11T00:00Z`).
+
+### 4.2. Timezone Representation
+
+There is no widely accepted standard to identify timezones.
+
+Oozie Coordinator will understand the following timezone identifiers:
+
+ * Generic NON-DST timezone identifier: `GMT[+/-]##:##` (i.e.: GMT+05:30)
+ * UTC timezone identifier: `UTC` (i.e.: 2009-06-06T00:00Z)
+ * ZoneInfo identifiers, with DST support, understood by Java JDK (about 600 IDs) (i.e.: America/Los_Angeles)
+
+Due to DST shift from PST to PDT, it is preferred that GMT, UTC or Region/City timezone notation is used in
+favor of direct three-letter ID (PST, PDT, BST, etc.). For example, America/Los_Angeles switches from PST to PDT
+at a DST shift. If used directly, PST will not handle DST shift when time is switched to PDT.
+
+Oozie Coordinator must provide a tool for developers to list all supported timezone identifiers.
+
+### 4.3. Timezones and Daylight-Saving
+
+While Oozie coordinator engine works in a fixed timezone with no DST (typically `UTC`), it provides DST support for coordinator applications.
+
+The baseline datetime for datasets and coordinator applications are expressed in UTC. The baseline datetime is the time of the first occurrence.
+
+Datasets and coordinator applications also contain a timezone indicator.
+
+The use of UTC as baseline enables a simple way of mix and matching datasets and coordinator applications that use a different timezone by just adding the timezone offset.
+
+The timezone indicator enables Oozie coordinator engine to properly compute frequencies that are daylight-saving sensitive. For example: a daily frequency can be 23, 24 or 25 hours for timezones that observe daylight-saving. Weekly and monthly frequencies are also affected by this as the number of hours in the day may change.
+
+Section #7 'Handling Timezones and Daylight Saving Time' explains how coordinator applications can be written to handle timezones and daylight-saving-time properly.
+
+### 4.4. Frequency and Time-Period Representation
+
+Frequency is used to capture the periodic intervals at which datasets that are produced, and coordinator applications are scheduled to run.
+
+This time periods representation is also used to specify non-recurrent time-periods, for example a timeout interval.
+
+For datasets and coordinator applications the frequency time-period is applied `N` times to the baseline datetime to compute recurrent times.
+
+Frequency is always expressed in minutes.
+
+Because the number of minutes in day may vary for timezones that observe daylight saving time, constants cannot be use to express frequencies greater than a day for datasets and coordinator applications for such timezones. For such uses cases, Oozie coordinator provides 2 EL functions, `${coord:days(int n)}` and `${coord:months(int n)}`.
+
+Frequencies can be expressed using EL constants and EL functions that evaluate to an positive integer number.
+
+Coordinator Frequencies can also be expressed using cron syntax.
+
+**<font color="#008000"> Examples: </font>**
+
+| **EL Constant** | **Value** | **Example** |
+| --- | --- | --- |
+| `${coord:minutes(int n)}` | _n_ | `${coord:minutes(45)}` --> `45` |
+| `${coord:hours(int n)}` | _n * 60_ | `${coord:hours(3)}` --> `180` |
+| `${coord:days(int n)}` | _variable_ | `${coord:days(2)}` --> minutes in 2 full days from the current date |
+| `${coord:months(int n)}` | _variable_ | `${coord:months(1)}` --> minutes in a 1 full month from the current date |
+| `${cron syntax}` | _variable_ | `${0,10 15 * * 2-6}` --> a job that runs every weekday at 3:00pm and 3:10pm UTC time|
+
+Note that, though `${coord:days(int n)}` and `${coord:months(int n)}` EL functions are used to calculate minutes precisely including
+variations due to daylight saving time for Frequency representation, when specified for coordinator timeout interval, one day is
+calculated as 24 hours and one month is calculated as 30 days for simplicity.
+
+#### 4.4.1. The coord:days(int n) and coord:endOfDays(int n) EL functions
+
+The `${coord:days(int n)}` and `${coord:endOfDays(int n)}` EL functions should be used to handle day based frequencies.
+
+Constant values should not be used to indicate a day based frequency (every 1 day, every 1 week, etc) because the number of hours in
+every day is not always the same for timezones that observe daylight-saving time.
+
+It is a good practice to use always these EL functions instead of using a constant expression (i.e. `24 * 60`) even if the timezone
+for which the application is being written for does not support daylight saving time. This makes application foolproof to country
+legislation changes and also makes applications portable across timezones.
+
+##### 4.4.1.1. The coord:days(int n) EL function
+
+The `${coord:days(int n)}` EL function returns the number of minutes for 'n' complete days starting with the day of the specified nominal time for which the computation is being done.
+
+The `${coord:days(int n)}` EL function includes **all** the minutes of the current day, regardless of the time of the day of the current nominal time.
+
+**<font color="#008000"> Examples: </font>**
+
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:days(1)}` | 1440 | `2009-01-01T08:00Z` | total minutes on 2009JAN01 UTC time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1440 | `2009-01-01T08:00Z` | total minutes in 2009JAN01 PST8PDT time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:days(2)}` | 2880 | `2009-01-01T08:00Z` | total minutes in 2009JAN01 and 2009JAN02 PST8PDT time |
+| |||||
+| `2009-03-08T08:00Z` | `UTC` | `${coord:days(1)}` | 1440 | `2009-03-08T08:00Z` | total minutes on 2009MAR08 UTC time |
+| `2009-03-08T08:00Z` | `Europe/London` | `${coord:days(1)}` | 1440 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 BST1BDT time |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1380 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
+| `2009-03-08T08:00Z` | `UTC` | `${coord:days(2)}` | 2880 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 and 2009MAR09 UTC time |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:days(2)}` | 2820 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 and 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
+| `2009-03-09T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1440 | `2009-03-09T07:00Z` | total minutes in 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST ON, frequency tick is earlier in UTC) |
+
+For all these examples, the first occurrence of the frequency will be at `08:00Z` (UTC time).
+
+##### 4.4.1.2. The coord:endOfDays(int n) EL function
+
+The `${coord:endOfDays(int n)}` EL function is identical to the `${coord:days(int n)}` except that it shifts the first occurrence to the end of the day for the specified timezone before computing the interval in minutes.
+
+**<font color="#008000"> Examples: </font>**
+
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T00:00Z` | first occurrence in 2009JAN02 00:00 UTC time, <br/> first occurrence shifted to the end of the UTC day |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-01-01T08:01Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-01-01T18:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
+| |||||
+| `2009-03-07T09:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1380 | `2009-03-08T08:00Z` | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-03-08T07:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-03-08T08:00Z` | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-03-09T07:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-03-10T07:00Z` | first occurrence in 2009MAR10 07:00 UTC time <br/> (2009MAR08 is DST switch in the US), <br/> first occurrence shifted to the end of the PST8PDT day |
+
+
+```
+<coordinator-app name="hello-coord" frequency="${coord:days(1)}"
+ start="2009-01-02T08:00Z" end="2009-01-04T08:00Z" timezone="America/Los_Angeles"
+ xmlns="uri:oozie:coordinator:0.5">
+ <controls>
+ <timeout>10</timeout>
+ <concurrency>${concurrency_level}</concurrency>
+ <execution>${execution_order}</execution>
+ <throttle>${materialization_throttle}</throttle>
+ </controls>
+
+ <datasets>
+ <dataset name="din" frequency="${coord:endOfDays(1)}"
+ initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
+ <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
+ </dataset>
+ <dataset name="dout" frequency="${coord:minutes(30)}"
+ initial-instance="2009-01-02T08:00Z" timezone="UTC">
+ <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
+ </dataset>
+ </datasets>
+
+ <input-events>
+ <data-in name="input" dataset="din">
+ <instance>${coord:current(0)}</instance>
+ </data-in>
+ </input-events>
+
+ <output-events>
+ <data-out name="output" dataset="dout">
+ <instance>${coord:current(1)}</instance>
+ </data-out>
+ </output-events>
+
+ <action>
+ <workflow>
+ <app-path>${wf_app_path}</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>${coord:dataIn('input')}</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>${coord:dataOut('output')}</value>
+ </property>
+ </configuration>
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+#### 4.4.2. The coord:months(int n) and coord:endOfMonths(int n) EL functions
+
+The `${coord:months(int n)}` and `${coord:endOfMonths(int n)}` EL functions should be used to handle month based frequencies.
+
+Constant values cannot be used to indicate a month based frequency because the number of days in a month changes month to month and on leap years; plus the number of hours in every day of the month are not always the same for timezones that observe daylight-saving time.
+
+##### 4.4.2.1. The coord:months(int n) EL function
+
+The `${coord:months(int n)}` EL function returns the number of minutes for 'n' complete months starting with the month of the current nominal time for which the computation is being done.
+
+The `${coord:months(int n)}` EL function includes **all** the minutes of the current month, regardless of the day of the month of the current nominal time.
+
+**<font color="#008000"> Examples: </font>**
+
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:months(1)}` | 44640 | `2009-01-01T08:00Z` |total minutes for 2009JAN UTC time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:months(1)}` | 44640 | `2009-01-01T08:00Z` | total minutes in 2009JAN PST8PDT time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:months(2)}` | 84960 | `2009-01-01T08:00Z` | total minutes in 2009JAN and 2009FEB PST8PDT time |
+| |||||
+| `2009-03-08T08:00Z` | `UTC` | `${coord:months(1)}` | 44640 | `2009-03-08T08:00Z` | total minutes on 2009MAR UTC time |
+| `2009-03-08T08:00Z` | `Europe/London` | `${coord:months(1)}` | 44580 | `2009-03-08T08:00Z` | total minutes in 2009MAR BST1BDT time <br/> (2009MAR29 is DST switch in Europe) |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:months(1)}` | 44580 | `2009-03-08T08:00Z` | total minutes in 2009MAR PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
+| `2009-03-08T08:00Z` | `UTC` | `${coord:months(2)}` | 87840 | `2009-03-08T08:00Z` | total minutes in 2009MAR and 2009APR UTC time |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:months(2)}` | 87780 | `2009-03-08T08:00Z` | total minutes in 2009MAR and 2009APR PST8PDT time <br/> (2009MAR08 is DST switch in US) |
+
+##### 4.4.2.2. The coord:endOfMonths(int n) EL function
+
+The `${coord:endOfMonths(int n)}` EL function is identical to the `${coord:months(int n)}` except that it shifts the first occurrence to the end of the month for the specified timezone before computing the interval in minutes.
+
+**<font color="#008000"> Examples: </font>**
+
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T00:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
+| `2009-01-31T08:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T08:00Z` | first occurrence in 2009FEB 08:00 UTC time |
+| `2009-02-02T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 44580 | `2009-03-01T08:00Z` | first occurrence in 2009MAR 08:00 UTC time |
+| `2009-02-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 44580 | `2009-03-01T08:00Z` | first occurrence in 2009MAR 08:00 UTC time |
+
+
+```
+<coordinator-app name="hello-coord" frequency="${coord:months(1)}"
+ start="2009-01-02T08:00Z" end="2009-04-02T08:00Z" timezone="America/Los_Angeles"
+ xmlns="uri:oozie:coordinator:0.5">
+ <controls>
+ <timeout>10</timeout>
+ <concurrency>${concurrency_level}</concurrency>
+ <execution>${execution_order}</execution>
+ <throttle>${materialization_throttle}</throttle>
+ </controls>
+
+ <datasets>
+ <dataset name="din" frequency="${coord:endOfMonths(1)}"
+ initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
+ <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
+ </dataset>
+ <dataset name="dout" frequency="${coord:minutes(30)}"
+ initial-instance="2009-01-02T08:00Z" timezone="UTC">
+ <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
+ </dataset>
+ </datasets>
+
+ <input-events>
+ <data-in name="input" dataset="din">
+ <instance>${coord:current(0)}</instance>
+ </data-in>
+ </input-events>
+
+ <output-events>
+ <data-out name="output" dataset="dout">
+ <instance>${coord:current(1)}</instance>
+ </data-out>
+ </output-events>
+
+ <action>
+ <workflow>
+ <app-path>${wf_app_path}</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>${coord:dataIn('input')}</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>${coord:dataOut('output')}</value>
+ </property>
+ </configuration>
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+#### 4.4.3. The coord:endOfWeeks(int n) EL function
+
+The `${coord:endOfWeeks(int n)}` EL function shifts the first occurrence to the start of the week for the specified
+timezone before computing the interval in minutes. The start of the week depends on the Java's implementation of
+[Calendar.getFirstDayOfWeek()](https://docs.oracle.com/javase/8/docs/api/java/util/Calendar.html#getFirstDayOfWeek--)
+ i.e. first day of the week is SUNDAY in the U.S., MONDAY in France.
+
+**<font color="#008000"> Examples: </font>**
+
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2017-01-04T00:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T00:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
+| `2017-01-04T08:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
+| `2017-01-06T08:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
+| `2017-01-04T08:00Z` | `America/Los_Angeles` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence in 2017JAN08 08:00 UTC time |
+| `2017-01-06T08:00Z` | `America/Los_Angeles` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence in 2017JAN08 08:00 UTC time |
+
+
+```
+<coordinator-app name="hello-coord" frequency="${coord:endOfWeeks(1)}"
+ start="2017-01-04T08:00Z" end="2017-12-31T08:00Z" timezone="America/Los_Angeles"
+ xmlns="uri:oozie:coordinator:0.5">
+ <controls>
+ <timeout>10</timeout>
+ <concurrency>${concurrency_level}</concurrency>
+ <execution>${execution_order}</execution>
+ <throttle>${materialization_throttle}</throttle>
+ </controls>
+
+ <datasets>
+ <dataset name="din" frequency="${coord:endOfWeeks(1)}"
+ initial-instance="2017-01-01T08:00Z" timezone="America/Los_Angeles">
+ <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}</uri-template>
+ </dataset>
+ <dataset name="dout" frequency="${coord:endOfWeeks(1)}"
+ initial-instance="2017-01-01T08:00Z" timezone="UTC">
+ <uri-template>${baseFsURI}/${YEAR}/${MONTH}/${DAY}</uri-template>
+ </dataset>
+ </datasets>
+
+ <input-events>
+ <data-in name="input" dataset="din">
+ <instance>${coord:current(0)}</instance>
+ </data-in>
+ </input-events>
+
+ <output-events>
+ <data-out name="output" dataset="dout">
+ <instance>${coord:current(1)}</instance>
+ </data-out>
+ </output-events>
+
+ <action>
+ <workflow>
+ <app-path>${wf_app_path}</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>${coord:dataIn('input')}</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>${coord:dataOut('output')}</value>
+ </property>
+ </configuration>
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+#### 4.4.4. Cron syntax in coordinator frequency
+
+Oozie has historically allowed only very basic forms of scheduling: You could choose
+to run jobs separated by a certain number of minutes, hours, days or weeks. That's
+all. This works fine for processes that need to run continuously all year like building
+a search index to power an online website.
+
+However, there are a lot of cases that don't fit this model. For example, maybe you
+want to export data to a reporting system used during the day by business analysts.
+It would be wasteful to run the jobs when no analyst is going to take advantage of
+the new information, such as overnight. You might want a policy that says "only run
+these jobs on weekdays between 6AM and 8PM". Previous versions of Oozie didn't support
+this kind of complex scheduling policy without requiring multiple identical coordinators.
+Cron-scheduling improves the user experience in this area, allowing for a lot more flexibility.
+
+Cron is a standard time-based job scheduling mechanism in unix-like operating system. It is used extensively by system
+administrators to setup jobs and maintain software environment. Cron syntax generally consists of five fields, minutes,
+hours, date of month, month, and day of week respectively although multiple variations do exist.
+
+
+```
+<coordinator-app name="cron-coord" frequency="0/10 1/2 ** ** *" start="${start}" end="${end}" timezone="UTC"
+ xmlns="uri:oozie:coordinator:0.2">
+ <action>
+ <workflow>
+ <app-path>${workflowAppUri}</app-path>
+ <configuration>
+ <property>
+ <name>jobTracker</name>
+ <value>${jobTracker}</value>
+ </property>
+ <property>
+ <name>nameNode</name>
+ <value>${nameNode}</value>
+ </property>
+ <property>
+ <name>queueName</name>
+ <value>${queueName}</value>
+ </property>
+ </configuration>
+ </workflow>
+ </action>
+</coordinator-app>
+```
+
+Cron expressions are comprised of 5 required fields. The fields respectively are described as follows:
+
+| **Field name** | **Allowed Values** | **Allowed Special Characters** |
+| --- | --- | --- |
+| `Minutes` | `0-59` | , - * / |
+| `Hours` | `0-23` | , - * / |
+| `Day-of-month` | `1-31` | , - * ? / L W |
+| `Month` | `1-12 or JAN-DEC` | , - * / |
+| `Day-of-Week` | `1-7 or SUN-SAT` | , - * ? / L #|
+
+The '**' character is used to specify all values. For example, "**" in the minute field means "every minute".
+
+The '?' character is allowed for the day-of-month and day-of-week fields. It is used to specify 'no specific value'.
+This is useful when you need to specify something in one of the two fields, but not the other.
+
+The '-' character is used to specify ranges For example "10-12" in the hour field means "the hours 10, 11 and 12".
+
+The ',' character is used to specify additional values. For example "MON,WED,FRI" in the day-of-week field means
+"the days Monday, Wednesday, and Friday".
+
+The '/' character is used to specify increments. For example "0/15" in the minutes field means "the minutes 0, 15, 30, and 45".
+And "5/15" in the minutes field means "the minutes 5, 20, 35, and 50". Specifying '*' before the '/' is equivalent to
+specifying 0 is the value to start with.
+Essentially, for each field in the expression, there is a set of numbers that can be turned on or off.
+For minutes, the numbers range from 0 to 59. For hours 0 to 23, for days of the month 0 to 31, and for months 1 to 12.
+The "/" character simply helps you turn on every "nth" value in the given set. Thus "7/6" in the month field only turns on
+month "7", it does NOT mean every 6th month, please note that subtlety.
+
+The 'L' character is allowed for the day-of-month and day-of-week fields. This character is short-hand for "last",
+but it has different meaning in each of the two fields.
+For example, the value "L" in the day-of-month field means "the last day of the month" - day 31 for January, day 28 for
+February on non-leap years.
+If used in the day-of-week field by itself, it simply means "7" or "SAT".
+But if used in the day-of-week field after another value, it means "the last xxx day of the month" - for example
+"6L" means "the last Friday of the month".
+You can also specify an offset from the last day of the month, such as "L-3" which would mean the third-to-last day of the
+calendar month.
+When using the 'L' option, it is important not to specify lists, or ranges of values, as you'll get confusing/unexpected results.
+
+The 'W' character is allowed for the day-of-month field. This character is used to specify the weekday (Monday-Friday)
+nearest the given day.
+As an example, if you were to specify "15W" as the value for the day-of-month field, the meaning is:
+"the nearest weekday to the 15th of the month". So if the 15th is a Saturday, the trigger will fire on Friday the 14th.
+If the 15th is a Sunday, the trigger will fire on Monday the 16th. If the 15th is a Tuesday, then it will fire on Tuesday the 15th.
+However if you specify "1W" as the value for day-of-month, and the 1st is a Saturday, the trigger will fire on Monday the 3rd,
+as it will not 'jump' over the boundary of a month's days.
+The 'W' character can only be specified when the day-of-month is a single day, not a range or list of days.
+
+The 'L' and 'W' characters can also be combined for the day-of-month expression to yield 'LW', which translates to
+"last weekday of the month".
+
+The '#' character is allowed for the day-of-week field. This character is used to specify "the nth" XXX day of the month.
+For example, the value of "6#3" in the day-of-week field means the third Friday of the month (day 6 = Friday and "#3" =
+the 3rd one in the month).
+Other examples: "2#1" = the first Monday of the month and "4#5" = the fifth Wednesday of the month.
+Note that if you specify "#5" and there is not 5 of the given day-of-week in the month, then no firing will occur that month.
+If the '#' character is used, there can only be one expression in the day-of-week field ("3#1,6#3" is not valid,
+since there are two expressions).
+
+The legal characters and the names of months and days of the week are not case sensitive.
+
+If a user specifies an invalid cron syntax to run something on Feb, 30th for example: "0 10 30 2 *", the coordinator job
+will not be created and an invalid coordinator frequency parse exception will be thrown.
+
+If a user has a coordinator job that materializes no action during run time, for example: frequency of "0 10 ** ** *" with
+start time of 2013-10-18T21:00Z and end time of 2013-10-18T22:00Z, the coordinator job submission will be rejected and
+an invalid coordinator attribute exception will be thrown.
+
+**<font color="#008000"> Examples: </font>**
+
+| **Cron Expression** | **Meaning** |
+| --- | --- |
+| 10 9 ** ** * | Runs everyday at 9:10am |
+| 10,30,45 9 ** ** * | Runs everyday at 9:10am, 9:30am, and 9:45am |
+| `0 * 30 JAN 2-6` | Runs at 0 minute of every hour on weekdays and 30th of January |
+| `0/20 9-17 ** ** 2-5` | Runs every Mon, Tue, Wed, and Thurs at minutes 0, 20, 40 from 9am to 5pm |
+| 1 2 L-3 ** ** | Runs every third-to-last day of month at 2:01am |
+| `1 2 6W 3 ?` | Runs on the nearest weekday to March, 6th every year at 2:01am |
+| `1 2 * 3 3#2` | Runs every second Tuesday of March at 2:01am every year |
+| `0 10,13 ** ** MON-FRI` | Runs every weekday at 10am and 1pm |
+
+
+NOTES:
+
+ Cron expression and syntax in Oozie are inspired by Quartz:http://quartz-scheduler.org/api/2.0.0/org/quartz/CronExpression.html.
+ However, there is a major difference between Quartz cron and Oozie cron in which Oozie cron doesn't have "Seconds" field
+ since everything in Oozie functions at the minute granularity at most. Everything related to Oozie cron syntax should be based
+ on the documentation in the Oozie documentation.
+
+ Cron expression uses oozie server processing timezone. Since default oozie processing timezone is UTC, if you want to
+ run a job on every weekday at 10am in Tokyo, Japan(UTC + 9), your cron expression should be "0 1 * * 2-6" instead of
+ the "0 10 * * 2-6" which you might expect.
+
+ Overflowing ranges is supported but strongly discouraged - that is, having a larger number on the left hand side than the right.
+ You might do 22-2 to catch 10 o'clock at night until 2 o'clock in the morning, or you might have NOV-FEB.
+ It is very important to note that overuse of overflowing ranges creates ranges that don't make sense and
+ no effort has been made to determine which interpretation CronExpression chooses.
+ An example would be "0 14-6 ? * FRI-MON".
+
+## 5. Dataset
+
+A dataset is a collection of data referred to by a logical name.
+
+A dataset instance is a particular occurrence of a dataset and it is represented by a unique set of URIs. A dataset instance can be individually referred. Dataset instances for datasets containing ranges are identified by a set of unique URIs, otherwise a dataset instance is identified by a single unique URI.
+
+Datasets are typically defined in some central place for a business domain and can be accessed by the coordinator. Because of this, they can be defined once and used many times.
+
+A dataset is a synchronous (produced at regular time intervals, it has an expected frequency) input.
+
+A dataset instance is considered to be immutable while it is being consumed by coordinator jobs.
+
+### 5.1. Synchronous Datasets
+
+Instances of synchronous datasets are produced at regular time intervals, at an expected frequency. They are also referred to as "clocked datasets".
+
+Synchronous dataset instances are identified by their nominal creation time. The nominal creation time is normally specified in the dataset instance URI.
+
+A synchronous dataset definition contains the following information:
+
+ * **<font color="#0000ff"> name: </font>** The dataset name. It must be a valid Java identifier.
+ * **<font color="#0000ff"> frequency: </font>*** It represents the rate, in minutes at which data is _periodically_ created. The granularity is in minutes and can be expressed using EL expressions, for example: ${5 ** HOUR}.
+ * **<font color="#0000ff"> initial-instance: </font>** The UTC datetime of the initial instance of the dataset. The initial-instance also provides the baseline datetime to compute instances of the dataset using multiples of the frequency.
+ * **<font color="#0000ff"> timezone:</font>** The timezone of the dataset.
+ * **<font color="#0000ff"> uri-template:</font>** The URI template that identifies the dataset and can be resolved into concrete URIs to identify a particular dataset instance. The URI template is constructed using:
+ * **<font color="#0000ff"> constants </font>** See the allowable EL Time Constants below. Ex: ${YEAR}/${MONTH}.
+ * **<font color="#0000ff"> variables </font>** Variables must be resolved at the time a coordinator job is submitted to the coordinator engine. They are normally provided a job parameters (configuration properties). Ex: ${market}/${language}
+ * **<font color="#0000ff"> done-flag:</font>** This flag denotes when a dataset instance is ready to be consumed.
+ * If the done-flag is omitted the coordinator will wait for the presence of a _SUCCESS file in the directory (Note: MapReduce jobs create this on successful completion automatically).
+ * If the done-flag is present but empty, then the existence of the directory itself indicates that the dataset is ready.
+ * If the done-flag is present but non-empty, Oozie will check for the presence of the named file within the directory, and will be considered ready (done) when the file exists.
+
+The following EL constants can be used within synchronous dataset URI templates:
+
+| **EL Constant** | **Resulting Format** | **Comments** |
+| --- | --- | --- |
+| `YEAR` | _YYYY_ | 4 digits representing the year |
+| `MONTH` | _MM_ | 2 digits representing the month of the year, January = 1 |
+| `DAY` | _DD_ | 2 digits representing the day of the month |
+| `HOUR` | _HH_ | 2 digits representing the hour of the day, in 24 hour format, 0 - 23 |
+| `MINUTE` | _mm_ | 2 digits representing the minute of the hour, 0 - 59 |
+
+**<font color="#800080">Syntax: </font>**
+
+
+```
+ <dataset name="[NAME]" frequency="[FREQUENCY]"
+ initial-instance="[DATETIME]" timezone="[TIMEZONE]">
+ <uri-template>[URI TEMPLATE]</uri-template>
+ <done-flag>[FILE NAME]</done-flag>
+ </dataset>
+```
+
+IMPORTANT: The values of the EL constants in the dataset URIs (in HDFS) are expected in UTC. Oozie Coordinator takes care of the timezone conversion when performing calculations.
+
+**<font color="#008000"> Examples: </font>**
+
+1. **A dataset produced once every day at 00:15 PST8PDT and done-flag is set to empty:**
+
+
+ ```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/app/logs/${market}/${YEAR}${MONTH}/${DAY}/data
+ </uri-template>
+ <done-flag></done-flag>
+ </dataset>
+ ```
+
+
+ The dataset would resolve to the following URIs and Coordinator looks for the existence of the directory itself:
+
+
+ ```
+ [market] will be replaced with user given property.
+
+ hdfs://foo:8020/usr/app/[market]/2009/02/15/data
+ hdfs://foo:8020/usr/app/[market]/2009/02/16/data
+ hdfs://foo:8020/usr/app/[market]/2009/02/17/data
+ ...
+ ```
+
+
+2. **A dataset available on the 10th of each month and done-flag is default '_SUCCESS':**
+
+
+ ```
+ <dataset name="stats" frequency="${coord:months(1)}"
+ initial-instance="2009-01-10T10:00Z" timezone="America/Los_Angeles">
+ <uri-template>hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data</uri-template>
+ </dataset>
+ ```
+
+ The dataset would resolve to the following URIs:
+
+
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data
+ hdfs://foo:8020/usr/app/stats/2009/02/data
+ hdfs://foo:8020/usr/app/stats/2009/03/data
+ ...
+ ```
+
+ The dataset instances are not ready until '_SUCCESS' exists in each path:
+
+
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data/_SUCCESS
+ hdfs://foo:8020/usr/app/stats/2009/02/data/_SUCCESS
+ hdfs://foo:8020/usr/app/stats/2009/03/data/_SUCCESS
+ ...
+ ```
+
+
+3. **A dataset available at the end of every quarter and done-flag is 'trigger.dat':**
+
+
+ ```
+ <dataset name="stats" frequency="${coord:months(3)}"
+ initial-instance="2009-01-31T20:00Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data
+ </uri-template>
+ <done-flag>trigger.dat</done-flag>
+ </dataset>
+ ```
+
+ The dataset would resolve to the following URIs:
+
+
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data
+ hdfs://foo:8020/usr/app/stats/2009/04/data
+ hdfs://foo:8020/usr/app/stats/2009/07/data
+ ...
+ ```
+
+ The dataset instances are not ready until 'trigger.dat' exists in each path:
+
+
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data/trigger.dat
+ hdfs://foo:8020/usr/app/stats/2009/04/data/trigger.dat
+ hdfs://foo:8020/usr/app/stats/2009/07/data/trigger.dat
+ ...
+ ```
+
+
+4. **Normally the URI template of a dataset has a precision similar to the frequency:**
+
+
+ ```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/data
+ </uri-template>
+ </dataset>
+ ```
+
+ The dataset would resolve to the following URIs:
+
+
+ ```
+ hdfs://foo:8020/usr/app/logs/2009/01/01/data
+ hdfs://foo:8020/usr/app/logs/2009/01/02/data
+ hdfs://foo:8020/usr/app/logs/2009/01/03/data
+ ...
+ ```
+
+5. **However, if the URI template has a finer precision than the dataset frequency:**
+
+
+ ```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}/data
+ </uri-template>
+ </dataset>
+ ```
+
+ The dataset resolves to the following URIs with fixed values for the finer precision template variables:
+
+
+ ```
+ hdfs://foo:8020/usr/app/logs/2009/01/01/10/30/data
+ hdfs://foo:8020/usr/app/logs/2009/01/02/10/30/data
+ hdfs://foo:8020/usr/app/logs/2009/01/03/10/30/data
+ ...
+ ```
+
+### 5.2. Dataset URI-Template types
+
+Each dataset URI could be a HDFS path URI denoting a HDFS directory: `hdfs://foo:8020/usr/logs/20090415` or a
+HCatalog partition URI identifying a set of table partitions: `hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US`.
+
+HCatalog enables table and storage management for Pig, Hive and MapReduce. The format to specify a HCatalog table partition URI is
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...`
+
+For example,
+
+```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hcat://myhcatmetastore:9080/database1/table1/myfirstpartitionkey=myfirstvalue;mysecondpartitionkey=mysecondvalue
+ </uri-template>
+ <done-flag></done-flag>
+ </dataset>
+```
+
+### 5.3. Asynchronous Datasets
+ * TBD
+
+### 5.4. Dataset Definitions
+
+Dataset definitions are grouped in XML files.
+**IMPORTANT:** Please note that if an XML namespace version is specified for the coordinator-app element in the coordinator.xml file, no namespace needs to be defined separately for the datasets element (even if the dataset is defined in a separate file). Specifying it at multiple places might result in xml errors while submitting the coordinator job.
+
+**<font color="#800080">Syntax: </font>**
+
+
+```
+ <!-- Synchronous datasets -->
+<datasets>
+ <include>[SHARED_DATASETS]</include>
+ ...
+ <dataset name="[NAME]" frequency="[FREQUENCY]"
+ initial-instance="[DATETIME]" timezone="[TIMEZONE]">
+ <uri-template>[URI TEMPLATE]</uri-template>
+ </dataset>
+ ...
+</datasets>
+```
+
+**<font color="#008000"> Example: </font>**
+
+
+```
+<datasets>
+.
+ <include>hdfs://foo:8020/app/dataset-definitions/globallogs.xml</include>
+.
+ <dataset name="logs" frequency="${coord:hours(12)}"
+ initial-instance="2009-02-15T08:15Z" timezone="Americas/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/app/logs/${market}/${YEAR}${MONTH}/${DAY}/${HOUR}/${MINUTE}/data
+ </uri-template>
+ </dataset>
+.
+ <dataset name="stats" frequency="${coord:months(1)}"
+ initial-instance="2009-01-10T10:00Z" timezone="Americas/Los_Angeles">
+ <uri-template>hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data</uri-template>
+ </dataset>
+.
+</datasets>
+```
+
+## 6. Coordinator Application
+
+### 6.1. Concepts
+
+#### 6.1.1. Coordinator Application
+
+A coordinator application is a program that triggers actions (commonly workflow jobs) when a set of conditions are met. Conditions can be a time frequency, the availability of new dataset instances or other external events.
+
+Types of coordinator applications:
+
+ * **Synchronous:** Its coordinator actions are created at specified time intervals.
+
+Coordinator applications are normally parameterized.
+
+#### 6.1.2. Coordinator Job
+
+To create a coordinator job, a job configuration that resolves all coordinator application parameters must be provided to the coordinator engine.
+
+A coordinator job is a running instance of a coordinator application running from a start time to an end time. The start
+time must be earlier than the end time.
+
+At any time, a coordinator job is in one of the following status: **PREP, RUNNING, RUNNINGWITHERROR, PREPSUSPENDED, SUSPENDED, SUSPENDEDWITHERROR, PREPPAUSED, PAUSED, PAUSEDWITHERROR, SUCCEEDED, DONEWITHERROR, KILLED, FAILED**.
+
+Valid coordinator job status transitions are:
+
+ * **PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED**
+ * **RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED**
+ * **RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | KILLED | FAILED**
+ * **PREPSUSPENDED --> PREP | KILLED**
+ * **SUSPENDED --> RUNNING | KILLED**
+ * **SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED**
+ * **PREPPAUSED --> PREP | KILLED**
+ * **PAUSED --> SUSPENDED | RUNNING | KILLED**
+ * **PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED**
+ * **FAILED | KILLED --> IGNORED**
+ * **IGNORED --> RUNNING**
+
+When a coordinator job is submitted, oozie parses the coordinator job XML. Oozie then creates a record for the coordinator with status **PREP** and returns a unique ID. The coordinator is also started immediately if pause time is not set.
+
+When a user requests to suspend a coordinator job that is in **PREP** state, oozie puts the job in status **PREPSUSPENDED**. Similarly, when pause time reaches for a coordinator job with **PREP** status, oozie puts the job in status **PREPPAUSED**.
+
+Conversely, when a user requests to resume a **PREPSUSPENDED** coordinator job, oozie puts the job in status **PREP**. And when pause time is reset for a coordinator job and job status is **PREPPAUSED**, oozie puts the job in status **PREP**.
+
+When a coordinator job starts, oozie puts the job in status **RUNNING** and start materializing workflow jobs based on job frequency. If any workflow job goes to **FAILED/KILLED/TIMEDOUT** state, the coordinator job is put in **RUNNINGWITHERROR**
+
+When a user requests to kill a coordinator job, oozie puts the job in status **KILLED** and it sends kill to all submitted workflow jobs.
+
+When a user requests to suspend a coordinator job that is in **RUNNING** status, oozie puts the job in status **SUSPENDED** and it suspends all submitted workflow jobs. Similarly, when a user requests to suspend a coordinator job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **SUSPENDEDWITHERROR** and it suspends all submitted workflow jobs.
+
+When pause time reaches for a coordinator job that is in **RUNNING** status, oozie puts the job in status **PAUSED**. Similarly, when pause time reaches for a coordinator job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **PAUSEDWITHERROR**.
+
+Conversely, when a user requests to resume a **SUSPENDED** coordinator job, oozie puts the job in status **RUNNING**. Also, when a user requests to resume a **SUSPENDEDWITHERROR** coordinator job, oozie puts the job in status **RUNNINGWITHERROR**. And when pause time is reset for a coordinator job and job status is **PAUSED**, oozie puts the job in status **RUNNING**. Also, when the pause time is reset for a coordinator job and job status is **PAUSEDWITHERROR**, oozie puts the job in status **RUNNINGWITHERROR**
+
+A coordinator job creates workflow jobs (commonly coordinator actions) only for the duration of the coordinator job and only if the coordinator job is in **RUNNING** status. If the coordinator job has been suspended, when resumed it will create all the coordinator actions that should have been created during the time it was suspended, actions will not be lost, they will delayed.
+
+When the coordinator job materialization finishes and all workflow jobs finish, oozie updates the coordinator status accordingly.
+For example, if all workflows are **SUCCEEDED**, oozie puts the coordinator job into **SUCCEEDED** status.
+If all workflows are **FAILED**, oozie puts the coordinator job into **FAILED** status. If all workflows are **KILLED**, the coordinator
+job status changes to KILLED. However, if any workflow job finishes with not **SUCCEEDED** and combination of **KILLED**, **FAILED** or
+**TIMEOUT**, oozie puts the coordinator job into **DONEWITHERROR**. If all coordinator actions are **TIMEDOUT**, oozie puts the
+coordinator job into **DONEWITHERROR**.
+
+A coordinator job in **FAILED** or **KILLED** status can be changed to **IGNORED** status. A coordinator job in **IGNORED** status can be changed to
+ **RUNNING** status.
+
+#### 6.1.3. Coordinator Action
+
+A coordinator job creates and executes coordinator actions.
+
+A coordinator action is normally a workflow job that consumes and produces dataset instances.
+
+Once an coordinator action is created (this is also referred as the action being materialized), the coordinator action will be in waiting until all required inputs for execution are satisfied or until the waiting times out.
+
+##### 6.1.3.1. Coordinator Action Creation (Materialization)
+
+A coordinator job has one driver event that determines the creation (materialization) of its coordinator actions (typically a workflow job).
+
+ * For synchronous coordinator jobs the driver event is the frequency of the coordinator job.
+
+##### 6.1.3.2. Coordinator Action Status
+
+Once a coordinator action has been created (materialized) the coordinator action qualifies for execution. At this point, the action status is **WAITING**.
+
+A coordinator action in **WAITING** status must wait until all its input events are available before is ready for execution. When a coordinator action is ready for execution its status is **READY**.
+
+A coordinator action in **WAITING** status may timeout before it becomes ready for execution. Then the action status is **TIMEDOUT**.
+
+A coordinator action may remain in **READY** status for a while, without starting execution, due to the concurrency execution policies of the coordinator job.
+
+A coordinator action in **READY** or **WAITING** status changes to **SKIPPED** status if the execution strategy is LAST_ONLY and the
+current time is past the next action's nominal time. See section 6.3 for more details.
+
+A coordinator action in **READY** or **WAITING** status changes to **SKIPPED** status if the execution strategy is NONE and the
+current time is past the action's nominal time + 1 minute. See section 6.3 for more details.
+
+A coordinator action in **READY** status changes to **SUBMITTED** status if total current **RUNNING** and **SUBMITTED** actions are less than concurrency execution limit.
+
+A coordinator action in **SUBMITTED** status changes to **RUNNING** status when the workflow engine start execution of the coordinator action.
+
+A coordinator action is in **RUNNING** status until the associated workflow job completes its execution. Depending on the workflow job completion status, the coordinator action will be in **SUCCEEDED**, **KILLED** or **FAILED** status.
+
+A coordinator action in **WAITING**, **READY**, **SUBMITTED** or **RUNNING** status can be killed, changing to **KILLED** status.
+
+A coordinator action in **SUBMITTED** or **RUNNING** status can also fail, changing to **FAILED** status.
+
+A coordinator action in **FAILED**, **KILLED**, or **TIMEDOUT** status can be changed to **IGNORED** status. A coordinator action in **IGNORED** status can be
+ rerun, changing to **WAITING** status.
+
+Valid coordinator action status transitions are:
+
+ * **WAITING --> READY | TIMEDOUT | SKIPPED | KILLED**
+ * **READY --> SUBMITTED | SKIPPED | KILLED**
+ * **SUBMITTED --> RUNNING | KILLED | FAILED**
+ * **RUNNING --> SUCCEEDED | KILLED | FAILED**
+ * **FAILED | KILLED | TIMEDOUT --> IGNORED**
+ * **IGNORED --> WAITING**
+
+#### 6.1.4. Input Events
+
+The Input events of a coordinator application specify the input conditions that are required in order to execute a coordinator action.
+
+In the current specification input events are restricted to dataset instances availability.
+
+All the datasets instances defined as input events must be available for the coordinator action to be ready for execution ( **READY** status).
+
+Input events are normally parameterized. For example, the last 24 hourly instances of the 'searchlogs' dataset.
+
+Input events can be refer to multiple instances of multiple datasets. For example, the last 24 hourly instances of the 'searchlogs' dataset and the last weekly instance of the 'celebrityRumours' dataset.
+
+#### 6.1.5. Output Events
+
+A coordinator action can produce one or more dataset(s) instances as output.
+
+Dataset instances produced as output by one coordinator actions may be consumed as input by another coordinator action(s) of other coordinator job(s).
+
+The chaining of coordinator jobs via the datasets they produce and consume is referred as a **data pipeline.**
+
+In the current specification coordinator job output events are restricted to dataset instances.
+
+#### 6.1.6. Coordinator Action Execution Policies
+
+The execution policies for the actions of a coordinator job can be defined in the coordinator application.
+
+ * Timeout: A coordinator job can specify the timeout for its coordinator actions, this is, how long the coordinator action will be in *WAITING* or *READY* status before giving up on its execution.
+ * Concurrency: A coordinator job can specify the concurrency for its coordinator actions, this is, how many coordinator actions are allowed to run concurrently ( **RUNNING** status) before the coordinator engine starts throttling them.
+ * Execution strategy: A coordinator job can specify the execution strategy of its coordinator actions when there is backlog of coordinator actions in the coordinator engine. The different execution strategies are 'oldest first', 'newest first', 'none' and 'last one only'. A backlog normally happens because of delayed input data, concurrency control or because manual re-runs of coordinator jobs.
+ * Throttle: A coordinator job can specify the materialization or creation throttle value for its coordinator actions, this is, how many maximum coordinator actions are allowed to be in WAITING state concurrently.
+
+#### 6.1.7. Data Pipeline Application
+
+Commonly, multiple workflow applications are chained together to form a more complex application.
+
+Workflow applications are run on regular basis, each of one of them at their own frequency. The data consumed and produced by these workflow applications is relative to the nominal time of workflow job that is processing the data. This is a **coordinator application**.
+
+The output of multiple workflow jobs of a single workflow application is then consumed by a single workflow job of another workflow application, this is done on regular basis as well. These workflow jobs are triggered by recurrent actions of coordinator jobs. This is a set of **coordinator jobs** that inter-depend on each other via the data they produce and consume.
+
+This set of interdependent **coordinator applications** is referred as a **data pipeline application**.
+
+### 6.2. Synchronous Coordinator Application Example
+
+ * The `checkouts` synchronous dataset is created every 15 minutes by an online checkout store.
+ * The `hourlyRevenue` synchronous dataset is created every hour and contains the hourly revenue.
+ * The `dailyRevenue` synchronous dataset is created every day and contains the daily revenue.
+ * The `monthlyRevenue` synchronous dataset is created every month and contains the monthly revenue.
+
+ * The `revenueCalculator-wf` workflow consumes checkout data and produces as output the corresponding revenue.
+ * The `rollUpRevenue-wf` workflow consumes revenue data and produces a consolidated output.
+
+ * The `hourlyRevenue-coord` coordinator job triggers, every hour, a `revenueCalculator-wf` workflow. It specifies as input the last 4 `checkouts` dataset instances and it specifies as output a new instance of the `hourlyRevenue` dataset.
+ * The `dailyRollUpRevenue-coord` coordinator job triggers, every day, a `rollUpRevenue-wf` workflow. It specifies as input the last 24 `hourlyRevenue` dataset instances and it specifies as output a new instance of the `dailyRevenue` dataset.
+ * The `monthlyRollUpRevenue-coord` coordinator job triggers, once a month, a `rollUpRevenue-wf` workflow. It specifies as input all the `dailyRevenue` dataset instance of the month and it specifies as output a new instance of the `monthlyRevenue` dataset.
+
+This example contains describes all the components that conform a data pipeline: datasets, coordinator jobs and coordinator actions (workflows).
+
+The coordinator actions (the workflows) are completely agnostic of datasets and their frequencies, they just use them as input and output data (i.e. HDFS files or directories). Furthermore, as the example shows, the same workflow can be used to process similar datasets of different frequencies.
+
+The frequency of the `hourlyRevenue-coord` coordinator job is 1 hour, this means that every hour a coordinator action is created. A coordinator action will be executed only when the 4 `checkouts` dataset instances for the corresponding last hour are available, until then the coordinator action will remain as created (materialized), in **WAITING** status. Once the 4 dataset instances for the corresponding last hour are available, the coordinator action will be executed and it will start a `revenueCalculator-wf` workflow job.
+
+### 6.3. Synchronous Coordinator Application Definition
+
+A synchronous coordinator definition is a is defined by a name, start time and end time, the frequency of creation of its coordinator actions, the input events, the output events and action control information:
+
+ * **<font color="#0000ff"> start: </font>** The start datetime for the job. Starting at this time actions will be materialized. Refer to section #3 'Datetime Representation' for syntax details.
+ * **<font color="#0000ff"> end: </font>** The end datetime for the job. When actions will stop being materialized. Refer to section #3 'Datetime Representation' for syntax details.
+ * **<font color="#0000ff"> timezone:</font>** The timezone of the coordinator application.
+ * **<font color="#0000ff"> frequency: </font>** The frequency, in minutes, to materialize actions. Refer to section #4 'Time Interval Representation' for syntax details.
+ * Control information:
+ * **<font color="#0000ff"> timeout: </font>** The maximum time, in minutes, that a materialized action will be waiting for the additional conditions to be satisfied before being discarded. A timeout of `0` indicates that at the time of materialization all the other conditions must be satisfied, else the action will be discarded. A timeout of `0` indicates that if all the input events are not satisfied at the time of action materialization, the action should timeout immediately. A timeout of `-1` indicates no timeout, the materialized action will wait forever for the other conditions to be satisfied. The default value is `-1`. The timeout can only cause a `WAITING` action to transition to `TIMEDOUT`; once the data dependency is satisified, a `WAITING` action transitions to `READY`, and the timeout no longer has any affect, even if the action hasn't transitioned to `SUBMITTED` or `RUNNING` when it expires.
+ * **<font color="#0000ff"> concurrency: </font>** The maximum number of actions for this job that can be running at the same time. This value allows to materialize and submit multiple instances of the coordinator app, and allows operations to catchup on delayed processing. The default value is `1`.
+ * **<font color="#0000ff"> execution: </font>** Specifies the execution order if multiple instances of the coordinator job have satisfied their execution criteria. Valid values are:
+ ** `FIFO` (oldest first) **default*.
+ * `LIFO` (newest first).
+ * `LAST_ONLY` (see explanation below).
+ * `NONE` (see explanation below).
+ * **<font color="#0000ff"> throttle: </font>** The maximum coordinator actions are allowed to be in WAITING state concurrently. The default value is `12`.
+ * **<font color="#0000ff"> datasets: </font>** The datasets coordinator application uses.
+ * **<font color="#0000ff"> input-events: </font>** The coordinator job input events.
+ * **<font color="#0000ff"> data-in: </font>** It defines one job input condition that resolves to one or more instances of a dataset.
+ * **<font color="#0000ff"> name: </font>** input condition name.
+ * **<font color="#0000ff"> dataset: </font>** dataset name.
+ * **<font color="#0000ff"> instance: </font>** refers to a single dataset instance (the time for a synchronous dataset).
+ * **<font color="#0000ff"> start-instance: </font>** refers to the beginning of an instance range (the time for a synchronous dataset).
+ * **<font color="#0000ff"> end-instance: </font>** refers to the end of an instance range (the time for a synchronous dataset).
+ * **<font color="#0000ff"> output-events: </font>** The coordinator job output events.
+ * **<font color="#0000ff"> data-out: </font>** It defines one job output that resolves to a dataset instance.
+ * **<font color="#0000ff"> name: </font>** output name.
+ * **<font color="#0000ff"> dataset: </font>** dataset name.
+ * **<font color="#0000ff"> instance: </font>** dataset instance that will be generated by coordinator action.
+ * **<font color="#0000ff"> nocleanup: </font>** disable cleanup of the output dataset in rerun if true, even when nocleanup option is not used in CLI command.
+ * **<font color="#0000ff"> action: </font>** The coordinator action to execute.
+ * **<font color="#0000ff"> workflow: </font>** The workflow job invocation. Workflow job properties can refer to the defined data-in and data-out elements.
+
+**LAST_ONLY:** While `FIFO` and `LIFO` simply specify the order in which READY actions should be executed, `LAST_ONLY` can actually
+cause some actions to be SKIPPED and is a little harder to understand. When `LAST_ONLY` is set, an action that is `WAITING`
+or `READY` will be `SKIPPED` when the current time is past the next action's nominal time. For example, suppose action 1 and 2
+are both `READY`, the current time is 5:00pm, and action 2's nominal time is 5:10pm. In 10 minutes from now, at 5:10pm, action 1
+will become SKIPPED, assuming it doesn't transition to `SUBMITTED` (or a terminal state) before then. This sounds similar to the
+timeout control, but there are some important differences:
+
+ * The timeout time is configurable while the `LAST_ONLY` time is effectively the frequency.
+ * Reaching the timeout causes an action to transition to `TIMEDOUT`, which will cause the Coordinator Job to become `RUNNINGWITHERROR` and eventually `DONEWITHERROR`. With `LAST_ONLY`, an action becomes `SKIPPED` and the Coordinator Job remains `RUNNING` and eventually `DONE`.
+ * The timeout is looking satisfying the data dependency, while `LAST_ONLY` is looking at the action itself. This means that the timeout can only cause a transition from `WAITING`, while `LAST_ONLY` can cause a transition from `WAITING` or `READY`.
+
+`LAST_ONLY` is useful if you want a recurring job, but do not actually care about the individual instances and just
+always want the latest action. For example, if you have a coordinator running every 10 minutes and take Oozie down for 1 hour, when
+Oozie comes back, there would normally be 6 actions `WAITING` or `READY` to run. However, with `LAST_ONLY`, only the current one
+will go to `SUBMITTED` and then `RUNNING`; the others will go to `SKIPPED`.
+
+**NONE:** Similar to `LAST_ONLY` except instead of looking at the next action's nominal time, it looks
+at `oozie.coord.execution.none.tolerance` in oozie-site.xml (default is 1 minute). When `NONE` is set, an action that is `WAITING`
+or `READY` will be `SKIPPED` when the current time is more than the configured number of minutes (tolerance) past that action's
+nominal time. For example, suppose action 1 and 2 are both `READY`, the current time is 5:20pm, and both actions' nominal times are
+before 5:19pm. Both actions will become `SKIPPED`, assuming they don't transition to `SUBMITTED` (or a terminal state) before then.
+
+**<font color="#800080">Syntax: </font>**
+
+
+```
+ <coordinator-app name="[NAME]" frequency="[FREQUENCY]"
+ start="[DATETIME]" end="[DATETIME]" timezone="[TIMEZONE]"
+ xmlns="uri:oozie:coordinator:0.1">
+ <controls>
+ <timeout>[TIME_PERIOD]</timeout>
+ <concurrency>[CONCURRENCY]</concurrency>
+ <execution>[EXECUTION_STRATEGY]</execution>
+ </controls>
+.
+ <datasets>
+ <include>[SHARED_DATASETS]</include>
+ ...
+.
+ <!-- Synchronous datasets -->
+ <dataset name="[NAME]" frequency="[FREQUENCY]"
+ initial-instance="[DATETIME]" timezone="[TIMEZONE]">
+ <uri-template>[URI_TEMPLATE]</uri-template>
+ </dataset>
+ ...
+.
+ </datasets>
+.
+ <input-events>
+ <data-in name="[NAME]" dataset="[DATASET]">
+ <instance>[INSTANCE]</instance>
+ ...
+ </data-in>
+ ...
+ <data-in name="[NAME]" dataset="[DATASET]">
+ <start-instance>[INSTANCE]</start-instance>
+ <end-instance>[INSTANCE]</end-instance>
+ </data-in>
+ ...
+ </input-events>
+ <output-events>
+ <data-out name="[NAME]" dataset="[DATASET]">
+ <instance>[INSTANCE]</instance>
+ </data-out>
+ ...
+ </output-events>
+ <action>
+ <workflow>
+ <app-path>[WF-APPLICATION-PATH]</app-path>
+ <configuration>
+ <property>
+ <name>[PROPERTY-NAME]</name>
+ <value>[PROPERTY-VALUE]</value>
+ </property>
+ ...
+ </configuration>
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+**<font color="#008000"> Examples: </font>**
+
+**1. A Coordinator Job that creates an executes a single coordinator action:**
+
+The following example describes a synchronous coordinator application that runs once a day for 1 day at the end of the day. It consumes an instance of a daily 'logs' dataset and produces an instance of a daily 'siteAccessStats' dataset.
+
+**Coordinator application definition:**
+
+
+```
+ <coordinator-app name="hello-coord" frequency="${coord:days(1)}"
+ start="2009-01-02T08:00Z" end="2009-01-02T08:00Z"
+ timezone="America/Los_Angeles"
+ xmlns="uri:oozie:coordinator:0.1">
+ <datasets>
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
+ <uri-template>hdfs://bar:8020/app/logs/${YEAR}${MONTH}/${DAY}/data</uri-template>
+ </dataset>
+ <dataset name="siteAccessStats" frequency="${coord:days(1)}"
+ initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
+ <uri-template>hdfs://bar:8020/app/stats/${YEAR}/${MONTH}/${DAY}/data</uri-template>
+ </dataset>
+ </datasets>
+ <input-events>
+ <data-in name="input" dataset="logs">
+ <instance>2009-01-02T08:00Z</instance>
+ </data-in>
+ </input-events>
+ <output-events>
+ <data-out name="output" dataset="siteAccessStats">
+ <instance>2009-01-02T08:00Z</instance>
+ </data-out>
+ </output-events>
+ <action>
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>${coord:dataIn('input')}</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>${coord:dataOut('output')}</value>
+ </property>
+ </configuration>
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+There are 2 synchronous datasets with a daily frequency and they are expected at the end of each PST8PDT day.
+
+This coordinator job runs for 1 day on January 1st 2009 at 24:00 PST8PDT.
+
+The workflow job invocation for the single coordinator action would resolve to:
+
+
+```
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>hdfs://bar:8020/app/logs/200901/02/data</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>hdfs://bar:8020/app/stats/2009/01/02/data</value>
+ </property>
+ </configuration>
+ </workflow>
+```
+
+IMPORTANT: Note Oozie works in UTC datetimes, all URI templates resolve to UTC datetime values. Because of the timezone difference between UTC and PST8PDT, the URIs resolves to `2009-01-02T08:00Z` (UTC) which is equivalent to 2009-01-01T24:00PST8PDT= (PST).
+
+There is single input event, which resolves to January 1st PST8PDT instance of the 'logs' dataset. There is single output event, which resolves to January 1st PST8PDT instance of the 'siteAccessStats' dataset.
+
+The `${coord:dataIn(String name)}` and `${coord:dataOut(String name)}` EL functions resolve to the dataset instance URIs of the corresponding dataset instances. These EL functions are properly defined in a subsequent section.
+
+Because the `${coord:dataIn(String name)}` and `${coord:dataOut(String name)}` EL functions resolve to URIs, which are HDFS URIs, the workflow job itself does not deal with dataset instances, just HDFS URIs.
+
+**2. A Coordinator Job that executes its coordinator action multiple times:**
+
+A more realistic version of the previous example would be a coordinator job that runs for a year creating a daily action an consuming the daily 'logs' dataset instance and producing the daily 'siteAccessStats' dataset instance.
+
+The coordinator application is identical, except for the frequency, 'end' date and parameterization in the input and output events sections:
+
+
+```
+ <coordinator-app name="hello-coord" frequency="${coord:days(1)}"
+ start="2009-01-02T08:00Z" end="2010-01-02T08:00Z"
+ timezone="America/Los_Angeles"
+ xmlns="uri:oozie:coordinator:0.1">
+ <datasets>
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
+ <uri-template>hdfs://bar:8020/app/logs/${YEAR}${MONTH}/${DAY}/data</uri-template>
+ </dataset>
+ <dataset name="siteAccessStats" frequency="${coord:days(1)}"
+ initial-instance="2009-01-02T08:00Z" timezone="America/Los_Angeles">
+ <uri-template>hdfs://bar:8020/app/stats/${YEAR}/${MONTH}/${DAY}/data</uri-template>
+ </dataset>
+ </datasets>
+ <input-events>
+ <data-in name="input" dataset="logs">
+ <instance>${coord:current(0)}</instance>
+ </data-in>
+ </input-events>
+ <output-events>
+ <data-out name="output" dataset="siteAccessStats">
+ <instance>${coord:current(0)}</instance>
+ </data-out>
+ </output-events>
+ <action>
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>${coord:dataIn('input')}</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>${coord:dataOut('output')}</value>
+ </property>
+ </configuration>
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+The `${coord:current(int offset)}` EL function resolves to coordinator action creation time, that would be the current day at the time the coordinator action is created: `2009-01-02T08:00 ... 2010-01-01T08:00`. This EL function is properly defined in a subsequent section.
+
+There is single input event, which resolves to the current day instance of the 'logs' dataset.
+
+There is single output event, which resolves to the current day instance of the 'siteAccessStats' dataset.
+
+The workflow job invocation for the first coordinator action would resolve to:
+
+
+```
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>hdfs://bar:8020/app/logs/200901/02/data</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>hdfs://bar:8020/app/stats/2009/01/02/data</value>
+ </property>
+ </configuration>
+ </workflow>
+```
+
+For the second coordinator action it would resolve to:
+
+
+```
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>hdfs://bar:8020/app/logs/200901/03/data</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>hdfs://bar:8020/app/stats/2009/01/03/data</value>
+ </property>
+ </configuration>
+ </workflow>
+```
+
+And so on.
+
+**3. A Coordinator Job that executes its coordinator action multiple times and as input takes multiple dataset instances:**
+
+The following example is a variation of the example #2 where the synchronous coordinator application runs weekly. It consumes the of the last 7 instances of a daily 'logs' dataset and produces an instance of a weekly 'weeklySiteAccessStats' dataset.
+
+'logs' is a synchronous dataset with a daily frequency and it is expected at the end of each day (24:00).
+
+'weeklystats' is a synchronous dataset with a weekly frequency and it is expected at the end (24:00) of every 7th day.
+
+The coordinator application frequency is weekly and it starts on the 7th day of the year:
+
+
+```
+ <coordinator-app name="hello2-coord" frequency="${coord:days(7)}"
+ start="2009-01-07T24:00Z" end="2009-12-12T24:00Z"
+ timezone="UTC"
+ xmlns="uri:oozie:coordinator:0.1">
+ <datasets>
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-01-01T24:00Z" timezone="UTC">
+ <uri-template>hdfs://bar:8020/app/logs/${YEAR}${MONTH}/${DAY}</uri-template>
+ </dataset>
+ <dataset name="weeklySiteAccessStats" frequency="${coord:days(7)}"
+ initial-instance="2009-01-07T24:00Z" timezone="UTC">
+ <uri-template>hdfs://bar:8020/app/weeklystats/${YEAR}/${MONTH}/${DAY}</uri-template>
+ </dataset>
+ </datasets>
+ <input-events>
+ <data-in name="input" dataset="logs">
+ <start-instance>${coord:current(-6)}</start-instance>
+ <end-instance>${coord:current(0)}</end-instance>
+ </data-in>
+ </input-events>
+ <output-events>
+ <data-out name="output" dataset="siteAccessStats">
+ <instance>${coord:current(0)}</instance>
+ </data-out>
+ </output-events>
+ <action>
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>${coord:dataIn('input')}</value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>${coord:dataOut('output')}</value>
+ </property>
+ </configuration>
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+The `${coord:current(int offset)}` EL function resolves to coordinator action creation time minus the specified offset multiplied by the dataset frequency. This EL function is properly defined in a subsequent section.
+
+The input event, instead resolving to a single 'logs' dataset instance, it refers to a range of 7 dataset instances - the instance for 6 days ago, 5 days ago, ... and today's instance.
+
+The output event resolves to the current day instance of the 'weeklySiteAccessStats' dataset. As the coordinator job will create a coordinator action every 7 days, dataset instances for the 'weeklySiteAccessStats' dataset will be created every 7 days.
+
+The workflow job invocation for the first coordinator action would resolve to:
+
+
+```
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>
+ hdfs://bar:8020/app/logs/200901/01,hdfs://bar:8020/app/logs/200901/02,
+ hdfs://bar:8020/app/logs/200901/03,hdfs://bar:8020/app/logs/200901/05,
+ hdfs://bar:8020/app/logs/200901/05,hdfs://bar:8020/app/logs/200901/06,
+ hdfs://bar:8020/app/logs/200901/07
+ </value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>hdfs://bar:8020/app/stats/2009/01/07</value>
+ </property>
+ </configuration>
+ </workflow>
+```
+
+For the second coordinator action it would resolve to:
+
+
+```
+ <workflow>
+ <app-path>hdfs://bar:8020/usr/joe/logsprocessor-wf</app-path>
+ <configuration>
+ <property>
+ <name>wfInput</name>
+ <value>
+ hdfs://bar:8020/app/logs/200901/08,hdfs://bar:8020/app/logs/200901/09,
+ hdfs://bar:8020/app/logs/200901/10,hdfs://bar:8020/app/logs/200901/11,
+ hdfs://bar:8020/app/logs/200901/12,hdfs://bar:8020/app/logs/200901/13,
+ hdfs://bar:8020/app/logs/200901/16
+ </value>
+ </property>
+ <property>
+ <name>wfOutput</name>
+ <value>hdfs://bar:8020/app/stats/2009/01/16</value>
+ </property>
+ </configuration>
+ </workflow>
+```
+
+And so on.
+
+### 6.4. Asynchronous Coordinator Application Definition
+ * TBD
+
+### 6.5. Parameterization of Coordinator Applications
+
+When a coordinator job is submitted to Oozie, the submitter may specify as many coordinator job configuration properties as required (similar to Hadoop JobConf properties).
+
+Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as `${NAME}` variables within the coordinator application definition.
+
+Configuration Properties that are not a valid Java identifier, for example `job.tracker`, are available via the `${coord:conf(String name)}` function. Valid Java identifier properties are available via this function as well.
+
+Using properties that are valid Java identifiers result in a more readable and compact definition.
+
+Dataset definitions can be also parameterized, the parameters are resolved using the configuration properties of Job configuration used to submit the coordinator job.
+
+If a configuration property used in the definitions is not provided with the job configuration used to submit a coordinator job, the value of the parameter will be undefined and the job submission will fail.
+
+**<font color="#008000"> Example: </font>**
+
+Coordinator application definition:
+
+
+```
+ <coordinator-app name="app-coord" frequency="${coord:days(1)}"
+ start="${jobStart}" end="${jobEnd}" timezone="${timezone}"
+ xmlns="uri:oozie:coordinator:0.1">
+ <datasets>
+ <dataset name="logs" frequency="${coord:hours(1)}"
+ initial-instance="${logsInitialInstance}" timezone="${timezone}">
+ <uri-template>
+ hdfs://bar:8020/app/logs/${market}/${language}/${YEAR}${MONTH}/${DAY}/${HOUR}
+ </uri-template>
+ </dataset>
+ </datasets>
+ <input-events>
+ <data-in name="input" dataset="logs">
+ <start-instance>${coord:current(-23)}</start-instance>
+ <end-instance>${coord:current(0)}</end-instance>
+ </data-in>
+ </input-events>
+ <action>
+ <workflow>
+ ...
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+In the above example there are 6 configuration parameters (variables) that have to be provided when submitting a job:
+
+ * `jobStart` : start datetime for the job, in UTC
+ * `jobEnd` : end datetime for the job, in UTC
+ * `logsInitialInstance` : expected time of the first logs instance, in UTC
+ * `timezone` : timezone for the job and the dataset
+ * `market` : market to compute by this job, used in the uri-template
+ * `language` : language to compute by this job, used in the uri-template
+
+IMPORTANT: Note that this example is not completely correct as it always consumes the last 24 instances of the 'logs' dataset. It is assumed that all days have 24 hours. For timezones that observe daylight saving this application will not work as expected as it will consume the wrong number of dataset instances in DST switch days. To be able to handle these scenarios, the `${coord:hoursInDays(int n)}` and `${coord:daysInMonths(int n)}` EL functions must be used (refer to section #6.6.2 and #6.6.3).
+
+If the above 6 properties are not specified, the job will fail.
+
+As of schema 0.4, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
+properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
+
+**Example:**
+
+The previous parameterized coordinator application definition with formal parameters:
+
+
+```
+ <coordinator-app name="app-coord" frequency="${coord:days(1)}"
+ start="${jobStart}" end="${jobEnd}" timezone="${timezone}"
+ xmlns="uri:oozie:coordinator:0.1">
+ <parameters>
+ <property>
+ <name>jobStart</name>
+ </property>
+ <property>
+ <name>jobEnd</name>
+ <value>2012-12-01T22:00Z</value>
+ </property>
+ </parameters>
+ <datasets>
+ <dataset name="logs" frequency="${coord:hours(1)}"
+ initial-instance="${logsInitialInstance}" timezone="${timezone}">
+ <uri-template>
+ hdfs://bar:8020/app/logs/${market}/${language}/${YEAR}${MONTH}/${DAY}/${HOUR}
+ </uri-template>
+ </dataset>
+ </datasets>
+ <input-events>
+ <data-in name="input" dataset="logs">
+ <start-instance>${coord:current(-23)}</start-instance>
+ <end-instance>${coord:current(0)}</end-instance>
+ </data-in>
+ </input-events>
+ <action>
+ <workflow>
+ ...
+ </workflow>
+ </action>
+ </coordinator-app>
+```
+
+In the above example, if `jobStart` is not specified, Oozie will print an error message instead of submitting the job. If
+`jobEnd` is not specified, Oozie will use the default value, `2012-12-01T22:00Z`.
+
+### 6.6. Parameterization of Dataset Instances in Input and Output Events
+
+A coordinator application job typically launches several coordinator actions during its lifetime. A coordinator action typically uses its creation (materialization) time to resolve the specific datasets instances required for its input and output events.
+
+The following EL functions are the means for binding the coordinator action creation time to the datasets instances of its input and output events.
+
+#### 6.6.1. coord:current(int n) EL Function for Synchronous Datasets
+
+`${coord:current(int n)}` represents the n<sup>th</sup> dataset instance for a **synchronous** dataset, relative to the coordinator action creation (materialization) time. The coordinator action creation (materialization) time is computed based on the coordinator job start time and its frequency. The n<sup>th</sup> dataset instance is computed based on the dataset's initial-instance datetime, its frequency and the (current) coordinator action creation (materialization) time.
+
+`n` can be a negative integer, zero or a positive integer.
+
+`${coord:current(int n)}` returns the nominal datetime for n<sup>th</sup> dataset instance relative to the coordinator action creation (materialization) time.
+
+`${coord:current(int n)}` performs the following calculation:
+
+
+```
+DS_II : dataset initial-instance (datetime)
+DS_FREQ: dataset frequency (minutes)
+CA_NT: coordinator action creation (materialization) nominal time
+
+coord:current(int n) = DS_II + DS_FREQ * ( (CA_NT - DS_II) div DS_FREQ + n)
+```
+
+NOTE: The formula above is not 100% correct, because DST changes the calculation has to account for hour shifts. Oozie Coordinator must make the correct calculation accounting for DST hour shifts.
+
+When a positive integer is used with the `${coord:current(int n)}`, it refers to a d
<TRUNCATED>