You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oozie.apache.org by ro...@apache.org on 2013/07/12 16:48:38 UTC

svn commit: r1502578 - in /oozie/branches/branch-4.0/docs/src/site/twiki: DG_JMSNotifications.twiki DG_SLAMonitoring.twiki

Author: rohini
Date: Fri Jul 12 14:48:37 2013
New Revision: 1502578

URL: http://svn.apache.org/r1502578
Log:
OOZIE-1249 SLA Documentation - added missing new files (mona,virag,rohini via rohini)

Added:
    oozie/branches/branch-4.0/docs/src/site/twiki/DG_JMSNotifications.twiki
    oozie/branches/branch-4.0/docs/src/site/twiki/DG_SLAMonitoring.twiki

Added: oozie/branches/branch-4.0/docs/src/site/twiki/DG_JMSNotifications.twiki
URL: http://svn.apache.org/viewvc/oozie/branches/branch-4.0/docs/src/site/twiki/DG_JMSNotifications.twiki?rev=1502578&view=auto
==============================================================================
--- oozie/branches/branch-4.0/docs/src/site/twiki/DG_JMSNotifications.twiki (added)
+++ oozie/branches/branch-4.0/docs/src/site/twiki/DG_JMSNotifications.twiki Fri Jul 12 14:48:37 2013
@@ -0,0 +1,216 @@
+<noauolink>
+
+[[index][::Go back o Oozie Documentation Index::]]
+
+---+!! JMS Notifications
+
+%TOC%
+
+---++ Overview
+Since Oozie 4.0, Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss
+events. This provides an alternative to polling Oozie for Job or SLA related information and getting events as they
+happen without any delay. Clients can be written to consume these notifications and integrate with different monitoring
+and alerting systems.
+
+---++ Oozie Server Configuration
+Refer to [[AG_Install#Notifications_Configuration][Notifications Configuration]] section of [[AG_Install][Oozie Install]]
+documentation for the Oozie server side configuration required to support publishing notifications to a JMS Provider.
+The JNDI properties for the JMS provider, the topics to publish to and the notification types to publish (Job and/or SLA)
+need to be configured.
+
+---++ Consuming Notifications
+
+---+++ Notification types
+Job and SLA notifications are published to the configured JMS Provider on the configured topics.
+
+Job status change notifications include job start, success, failure, suspended, etc. Currently only workflow job and
+coordinator action status change notifications are published.
+
+SLA notifications include START_MET, END_MET, DURATION_MET, START_MISS, END_MISS, DURATION_MISS events and are published
+for a workflow job, workflow action or coordinator action for which SLA information is configured in the job xml. Refer
+to [[DG_SLAMonitoring#Configuring_SLA_in_Applications][SLA Configuration]] for information on configuring SLA for a workflow or
+coordinator.
+
+---+++ JMS Topic
+Consumers interested in notification on events will require to know the JNDI properties to connect to the JMS provider.
+They will also need to know the JMS topic on which notifications for a particular job are published.
+
+Oozie Client provides the following APIs :
+<verbatim>
+public JMSConnectionInfo getJMSConnectionInfo()
+public String getJMSTopicName(String jobId)
+</verbatim>
+
+The JMSConnectionInfo exposes 3 methods:
+
+<verbatim>
+Properties getJNDIProperties();
+String getTopicPattern(AppType appType);
+String getTopicPrefix();
+</verbatim>
+
+The topic is obtained by concatenating topic prefix and the substituted value for topic pattern. The topic pattern
+can be a constant value like workflow or coordinator which the administrator has configured or ${username}.
+If ${username}, it has to be substitued with the name of the user who has submitted the job. Administrators can chose
+to publish messages to topics containing user names to avoid having one topic containing all messages and all users having
+to apply selectors to filter the message they are interested in.
+
+The getJMSTopicName API can be used if the job id is already known and will give the exact topic name to which the
+notifications for that job are published.
+
+
+---+++ JMS Message Format
+JMS messages published are =javax.jms.TextMessage=. The body contains JSON and the header contains multiple properties
+that can be used as selectors. The header properties are not repeated in the body of the message to keep the messages
+small.
+
+<b>Message Header:</b> <br/>
+The different header properties are:
+   * msgType - Value can be JOB or SLA.
+   * user - The user who submitted the job.
+   * appName - Application name of the job.
+   * appType - Type of the job. One of WORKFLOW_JOB, WORKFLOW_ACTION, COORDINATOR_ACTION
+   * slaStatus - Applicable only to SLA messages. Value is one of NOT_STARTED, IN_PROCESS, MET, MISS
+   * eventStatus - It takes one of the following values- START_MET, START_MISS, DURATION_MET, DURATION_MISS, END_MET, END_MISS for SLA notifications. It can take any of the following values- WAITING, STARTED, SUCCESS, SUSPEND, FAILURE for job notifications. Note that event status is different from the Job status. It is included in the header to provide better filtering. Below is the mapping of event status to the actual status of workflow job or coordinator action.
+
+WAITING = When a Coordinator Action is waiting for input dependency
+
+STARTED = When the Workflow Job or Coordinator Action is in RUNNING state
+
+SUCCESS = When the Workflow Job or Coordinator Action is in SUCCEEDED state
+
+SUSPEND = When the Workflow Job or Coordinator Action is in SUSPENDED state
+
+FAILURE = When the Workflow Job or Coordinator Action is in terminal state other than SUCCEEDED.
+
+<b>Message Body for Job Notifications:</b> <br/>
+    Sample JSON response for different job and sla events as below.
+
+<verbatim>
+Workflow Job in RUNNING state:
+{"status":"RUNNING","id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000}
+</verbatim>
+
+<verbatim>
+Workflow Job in FAILED state:
+{"status":"FAILED","errorCode":"EL_ERROR","errorMessage":"variable [dummyvalue] cannot be resolved",
+"id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000,"endTime":1366672183543}
+</verbatim>
+
+<verbatim>
+Workflow Job in SUCCEEDED state:
+{"status":"SUCCEEDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
+"parentId":"0000025-130618221729631-oozie-oozi-C@1","endTime":1366676224154}
+</verbatim>
+
+<verbatim>
+Workflow Job in SUSPENDED state:
+{"status":"SUSPENDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
+"parentId":"0000025-130618221729631-oozie-oozi-C@1"}
+</verbatim>
+
+<verbatim>
+Coordinator Action in WAITING state:
+{"status":"WAITING","nominalTime":1310342400000,"missingDependency":"hdfs://gsbl90107.blue.com:8020/user/john/dir1/file1",
+"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
+</verbatim>
+
+<verbatim>
+Coordinator Action in RUNNING state:
+{"status":"RUNNING","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
+"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
+</verbatim>
+
+<verbatim>
+Coordinator Action in SUCCEEDED state:
+{"status":"SUCCEEDED","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
+"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677082799}
+</verbatim>
+
+<verbatim>
+Coordinator Action in FAILED state:
+{"status":"FAILED","errorCode":"E0101","errorMessage":"dummyError","nominalTime":1310342400000,
+"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,
+"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677140818}
+</verbatim>
+
+<b>Message Body for SLA Notifications:</b> <br/>
+
+<verbatim>
+Workflow Job in sla END_MISS state:
+{"id":"0000000-000000000000001-oozie-wrkf-C@1","parentId":"0000000-000000000000001-oozie-wrkf-C",
+"expectedStartTime":1356998400000,"notificationMessage":"notification of start miss","actualStartTime":1357002000000,
+"expectedDuration":-1, "actualDuration":3600,"expectedEndTime":1356998400000,"actualEndTime":1357002000000}
+</verbatim>
+
+---+++ JMS Client
+
+Oozie provides a helper class JMSMessagingUtils for consumers to deserialize the JMS messages back to Java objects.
+The below method getEventMessage() expects a sub type of EventMessage.
+There are different implementations of EventMessage - WorkflowJobMessage, CoordinatorActionMessage and SLAMessage.
+
+<verbatim>
+<T extends EventMessage> T JMSMessagingUtils.getEventMessage(Message jmsMessage)
+</verbatim>
+---++++ Example
+Below is a sample code to consume notifications.
+
+First, create the Oozie client and retrieve the JNDI properties to make a connection to the JMS server.
+<verbatim>
+   OozieClient oc = new OozieClient("http://localhost:11000/oozie");
+   JMSConnectionInfo jmsInfo = oc.getJMSConnectionInfo();
+   Properties jndiProperties = jmsInfo.getJNDIProperties();
+   Context jndiContext = new InitialContext(jndiProperties);
+   String connectionFactoryName = (String) jndiContext.getEnvironment().get("connectionFactoryNames");
+   ConnectionFactory connectionFactory = (ConnectionFactory);
+   jndiContext.lookup(connectionFactoryName);
+   Connection connection = connectionFactory.createConnection();
+   Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
+   String topicPrefix = jmsInfo.getTopicPrefix();
+   String topicPattern = jmsInfo.getTopicPattern(AppType.WORKFLOW_JOB);
+   // Following code checks if the topic pattern is
+   //'username', then the topic name is set to the actual user submitting the job
+   String topicName = null;
+   if (topicPattern.equals("${username}")) {
+       topicName = "john";
+   }
+   Destination topic = session.createTopic(topicPrefix+topicName);
+   MessageConsumer consumer = session.createConsumer(topic);
+   consumer.setMessageListener(this);
+   connection.start();
+</verbatim>
+
+To start receiving messages, the JMS [[http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html][MessageListener]]
+interface needs to be implemented. Also, its onMessage() method  needs to be implemented.
+This method will be called whenever a message is available on the JMS bus.
+
+<verbatim>
+    public void onMessage(Message m) {
+       if (message.getStringProperty(JMSHeaderConsants.MSG_TYPE).equals(MessageType.SLA.name()){
+          SLAMessage slaMessage = JMSMessagingUtils.getEventMessage(message);
+          // Further processing
+       }
+       else if (message.getStringProperty(JMSHeaderConsants.APP_TYPE).equals(AppType.WORKFLOW_JOB.name()){
+          WorkflowJobMessage wfJobMessage = JMSMessagingUtils.getEventMessage(message);
+          // Further processing
+       }
+    }
+</verbatim>
+
+---++++ Applying Selectors
+
+Below is a sample ActiveMQ text message header properties section.
+<verbatim>
+ActiveMQTextMessage
+{properties = {appName = map-reduce-wf, msgType=JOB, appType=WORKFLOW_JOB, user=john, msgFormat=json, eventStatus=STARTED} ...}
+</verbatim>
+
+On the header properties, consumers can apply JMS selectors to filter messages from JMS provider.
+They are listed at [[../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html][JMSHeaderConstants]]
+
+Sample use of selector to filter events related to Job which have failed and has a particular app-name
+
+<verbatim>
+String selector=JMSHeaderConstants.EVENT_STATUS + "='FAILURE' AND " + JMSHeaderConstants.APP_NAME + "='app-name'";
+MessageConsumer consumer = session.createConsumer(topic, selector);
+</verbatim>

Added: oozie/branches/branch-4.0/docs/src/site/twiki/DG_SLAMonitoring.twiki
URL: http://svn.apache.org/viewvc/oozie/branches/branch-4.0/docs/src/site/twiki/DG_SLAMonitoring.twiki?rev=1502578&view=auto
==============================================================================
--- oozie/branches/branch-4.0/docs/src/site/twiki/DG_SLAMonitoring.twiki (added)
+++ oozie/branches/branch-4.0/docs/src/site/twiki/DG_SLAMonitoring.twiki Fri Jul 12 14:48:37 2013
@@ -0,0 +1,299 @@
+<noautolink>
+
+[[index][::Go back to Oozie Documentation Index::]]
+
+---+!! Oozie SLA Monitoring
+
+%TOC%
+
+---++ Overview
+
+Critical jobs can have certain SLA requirements associated with them. This SLA can be in terms of time
+i.e. a maximum allowed time limit associated with when the job should start, by when should it end,
+and its duration of run. Oozie workflows and coordinators allow defining such SLA limits in the application definition xml.
+
+With the addition of SLA Monitoring, Oozie can now actively monitor the state of these SLA-sensitive jobs
+and send out notifications for SLA mets and misses.
+
+In versions earlier than 4.x, this was a passive feature where users needed to query the Oozie client SLA API
+to fetch the records regarding job status changes, and use their own custom calculation engine to compute
+whether SLA was met or missed, based on initial definition of time limits.
+
+Oozie now also has a SLA tab in the Oozie UI, where users can query for SLA information and have a summarized view
+of how their jobs fared against their SLAs.
+
+
+---++ Oozie Server Configuration
+
+Refer to [[AG_Install#Notifications_Configuration][Notifications Configuration]] for configuring Oozie server to track
+SLA for jobs and send notifications.
+
+---++ SLA Tracking
+
+Oozie allows tracking SLA for meeting the following criteria:
+   * Start time
+   * End time
+   * Job Duration
+
+---++++ Event Status
+Corresponding to each of these 3 criteria, your jobs are processed for whether Met or Miss i.e.
+   * START_MET, START_MISS
+   * END_MET, END_MISS
+   * DURATION_MET, DURATION_MISS
+
+---++++ SLA Status
+Expected end-time is the most important criterion for majority of users while deciding overall SLA Met or Miss.
+Hence the _"SLA_Status"_ for a job will transition through these four stages
+   * Not_Started <-- Job not yet begun
+   * In_Process <-- Job started and is running, and SLAs are being tracked
+   * Met <-- caused by an END_MET
+   * Miss <-- caused by an END_MISS
+
+In addition to overshooting expected end-time, and END_MISS (and so an eventual SLA MISS) also occurs when the
+job does not end successfully e.g. goes to error state - Failed/Killed/Error/Timedout.
+
+---++ Configuring SLA in Applications
+
+To make your jobs trackable for SLA, you simply need to add the =<sla:info>= tag to your workflow application definition.
+If you were already using the existing SLA schema in your workflows (Schema xmlns:sla="uri:oozie:sla:0.1"), you dont need to
+do anything extra to receive SLA notifications via JMS messages. This new SLA monitoring framework is backward-compatible -
+no need to change application XML for now and you can continue to fetch old records via the [[DG_CommandLineTool#SLAOperations][command line API]].
+However, usage of old schema and API is deprecated and we strongly recommend using new schema.
+   * New SLA schema is 'uri:oozie:sla:0.2'
+   * In order to use new SLA schema, you will need to upgrade your workflow/coordinator schema to 0.5 i.e. 'uri:oozie:workflow:0.5'
+
+---+++ SLA Definition in Workflow
+Example:
+<verbatim>
+<workflow-app name="test-wf-job-sla"
+              xmlns="uri:oozie:workflow:0.5"
+              xmlns:sla="uri:oozie:sla:0.2">
+    <start to="grouper"/>
+    <action name="grouper">
+        <map-reduce>
+            <job-tracker>jt</job-tracker>
+            <name-node>nn</name-node>
+            <configuration>
+                <property>
+                    <name>mapred.input.dir</name>
+                    <value>input</value>
+                </property>
+                <property>
+                    <name>mapred.output.dir</name>
+                    <value>output</value>
+                </property>
+            </configuration>
+        </map-reduce>
+        <ok to="end"/>
+        <error to="end"/>
+    </action>
+    <end name="end"/>
+    <sla:info>
+        <sla:nominal-time>${nominal_time}</sla:nominal-time>
+        <sla:should-start>${10 * MINUTES}</sla:should-start>
+        <sla:should-end>${30 * MINUTES}</sla:should-end>
+        <sla:max-duration>${30 * MINUTES}</sla:max-duration>
+        <sla:alert-events>start_miss,end_miss,duration_miss</sla:alert-events>
+        <sla:alert-contact>joe@example.com</sla:alert-contact>
+    </sla:info>
+</workflow-app>
+</verbatim>
+
+For the list of tags usable under =<sla:info>=, refer to [[WorkflowFunctionalSpec#SLASchema][Schemas Appendix]].
+This new schema is much more compact and meaningful, getting rid of redundant and unused tags.
+
+   * ==nominal-time==: As the name suggests, this is the time relative to which your jobs' SLAs will be calculated. Generally since Oozie workflows are aligned with synchronous data dependencies, this nominal time can be parameterized to be passed the value of your coordinator nominal time. Nominal time is also required in case of independent workflows and you can specify the time in which you expect the workflow to be run if you don't have a synchronous dataset associated with it.
+   * ==should-start==: Relative to =nominal-time= this is the amount of time (alongwith time-unit) within which, your job should *start running*, to meet SLA. This is optional.
+   * ==should-end==: Relative to =nominal-time= this is the amount of time (alongwith time-unit) within which, your job should *finish*, to meet SLA.
+   * ==max-duration==: This is the maximum amount of time (alongwith time-unit) for your job run, to meet SLA. This is optional.
+   * ==alert-events==: Specify the types of events for which *Email* alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not neccessary. However, note that this setting is only for alerts via *email* alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured.
+   * ==alert-contact==: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts.
+
+NOTE: All tags can be parameterized.
+
+Same schema can be applied to and embedded under Workflow-Action as well as Coordinator-Action XML.
+
+---+++ SLA Definition in Workflow Action
+
+<verbatim>
+<workflow-app name="test-wf-action-sla" xmlns="uri:oozie:workflow:0.5" xmlns:sla="uri:oozie:sla:0.2">
+    <start to="grouper"/>
+    <action name="grouper">
+        ...
+        <ok to="end"/>
+        <error to="end"/>
+        <sla:info>
+            <sla:nominal-time>${nominal_time}</sla:nominal-time>
+            <sla:should-start>${10 * MINUTES}</sla:should-start>
+        ...
+        </sla:info>
+    </action>
+    <end name="end"/>
+</workflow-app>
+</verbatim>
+
+---+++ SLA Definition in Coordinator Action
+<verbatim>
+<coordinator-app name="test-coord-sla" frequency="${coord:days(1)}" freq_timeunit="DAY"
+    end_of_duration="NONE" start="2013-06-20T08:01Z" end="2013-12-01T08:01Z"
+    timezone="America/Los_Angeles" xmlns="uri:oozie:coordinator:0.4" xmlns:sla="uri:oozie:sla:0.2">
+    <action>
+        <workflow>
+            <app-path>${wfAppPath}</app-path>
+        </workflow>
+        <sla:info>
+            <sla:nominal-time>${nominal_time}</sla:nominal-time>
+            ...
+        </sla:info>
+    </action>
+</coordinator-app>
+</verbatim>
+
+---++ Accessing SLA Information
+
+SLA information is accessible via the following two ways
+   * Through the SLA tab of the Oozie Web UI.
+   * JMS messages sent to a configured JMS provider for instantaneous tracking.
+   * RESTful API to query for SLA summary.
+
+For JMS Notifications, you have to have a message broker in place, on which Oozie publishes messages and you can
+hook on a subscriber to receive those messages. For more info on setting up and consuming JMS messages, refer
+[[DG_JMSNotifications][JMS Notifications]] documentation.
+
+In the REST API, the following filters can be applied while fetching SLA information:
+   * app_name - Application name
+   * id  - id of the workflow job, workflow action or coordinator action
+   * parent_id - Parent id of the workflow job, workflow action or coordinator action
+   * nominal_start and nominal_end - Start and End range for nominal time of the workflow or coordinator.
+
+When timezone query parameter is specified, the expected and actual start/end time returned is formatted. If not specified,
+the number of milliseconds that have elapsed since January 1, 1970 00:00:00.000 GMT is returned.
+
+The examples below demonstrate the use of REST API and explains the JSON response.
+
+---+++ Scenario 1: Workflow Job Start_Miss
+*Request:*
+<verbatim>
+GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=nominal_start=2013-06-18T00:01Z;nominal_end=2013-06-23T00:01Z;app_name=my-sla-app
+</verbatim>
+
+*JSON Response*
+<verbatim>
+{
+
+    id : "000056-1238791320234-oozie-joe-W"
+    parentId : "000001-1238791320234-oozie-joe-C@8"
+    appType : "WORKFLOW_JOB"
+    msgType : "SLA"
+    appName : "my-sla-app"
+    slaStatus : "IN_PROCESS"
+    eventStatus : "START_MISS"
+    user: "joe"
+    nominalTime: "2013-16-22T05:00Z"
+    expectedStartTime: "2013-16-22T05:10Z" <-- (should start by this time)
+    actualStartTime: "2013-16-22T05:30Z" <-- (20 min late relative to expected start)
+    expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time)
+    actualEndTime: null
+    expectedDuration: 15
+    actualDuration: null
+    notificationMessage: "My Job has encountered an SLA event!"
+    upstreamApps: "dependent-app-1, dependent-app-2"
+
+}
+</verbatim>
+
+---+++ Scenario 2: Workflow Action End_Miss
+*Request:*
+<verbatim>
+GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=parent_id=000056-1238791320234-oozie-joe-W
+</verbatim>
+
+*JSON Response*
+<verbatim>
+{
+
+    id : "000056-1238791320234-oozie-joe-W@map-reduce-action"
+    parentId : "000056-1238791320234-oozie-joe-W"
+    appType : "WORKFLOW_ACTION"
+    msgType : "SLA"
+    appName : "map-reduce-action"
+    slaStatus : "MISS"
+    eventStatus : "END_MISS"
+    user: "joe"
+    nominalTime: "2013-16-22T05:00Z"
+    expectedStartTime: "2013-16-22T05:10Z"
+    actualStartTime: "2013-16-22T05:05Z"
+    expectedEndTime: "2013-16-22T05:40Z" <-- (should end by this time)
+    actualEndTime: "2013-16-22T06:00Z" <-- (20 min late relative to expected end)
+    expectedDuration: 60
+    actualDuration: 55
+    notificationMessage: "My Job has encountered an SLA event!"
+    upstreamApps: "dependent-app-1, dependent-app-2"
+
+}
+</verbatim>
+
+---+++ Scenario 3: Coordinator Action Duration_Miss
+*Request:*
+<verbatim>
+GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=id=000001-1238791320234-oozie-joe-C
+</verbatim>
+
+*JSON Response*
+<verbatim>
+{
+
+    id : "000001-1238791320234-oozie-joe-C@2"
+    parentId : "000001-1238791320234-oozie-joe-C"
+    appType : "COORDINATOR_ACTION"
+    msgType : "SLA"
+    appName : "my-coord-app"
+    slaStatus : "MET"
+    eventStatus : "DURATION_MISS"
+    user: "joe"
+    nominalTime: "2013-16-22T05:00Z"
+    expectedStartTime: "2013-16-22T05:10Z"
+    actualStartTime: "2013-16-22T05:05Z"
+    expectedEndTime: "2013-16-22T05:40Z"
+    actualEndTime: "2013-16-22T05:30Z"
+    expectedDuration: 15 <-- (expected duration in minutes)
+    actualDuration: 25
+    notificationMessage: "My Job has encountered an SLA event!"
+    upstreamApps: "dependent-app-1, dependent-app-2"
+
+}
+</verbatim>
+
+Scenario #3 is particularly interesting because it is an overall "MET" because it met its expected End-time,
+but it is "Duration_Miss" because the actual run (between actual start and actual end) exceeded expected duration.
+
+---+++ Sample Email Alert
+<verbatim>
+Subject: OOZIE - SLA END_MISS (AppName=wf-sla-job, JobID=0000004-130610225200680-oozie-oozi-W)
+
+
+Status:
+  SLA Status - END_MISS
+  Job Status - RUNNING
+  Notification Message - Missed SLA for Data Pipeline job
+Job Details:
+  App Name - wf-sla-job
+  App Type - WORKFLOW_JOB
+  User - strat_ci
+  Job ID - 0000004-130610225200680-oozie-oozi-W
+  Job URL - http://host.domain.com:4080/oozie//?job=0000004-130610225200680-oozie-oozi-W
+  Parent Job ID - N/A
+  Parent Job URL - N/A
+  Upstream Apps - wf-sla-up-app
+SLA Details:
+  Nominal Time - Mon Jun 10 23:33:00 UTC 2013
+  Expected Start Time - Mon Jun 10 23:35:00 UTC 2013
+  Actual Start Time - Mon Jun 10 23:34:04 UTC 2013
+  Expected End Time - Mon Jun 10 23:38:00 UTC 2013
+  Expected Duration (in mins) - 300000
+  Actual Duration (in mins) - -1
+</verbatim>
+
+[[index][::Go back to Oozie Documentation Index::]]
+
+</noautolink>