You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Robert Kanter (JIRA)" <ji...@apache.org> on 2014/11/12 19:48:34 UTC
[jira] [Reopened] (OOZIE-2018) Coordinator materialization problems
with cron syntax
[ https://issues.apache.org/jira/browse/OOZIE-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter reopened OOZIE-2018:
----------------------------------
> Coordinator materialization problems with cron syntax
> -----------------------------------------------------
>
> Key: OOZIE-2018
> URL: https://issues.apache.org/jira/browse/OOZIE-2018
> Project: Oozie
> Issue Type: Bug
> Components: coordinator
> Affects Versions: trunk, 4.0.0
> Reporter: Robert Kanter
>
> Suppose you submit the following coordinator job:
> {code:xml}
> <coordinator-app name="DailySleep"
> frequency="*/2 * * * *"
> start="2013-06-01T00:00Z" end="2013-06-05T00:00Z" timezone="America/Los_Angeles"
> xmlns="uri:oozie:coordinator:0.2"
> >
> <controls>
> <timeout>-1</timeout>
> <concurrency>1</concurrency>
> <execution>FIFO</execution>
> <throttle>2</throttle>
> </controls>
> <datasets>
> <dataset name="sleep_time" frequency="${coord:days(1)}"
> initial-instance="2012-05-31T00:00Z" timezone="America/Los_Angeles">
> <uri-template>${DAY}</uri-template>
> <done-flag></done-flag>
> </dataset>
> </datasets>
> <action>
> <workflow>
> <app-path>${wf_application_path}</app-path>
> <configuration>
> <property>
> <name>REDUCER_SLEEP_TIME</name>
> <value>120000</value>
> </property>
> <property>
> <name>oozie.use.system.libpath</name>
> <value>true</value>
> </property>
> </configuration>
> </workflow>
> </action>
> </coordinator-app>
> {code}
> Where {{$\{wf_application_path}}} points to a workflow that simply runs a sleep MR job for 2 mins.
> Notice that the above coordinator job is set to run with a frequency of {{*/2 * * * *}}, which means every 2 minutes, and the throttle is 2.
> When you run this job, you’ll see a few anomalies:
> # Other than the first action, each action is materialized twice. The action numbering works fine, but you’ll see two actions for each Nominal Time. You can see this in the job info below.
> # You can’t see this in the job info below, but while it’s running, there are actually 3 jobs READY at the same time, when there should be only 2 (because throttle was set to 2)
> # OOZIE-1680 added an oozie-site config property {{oozie.service.coord.check.maximum.frequency=true}} which is supposed to block jobs with frequencies faster than 5 minutes; it didn’t stop this coordinator
> Points 1 and 2 above are likely the same problem. Point 3 is somewhat trivial.
> Here’s the job info (I killed the job before it finished, and I cut out non-relevent info to make it easier to read):
> {noformat}
> ---------------------------------------------------------------------------------------------------------------------------------------
> ID External ID Created Nominal Time
> ---------------------------------------------------------------------------------------------------------------------------------------
> 0000005-140922161548481-oozie-oozi-C@1 0000006-140922161548481-oozie-oozi-W 2014-09-22 23:34:38 GMT 2013-06-01 00:00:00 GMT
> ---------------------------------------------------------------------------------------------------------------------------------------
> 0000005-140922161548481-oozie-oozi-C@2 0000007-140922161548481-oozie-oozi-W 2014-09-22 23:34:38 GMT 2013-06-01 00:02:00 GMT
> ---------------------------------------------------------------------------------------------------------------------------------------
> 0000005-140922161548481-oozie-oozi-C@3 0000008-140922161548481-oozie-oozi-W 2014-09-22 23:36:11 GMT 2013-06-01 00:02:00 GMT
> ---------------------------------------------------------------------------------------------------------------------------------------
> 0000005-140922161548481-oozie-oozi-C@4 0000009-140922161548481-oozie-oozi-W 2014-09-22 23:36:11 GMT 2013-06-01 00:04:00 GMT
> ---------------------------------------------------------------------------------------------------------------------------------------
> 0000005-140922161548481-oozie-oozi-C@5 0000005-140922161548481-oozie-oozi-C 2014-09-22 23:41:11 GMT 2013-06-01 00:04:00 GMT
> ---------------------------------------------------------------------------------------------------------------------------------------
> 0000005-140922161548481-oozie-oozi-C@6 0000005-140922161548481-oozie-oozi-C 2014-09-22 23:41:11 GMT 2013-06-01 00:06:00 GMT
> ---------------------------------------------------------------------------------------------------------------------------------------
> {noformat}
> I tried the same coordinator job, but used the old frequency syntax ({{$\{coord:minutes(2)}}}, and even though we don’t recommend a 2 min frequency, it actually worked correctly (once I set {{oozie.service.coord.check.maximum.frequency=false}} of course). So this appears to be a problem with the cron syntax. If ({{$\{coord:minutes(2)}}} didn’t work either, then I’d say it’s just once of the quirks of too high a frequency, but that’s not the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)