You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Andrew O'Neill <ao...@paytronix.com> on 2014/02/07 18:37:18 UTC
Re: coordinator job not executing as per the data event
I do not see the correct use of “done-flag” in your dataset definition. With the following definition, the coordinator would look for the file “log.txt” within the directory described in uri-template, which sounds like what you are looking for.
<datasets>
<dataset name="input" frequency="${coord:minutes(1)}" initial-instance="${start}" timezone="UTC">
<uri-template>${nameNode}/user/${coord:user()}/${examplesRoot}/input-data/rawLogs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
<done-flag>log.txt</done-flag>
</dataset>
</datasets>
Thanks,
Andrew
From: Chandramohan <mc...@gmail.com>>
Reply-To: "user@oozie.apache.org<ma...@oozie.apache.org>" <us...@oozie.apache.org>>
Date: Monday, January 27, 2014 at 9:28 AM
To: "user@oozie.apache.org<ma...@oozie.apache.org>" <us...@oozie.apache.org>>
Subject: Re: coordinator job not executing as per the data event
I added don-flag as log.txt and kept this file in each folder on HDFS path with configuration as
<coordinator-app name="coord-pig" frequency="${coord:minutes(1)}" start="${start}" end="${end}" timezone="UTC" xmlns="uri:oozie:coordinator:0.2">
<controls>
<concurrency>1</concurrency>
<execution>FIFO</execution>
<throttle>3</throttle>
</controls>
<datasets>
<dataset name="input" frequency="${coord:minutes(1)}" initial-instance="${start}" timezone="UTC">
<uri-template>${nameNode}/user/${coord:user()}/${examplesRoot}/input-data/rawLogs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}/log.txt</uri-template>
</dataset>
</datasets>
<input-events>
<data-in name="inputEvent" dataset="input">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<action>
<workflow>
<app-path>${nameNode}/user/${coord:user()}/${examplesRoot}/apps/cordpig</app-path>
<configuration>
<property>
<name>jobTracker</name>
<value>${jobTracker}</value>
</property>
<property>
<name>nameNode</name>
<value>${nameNode}</value>
</property>
<property>
<name>queueName</name>
<value>${queueName}</value>
</property>
<property>
<name>inputData</name>
<value>${coord:dataIn('inputEvent')}</value>
</property>
<property>
<name>outputData</name>
<value>${nameNode}/user/${coord:user()}/${examplesRoot}/output-data/pig</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
Job .properties as :
nameNode=hdfs://indmtx260.corp.amdocs.com:9000<http://indmtx260.corp.amdocs.com:9000>
jobTracker=indmtx260.corp.amdocs.com:9001<http://indmtx260.corp.amdocs.com:9001>
queueName=default
examplesRoot=examples
oozie.use.system.libpath=true
oozie.coord.application.path=${nameNode}/user/${user.name<http://user.name>}/${examplesRoot}/apps/cordpig/coordinator.xml
#oozie.wf.application.path=${nameNode}/user/${user.name<http://user.name>}/${examplesRoot}/apps/cordpig
start=2014-01-27T02:10Z
end=2014-02-15T13:57Z
Now oozie started the job and all the worklfow jobs are in waiting state, they stay in waiting. I an not sure why this data-event is not working for me.
Please find attached snapshot from oozie portal and HDFS.
-Chandramohan
On Mon, Jan 27, 2014 at 7:24 PM, Chandramohan <mc...@gmail.com>> wrote:
Yes I do have the directory available on HDFS with path hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07>.
There are no _SUCCESS files in any of the folder. Do we need to have a _SUCCESS file, is there any way not to have it.
On Thu, Jan 16, 2014 at 4:40 PM, Chandramohan <mc...@gmail.com>> wrote:
Hi All,
I have sample coordinator application where log files needs to be moved on HDFS from one location to another every minute.
Once the coordination application started, it keeps running while workflow jobs stays in WAITING state forever.
pig script and only workflow job is tested separately and it's working fine.
Please find attached application and inlined log and help me out.
Log file is as below
2014-01-16 16:37:10,632 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: Action is in WAITING state.
2014-01-16 16:37:10,634 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::CoordActionInputCheck:: Missing deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS>
2014-01-16 16:37:10,636 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In checkResolvedUris...
2014-01-16 16:37:10,636 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In checkListOfPaths: hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS> is Missing.
2014-01-16 16:37:10,663 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS>, Exists? :false
2014-01-16 16:37:10,884 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: Action is in WAITING state.
2014-01-16 16:37:10,886 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::CoordActionInputCheck:: Missing deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS>
2014-01-16 16:37:10,888 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In checkResolvedUris...
2014-01-16 16:37:10,888 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In checkListOfPaths: hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS> is Missing.
2014-01-16 16:37:10,916 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS>, Exists? :false
2014-01-16 16:37:10,943 INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:37:10,962 INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:37:50,061 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2014-01-16 16:37:50,061 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time = 2014-01-16T11:06Z
2014-01-16 16:37:50,079 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set coordinator job [0000357-140113142545858-oozie-biad-C] status to RUNNINGWITHERROR' from 'RUNNING'
2014-01-16 16:37:50,079 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Coord job [0000357-140113142545858-oozie-biad-C] Pending set to FALSE
2014-01-16 16:37:50,083 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time = 2014-01-16T11:06Z
2014-01-16 16:37:50,084 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-01-16 16:38:10,713 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: Action is in WAITING state.
2014-01-16 16:38:10,716 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::CoordActionInputCheck:: Missing deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS>
2014-01-16 16:38:10,718 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In checkResolvedUris...
2014-01-16 16:38:10,718 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: In checkListOfPaths: hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS> is Missing.
2014-01-16 16:38:10,745 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@3] [0000357-140113142545858-oozie-biad-C@3]::ActionInputCheck:: File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/09/_SUCCESS>, Exists? :false
2014-01-16 16:38:10,963 INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:38:10,972 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: Action is in WAITING state.
2014-01-16 16:38:10,975 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::CoordActionInputCheck:: Missing deps:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS>
2014-01-16 16:38:10,977 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In checkResolvedUris...
2014-01-16 16:38:10,977 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: In checkListOfPaths: hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS> is Missing.
2014-01-16 16:38:10,983 INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2014-01-16 16:38:11,001 INFO CoordActionInputCheckXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] [0000357-140113142545858-oozie-biad-C@2]::ActionInputCheck:: File:hdfs://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS<http://indmtx260.corp.company.com:9000/user/biadmin/examples/input-data/rawLogs/2014/01/16/09/07/_SUCCESS>, Exists? :false
2014-01-16 16:38:11,276 INFO CoordActionNotificationXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] STARTED Coordinator Notification actionId=0000357-140113142545858-oozie-biad-C@2 : TIMEDOUT
2014-01-16 16:38:11,278 INFO CoordActionNotificationXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] No Notification URL is defined. Therefore nothing to notify for job 0000357-140113142545858-oozie-biad-C action ID 0000357-140113142545858-oozie-biad-C@2
2014-01-16 16:38:11,279 INFO CoordActionNotificationXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000357-140113142545858-oozie-biad-C] ACTION[0000357-140113142545858-oozie-biad-C@2] ENDED Coordinator Notification actionId=0000357-140113142545858-oozie-biad-C@2
2014-01-16 16:38:50,085 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2014-01-16 16:38:50,085 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time = 2014-01-16T11:07Z
2014-01-16 16:38:50,109 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set coordinator job [0000357-140113142545858-oozie-biad-C] status to RUNNINGWITHERROR' from 'RUNNING'
2014-01-16 16:38:50,109 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Coord job [0000357-140113142545858-oozie-biad-C] Pending set to FALSE
2014-01-16 16:38:50,111 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time = 2014-01-16T11:07Z
2014-01-16 16:38:50,112 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
Thanks and Regards,
Chandramohan