You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Jarek Jarcec Cecho <ja...@apache.org> on 2012/07/04 09:19:35 UTC
Re: question on oozie workflow and log output
Hi Steve,
I'm not sure how to process log output, but oozie do have the ability to work with counters. There is EL function hadoop:counters(action name) that will return all counters for given action. You can find examples in the documentation:
http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctionalSpec.html
Those counters are available even for the sqoop action if I remember correctly.
Jarcec
On Tue, Jul 03, 2012 at 04:29:38PM -0400, Steve.Charles@us.sogeti.com wrote:
> Does anyone know how to make use of log output from one workflow action in a following action?
> Specifically, I have a sqoop action to import data into the HDFS. Following that action (if ok) I want to make use of information like rows/bytes imported in the sqoop action. I can, via the oozie console, look at the log file for the sqoop action and scrolling down, I see something like:
>
> Job complete: job_201205212157_421694
> Counters: 12
> Job Counters
> SLOTS_MILLIS_MAPS=4794
> Total time spent by all reduces waiting after reserving slots (ms)=0
> Total time spent by all maps waiting after reserving slots (ms)=0
> Launched map tasks=1
> SLOTS_MILLIS_REDUCES=0
> FileSystemCounters
> HDFS_BYTES_READ=87
> FILE_BYTES_WRITTEN=79795
> HDFS_BYTES_WRITTEN=95560
> Map-Reduce Framework
> Map input records=1000
> Spilled Records=0
> Map output records=1000
> SPLIT_RAW_BYTES=87
>
> So my thought is, to have a follow up java action, pass it the path to the log file, and have the java class extract the data I want, and update a results table.
> I also considered whether I could direct the sqoop action to output information to a properties file. Oozie does NOT like the <capture-output> tag in the sqoop action.
> Any help on this is appreciated.
> Thanks.
>
>
> Steve Charles
> Senior Consultant
>
> Sogeti USA
> 3636 N. Central Ave.
> #810
> Phoenix, AZ 85012
> www.us.sogeti.com <http://www.us.sogeti.com/>
>
RE: question on oozie workflow and log output
Posted by St...@us.sogeti.com.
That's what I thought. So then, back to looking at how to get to the logs and just scan thru the text for the count information that way. I have found I can get to the console URL for a specific workflow action. Example code below.
The console URL is what one can get to for a workflow action via the oozie console web site.
However, I need to get to the task log for that action to get to what I want.
To do so manually, I have to click on the 'map' link, then when that page is presented, the task link (I only have one), then when that page is presented, the 'All' link. Then I have the task log page I can scroll down and get the result data I wan.
I believe there are libraries that will allow for doing that within Java, but would be nice if oozie just provided a task log url off the workflow action.
OozieClient wc = new OozieClient(oozieServer);
WorkflowJob wfj;
wfj = wc.getJobInfo(jobRunInfo.getWorkflowJobId());
WorkflowAction sqoopAction = null;
List<WorkflowAction> workflowActionsList;
workflowActionsList = wfj.getActions();
for (WorkflowAction wfa : workflowActionsList) {
// System.out.println("workflow action: " + wfa.getName() + ", " + wfa.getId() + ", " + wfa.getConsoleUrl() + ", " + wfa.getTrackerUri());
if (wfa.getName().trim().equals("sqoop_task")) {
sqoopAction = wfa;
break;
}
}
// if workflow succeeded, there should be a sqoop_action
if (sqoopAction != null) {
System.out.println("sqoopAction.getConsoleUrl(): " + sqoopAction.getConsoleUrl());
// System.out.println("sqoopAction.getTrackerUri(): " + sqoopAction.getTrackerUri());
// System.out.println("sqoopAction.getData(): " + sqoopAction.getData());
System.out.println("sqoopAction.getExternalStatus(): " + sqoopAction.getExternalStatus());
}
Steve Charles
Sogeti USA
3636 N. Central Ave.
#810
Phoenix, AZ 85012
www.us.sogeti.com <http://www.us.sogeti.com/>
________________________________
From: Virag Kothari [mailto:virag@yahoo-inc.com]
Sent: Wed 7/11/2012 4:08 PM
To: oozie-users@incubator.apache.org
Cc: oozie-dev@incubator.apache.org
Subject: Re: question on oozie workflow and log output
Hi Steve,
Currently the counters/stats information is only limited to Pig and
Map-reduce actions.
(http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFuncti
onalSpec.html#a4.2.5_Hadoop_EL_Functions)
Thanks,
Virag
On 7/11/12 4:01 PM, "Steve.Charles@us.sogeti.com"
<St...@us.sogeti.com> wrote:
> I'm finally getting back to this. Trying to get counter information looked
> promising. I tried it, but my workflow job fails with exception below. I'll
> keep looking over my syntax, but does anyone know if maybe this feature is
> limited to map/reduce actions? The action I'm using is:
> <sqoop xmlns="uri:oozie:sqoop-action:0.2">
>
> Any advice on what to check or try is appreciated.
> Thanks.
>
>
> ACTION[0035106-120705224209703-oozie-oozi-W@test_counters] Exception in
> ActionStartCommand
> java.lang.IllegalArgumentException: Hadoop counters not available for action
> [sqoop_task]
> at
> org.apache.oozie.action.hadoop.HadoopELFunctions.getCounters(HadoopELFunctions
> .java:57)
> at
> org.apache.oozie.action.hadoop.HadoopELFunctions.hadoop_counters(HadoopELFunct
> ions.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> ava:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
> at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:140)
> at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl
> .java:274)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl
> .java:190)
> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:199)
> at
>
org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:94>
)
> at
> org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java
> :249)
> at
> org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java
> :47)
> at org.apache.oozie.command.Command.call(Command.java:202)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueu
> eService.java:132)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java
> :886)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908>
)
> at java.lang.Thread.run(Thread.java:662)
>
>
> Steve Charles
>
> Sogeti USA
> 3636 N. Central Ave.
> #810
> Phoenix, AZ 85012
> www.us.sogeti.com <http://www.us.sogeti.com/>
>
>
> ________________________________
>
> From: Jarek Jarcec Cecho [mailto:jarcec@apache.org]
> Sent: Wed 7/4/2012 12:19 AM
> To: oozie-users@incubator.apache.org
> Cc: oozie-dev@incubator.apache.org
> Subject: Re: question on oozie workflow and log output
>
>
>
> Hi Steve,
> I'm not sure how to process log output, but oozie do have the ability to work
> with counters. There is EL function hadoop:counters(action name) that will
> return all counters for given action. You can find examples in the
> documentation:
>
> http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctiona
> lSpec.html
>
> Those counters are available even for the sqoop action if I remember
> correctly.
>
> Jarcec
>
> On Tue, Jul 03, 2012 at 04:29:38PM -0400, Steve.Charles@us.sogeti.com wrote:
>> Does anyone know how to make use of log output from one workflow action in a
>> following action?
>> Specifically, I have a sqoop action to import data into the HDFS. Following
>> that action (if ok) I want to make use of information like rows/bytes
>> imported in the sqoop action. I can, via the oozie console, look at the log
>> file for the sqoop action and scrolling down, I see something like:
>>
>> Job complete: job_201205212157_421694
>> Counters: 12
>> Job Counters
>> SLOTS_MILLIS_MAPS=4794
>> Total time spent by all reduces waiting after reserving slots (ms)=0
>> Total time spent by all maps waiting after reserving slots (ms)=0
>> Launched map tasks=1
>> SLOTS_MILLIS_REDUCES=0
>> FileSystemCounters
>> HDFS_BYTES_READ=87
>> FILE_BYTES_WRITTEN=79795
>> HDFS_BYTES_WRITTEN=95560
>> Map-Reduce Framework
>> Map input records=1000
>> Spilled Records=0
>> Map output records=1000
>> SPLIT_RAW_BYTES=87
>>
>> So my thought is, to have a follow up java action, pass it the path to the
>> log file, and have the java class extract the data I want, and update a
>> results table.
>> I also considered whether I could direct the sqoop action to output
>> information to a properties file. Oozie does NOT like the <capture-output>
>> tag in the sqoop action.
>> Any help on this is appreciated.
>> Thanks.
>>
>>
>> Steve Charles
>> Senior Consultant
>>
>> Sogeti USA
>> 3636 N. Central Ave.
>> #810
>> Phoenix, AZ 85012
>> www.us.sogeti.com <http://www.us.sogeti.com/>
>>
>
>
Re: question on oozie workflow and log output
Posted by Virag Kothari <vi...@yahoo-inc.com>.
Hi Steve,
Currently the counters/stats information is only limited to Pig and
Map-reduce actions.
(http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFuncti
onalSpec.html#a4.2.5_Hadoop_EL_Functions)
Thanks,
Virag
On 7/11/12 4:01 PM, "Steve.Charles@us.sogeti.com"
<St...@us.sogeti.com> wrote:
> I'm finally getting back to this. Trying to get counter information looked
> promising. I tried it, but my workflow job fails with exception below. I'll
> keep looking over my syntax, but does anyone know if maybe this feature is
> limited to map/reduce actions? The action I'm using is:
> <sqoop xmlns="uri:oozie:sqoop-action:0.2">
>
> Any advice on what to check or try is appreciated.
> Thanks.
>
>
> ACTION[0035106-120705224209703-oozie-oozi-W@test_counters] Exception in
> ActionStartCommand
> java.lang.IllegalArgumentException: Hadoop counters not available for action
> [sqoop_task]
> at
> org.apache.oozie.action.hadoop.HadoopELFunctions.getCounters(HadoopELFunctions
> .java:57)
> at
> org.apache.oozie.action.hadoop.HadoopELFunctions.hadoop_counters(HadoopELFunct
> ions.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> ava:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
> at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:140)
> at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl
> .java:274)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl
> .java:190)
> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:199)
> at
>
org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:94>
)
> at
> org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java
> :249)
> at
> org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java
> :47)
> at org.apache.oozie.command.Command.call(Command.java:202)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueu
> eService.java:132)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java
> :886)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908>
)
> at java.lang.Thread.run(Thread.java:662)
>
>
> Steve Charles
>
> Sogeti USA
> 3636 N. Central Ave.
> #810
> Phoenix, AZ 85012
> www.us.sogeti.com <http://www.us.sogeti.com/>
>
>
> ________________________________
>
> From: Jarek Jarcec Cecho [mailto:jarcec@apache.org]
> Sent: Wed 7/4/2012 12:19 AM
> To: oozie-users@incubator.apache.org
> Cc: oozie-dev@incubator.apache.org
> Subject: Re: question on oozie workflow and log output
>
>
>
> Hi Steve,
> I'm not sure how to process log output, but oozie do have the ability to work
> with counters. There is EL function hadoop:counters(action name) that will
> return all counters for given action. You can find examples in the
> documentation:
>
> http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctiona
> lSpec.html
>
> Those counters are available even for the sqoop action if I remember
> correctly.
>
> Jarcec
>
> On Tue, Jul 03, 2012 at 04:29:38PM -0400, Steve.Charles@us.sogeti.com wrote:
>> Does anyone know how to make use of log output from one workflow action in a
>> following action?
>> Specifically, I have a sqoop action to import data into the HDFS. Following
>> that action (if ok) I want to make use of information like rows/bytes
>> imported in the sqoop action. I can, via the oozie console, look at the log
>> file for the sqoop action and scrolling down, I see something like:
>>
>> Job complete: job_201205212157_421694
>> Counters: 12
>> Job Counters
>> SLOTS_MILLIS_MAPS=4794
>> Total time spent by all reduces waiting after reserving slots (ms)=0
>> Total time spent by all maps waiting after reserving slots (ms)=0
>> Launched map tasks=1
>> SLOTS_MILLIS_REDUCES=0
>> FileSystemCounters
>> HDFS_BYTES_READ=87
>> FILE_BYTES_WRITTEN=79795
>> HDFS_BYTES_WRITTEN=95560
>> Map-Reduce Framework
>> Map input records=1000
>> Spilled Records=0
>> Map output records=1000
>> SPLIT_RAW_BYTES=87
>>
>> So my thought is, to have a follow up java action, pass it the path to the
>> log file, and have the java class extract the data I want, and update a
>> results table.
>> I also considered whether I could direct the sqoop action to output
>> information to a properties file. Oozie does NOT like the <capture-output>
>> tag in the sqoop action.
>> Any help on this is appreciated.
>> Thanks.
>>
>>
>> Steve Charles
>> Senior Consultant
>>
>> Sogeti USA
>> 3636 N. Central Ave.
>> #810
>> Phoenix, AZ 85012
>> www.us.sogeti.com <http://www.us.sogeti.com/>
>>
>
>
Re: question on oozie workflow and log output
Posted by Virag Kothari <vi...@yahoo-inc.com>.
Hi Steve,
Currently the counters/stats information is only limited to Pig and
Map-reduce actions.
(http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFuncti
onalSpec.html#a4.2.5_Hadoop_EL_Functions)
Thanks,
Virag
On 7/11/12 4:01 PM, "Steve.Charles@us.sogeti.com"
<St...@us.sogeti.com> wrote:
> I'm finally getting back to this. Trying to get counter information looked
> promising. I tried it, but my workflow job fails with exception below. I'll
> keep looking over my syntax, but does anyone know if maybe this feature is
> limited to map/reduce actions? The action I'm using is:
> <sqoop xmlns="uri:oozie:sqoop-action:0.2">
>
> Any advice on what to check or try is appreciated.
> Thanks.
>
>
> ACTION[0035106-120705224209703-oozie-oozi-W@test_counters] Exception in
> ActionStartCommand
> java.lang.IllegalArgumentException: Hadoop counters not available for action
> [sqoop_task]
> at
> org.apache.oozie.action.hadoop.HadoopELFunctions.getCounters(HadoopELFunctions
> .java:57)
> at
> org.apache.oozie.action.hadoop.HadoopELFunctions.hadoop_counters(HadoopELFunct
> ions.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> ava:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
> at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:140)
> at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl
> .java:274)
> at
> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl
> .java:190)
> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:199)
> at
>
org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:94>
)
> at
> org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java
> :249)
> at
> org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java
> :47)
> at org.apache.oozie.command.Command.call(Command.java:202)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueu
> eService.java:132)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java
> :886)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908>
)
> at java.lang.Thread.run(Thread.java:662)
>
>
> Steve Charles
>
> Sogeti USA
> 3636 N. Central Ave.
> #810
> Phoenix, AZ 85012
> www.us.sogeti.com <http://www.us.sogeti.com/>
>
>
> ________________________________
>
> From: Jarek Jarcec Cecho [mailto:jarcec@apache.org]
> Sent: Wed 7/4/2012 12:19 AM
> To: oozie-users@incubator.apache.org
> Cc: oozie-dev@incubator.apache.org
> Subject: Re: question on oozie workflow and log output
>
>
>
> Hi Steve,
> I'm not sure how to process log output, but oozie do have the ability to work
> with counters. There is EL function hadoop:counters(action name) that will
> return all counters for given action. You can find examples in the
> documentation:
>
> http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctiona
> lSpec.html
>
> Those counters are available even for the sqoop action if I remember
> correctly.
>
> Jarcec
>
> On Tue, Jul 03, 2012 at 04:29:38PM -0400, Steve.Charles@us.sogeti.com wrote:
>> Does anyone know how to make use of log output from one workflow action in a
>> following action?
>> Specifically, I have a sqoop action to import data into the HDFS. Following
>> that action (if ok) I want to make use of information like rows/bytes
>> imported in the sqoop action. I can, via the oozie console, look at the log
>> file for the sqoop action and scrolling down, I see something like:
>>
>> Job complete: job_201205212157_421694
>> Counters: 12
>> Job Counters
>> SLOTS_MILLIS_MAPS=4794
>> Total time spent by all reduces waiting after reserving slots (ms)=0
>> Total time spent by all maps waiting after reserving slots (ms)=0
>> Launched map tasks=1
>> SLOTS_MILLIS_REDUCES=0
>> FileSystemCounters
>> HDFS_BYTES_READ=87
>> FILE_BYTES_WRITTEN=79795
>> HDFS_BYTES_WRITTEN=95560
>> Map-Reduce Framework
>> Map input records=1000
>> Spilled Records=0
>> Map output records=1000
>> SPLIT_RAW_BYTES=87
>>
>> So my thought is, to have a follow up java action, pass it the path to the
>> log file, and have the java class extract the data I want, and update a
>> results table.
>> I also considered whether I could direct the sqoop action to output
>> information to a properties file. Oozie does NOT like the <capture-output>
>> tag in the sqoop action.
>> Any help on this is appreciated.
>> Thanks.
>>
>>
>> Steve Charles
>> Senior Consultant
>>
>> Sogeti USA
>> 3636 N. Central Ave.
>> #810
>> Phoenix, AZ 85012
>> www.us.sogeti.com <http://www.us.sogeti.com/>
>>
>
>
RE: question on oozie workflow and log output
Posted by St...@us.sogeti.com.
I'm finally getting back to this. Trying to get counter information looked promising. I tried it, but my workflow job fails with exception below. I'll keep looking over my syntax, but does anyone know if maybe this feature is limited to map/reduce actions? The action I'm using is:
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
Any advice on what to check or try is appreciated.
Thanks.
ACTION[0035106-120705224209703-oozie-oozi-W@test_counters] Exception in ActionStartCommand
java.lang.IllegalArgumentException: Hadoop counters not available for action [sqoop_task]
at org.apache.oozie.action.hadoop.HadoopELFunctions.getCounters(HadoopELFunctions.java:57)
at org.apache.oozie.action.hadoop.HadoopELFunctions.hadoop_counters(HadoopELFunctions.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172)
at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:140)
at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:199)
at org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:94)
at org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java:249)
at org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java:47)
at org.apache.oozie.command.Command.call(Command.java:202)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:132)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Steve Charles
Sogeti USA
3636 N. Central Ave.
#810
Phoenix, AZ 85012
www.us.sogeti.com <http://www.us.sogeti.com/>
________________________________
From: Jarek Jarcec Cecho [mailto:jarcec@apache.org]
Sent: Wed 7/4/2012 12:19 AM
To: oozie-users@incubator.apache.org
Cc: oozie-dev@incubator.apache.org
Subject: Re: question on oozie workflow and log output
Hi Steve,
I'm not sure how to process log output, but oozie do have the ability to work with counters. There is EL function hadoop:counters(action name) that will return all counters for given action. You can find examples in the documentation:
http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctionalSpec.html
Those counters are available even for the sqoop action if I remember correctly.
Jarcec
On Tue, Jul 03, 2012 at 04:29:38PM -0400, Steve.Charles@us.sogeti.com wrote:
> Does anyone know how to make use of log output from one workflow action in a following action?
> Specifically, I have a sqoop action to import data into the HDFS. Following that action (if ok) I want to make use of information like rows/bytes imported in the sqoop action. I can, via the oozie console, look at the log file for the sqoop action and scrolling down, I see something like:
>
> Job complete: job_201205212157_421694
> Counters: 12
> Job Counters
> SLOTS_MILLIS_MAPS=4794
> Total time spent by all reduces waiting after reserving slots (ms)=0
> Total time spent by all maps waiting after reserving slots (ms)=0
> Launched map tasks=1
> SLOTS_MILLIS_REDUCES=0
> FileSystemCounters
> HDFS_BYTES_READ=87
> FILE_BYTES_WRITTEN=79795
> HDFS_BYTES_WRITTEN=95560
> Map-Reduce Framework
> Map input records=1000
> Spilled Records=0
> Map output records=1000
> SPLIT_RAW_BYTES=87
>
> So my thought is, to have a follow up java action, pass it the path to the log file, and have the java class extract the data I want, and update a results table.
> I also considered whether I could direct the sqoop action to output information to a properties file. Oozie does NOT like the <capture-output> tag in the sqoop action.
> Any help on this is appreciated.
> Thanks.
>
>
> Steve Charles
> Senior Consultant
>
> Sogeti USA
> 3636 N. Central Ave.
> #810
> Phoenix, AZ 85012
> www.us.sogeti.com <http://www.us.sogeti.com/>
>