You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Idris Ali <ps...@gmail.com> on 2012/12/05 11:18:05 UTC
oozie action jars are copied to FS dervied from workflow app-path
Hi.
Env oozie-3.2.2
I am running oozie's aggregator coordinator example to test one of my
framework on EMR.
The action element in workflow.xml has a name-node say
hdfs://namenode-A:8020/
I have setup oozie aggregator app by copying the example to a different
namenode say hdfs://namenode-B:8020/
Observations:
1. Since my app-path and data resides on namenode-B, oozie creates the
launcher staging directories in namenode-B.
2. JavaActionExecutor removes the scheme and authority while setting the
distributed cache (addToCache)
3. Hadoop (namenode-A) assumes and tries to find the lib files in its
filesystem and throws this exception:
Map/Reduce failed, error message[JA008: File does not exist:
/user/oozie/oozie-oozi/0000043-121204151259971-oozie-oozi-W/aggregator--map-reduce/map-reduce-launcher.jar]
Ideally I would want oozie to copy the launcher and users lib jars to
nameNode defined in action tag for creating staging dirs.
<action name="aggregator">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>hdfs://namenode-A:8020</name-node>
<prepare>
<delete path="${outputData}"/>
</prepare>
<configuration>
.......
</action>
Thanks,
-Idris
More details fro job.properties
nameNode=hdfs://10.190.199.24:9000
jobTracker=10.190.199.24:9001
queueName=default
examplesRoot=examples1
#storage has input/output and aggregator app.
storage=s3n://access:secret@inmobi-grid-emr-dev
oozie.coord.application.path=${nameNode}/${examplesRoot}/apps/aggregator
start=2010-01-01T01:00Z
end=2010-01-01T03:00Z
Re: oozie action jars are copied to FS dervied from workflow app-path
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Idris,
[moving to user@o.a.o, bcc-ing dev@o.a.o]
The Oozie app (coord and wf XMLs, files and JARs) must be in the same NN
configured in the JT.
Thx
On Wed, Dec 5, 2012 at 2:18 AM, Idris Ali <ps...@gmail.com> wrote:
> Hi.
>
> Env oozie-3.2.2
>
> I am running oozie's aggregator coordinator example to test one of my
> framework on EMR.
>
> The action element in workflow.xml has a name-node say
> hdfs://namenode-A:8020/
> I have setup oozie aggregator app by copying the example to a different
> namenode say hdfs://namenode-B:8020/
>
> Observations:
> 1. Since my app-path and data resides on namenode-B, oozie creates the
> launcher staging directories in namenode-B.
> 2. JavaActionExecutor removes the scheme and authority while setting the
> distributed cache (addToCache)
> 3. Hadoop (namenode-A) assumes and tries to find the lib files in its
> filesystem and throws this exception:
> Map/Reduce failed, error message[JA008: File does not exist:
>
> /user/oozie/oozie-oozi/0000043-121204151259971-oozie-oozi-W/aggregator--map-reduce/map-reduce-launcher.jar]
>
> Ideally I would want oozie to copy the launcher and users lib jars to
> nameNode defined in action tag for creating staging dirs.
> <action name="aggregator">
> <map-reduce>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>hdfs://namenode-A:8020</name-node>
> <prepare>
> <delete path="${outputData}"/>
> </prepare>
> <configuration>
> .......
> </action>
>
> Thanks,
> -Idris
>
> More details fro job.properties
> nameNode=hdfs://10.190.199.24:9000
> jobTracker=10.190.199.24:9001
> queueName=default
> examplesRoot=examples1
> #storage has input/output and aggregator app.
> storage=s3n://access:secret@inmobi-grid-emr-dev
> oozie.coord.application.path=${nameNode}/${examplesRoot}/apps/aggregator
> start=2010-01-01T01:00Z
> end=2010-01-01T03:00Z
>
--
Alejandro
Re: oozie action jars are copied to FS dervied from workflow app-path
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Idris,
[moving to user@o.a.o, bcc-ing dev@o.a.o]
The Oozie app (coord and wf XMLs, files and JARs) must be in the same NN
configured in the JT.
Thx
On Wed, Dec 5, 2012 at 2:18 AM, Idris Ali <ps...@gmail.com> wrote:
> Hi.
>
> Env oozie-3.2.2
>
> I am running oozie's aggregator coordinator example to test one of my
> framework on EMR.
>
> The action element in workflow.xml has a name-node say
> hdfs://namenode-A:8020/
> I have setup oozie aggregator app by copying the example to a different
> namenode say hdfs://namenode-B:8020/
>
> Observations:
> 1. Since my app-path and data resides on namenode-B, oozie creates the
> launcher staging directories in namenode-B.
> 2. JavaActionExecutor removes the scheme and authority while setting the
> distributed cache (addToCache)
> 3. Hadoop (namenode-A) assumes and tries to find the lib files in its
> filesystem and throws this exception:
> Map/Reduce failed, error message[JA008: File does not exist:
>
> /user/oozie/oozie-oozi/0000043-121204151259971-oozie-oozi-W/aggregator--map-reduce/map-reduce-launcher.jar]
>
> Ideally I would want oozie to copy the launcher and users lib jars to
> nameNode defined in action tag for creating staging dirs.
> <action name="aggregator">
> <map-reduce>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>hdfs://namenode-A:8020</name-node>
> <prepare>
> <delete path="${outputData}"/>
> </prepare>
> <configuration>
> .......
> </action>
>
> Thanks,
> -Idris
>
> More details fro job.properties
> nameNode=hdfs://10.190.199.24:9000
> jobTracker=10.190.199.24:9001
> queueName=default
> examplesRoot=examples1
> #storage has input/output and aggregator app.
> storage=s3n://access:secret@inmobi-grid-emr-dev
> oozie.coord.application.path=${nameNode}/${examplesRoot}/apps/aggregator
> start=2010-01-01T01:00Z
> end=2010-01-01T03:00Z
>
--
Alejandro