You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Gautam <ga...@gmail.com> on 2015/06/14 23:36:52 UTC
Re: Trouble submitting hadoop job via oozie..
Sorry for the delay in responding.
I set HADOOP_CLASSPATH to point to a dir containing a valid log4j.xml
Yes, the HADOOP_USER_NAME fix worked for me.
Cheers,
-Gautam.
On Fri, May 15, 2015 at 9:47 AM, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Hi Gautam,What did you set in HADOOP_CLASSPATH?It appears you are using
> Hadoop 1.x or earlier version in non-keberized world.
> You need to set "<env-var>HADOOP_USER_NAME=${wf:user()}</env-var>" in your
> shell action.This should solve the "marred" user issue.
> Regards,Mohammad
>
>
>
> On Friday, May 15, 2015 12:49 AM, Gautam <ga...@gmail.com>
> wrote:
>
>
> Got past the log4j problem. Made a bash script that sets the
> HADOOP_CLASSPATH and runs hadoop jar from within.
>
>
> Next problem, the job is getting submitted as 'mapred' user, for some
> reason, instead of the user that submitted the oozie job. Fails when
> creating the staging dir..
>
> Exception in thread "main" java.lang.RuntimeException: java.io.IOException:
> The ownership on the staging directory
> hdfs://{namenode}:8020/user/wand/.staging is not as expected. It is owned
> by wand. The directory must be owned by the submitter mapred or by mapred
> at com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:27)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by:
> java.io.IOException: The ownership on the staging directory hdfs://
> master22.stg.kontagent.com:8020/user/wand/.staging is not as expected. It
> is owned by wand. The directory must be owned by the submitter mapred or by
> mapred at
>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:982) at
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976)
>
>
>
>
> On Thu, May 14, 2015 at 11:16 PM, Mohammad Islam
> <mislam77@yahoo.com.invalid
> > wrote:
>
> > This is sometime tricky and needs careful debugging.I think it could be
> > related to different versions of same jar. or conflicting with other jar.
> > you can try to set "ooze.launcher.mapreduce.job.classloader" in shell
> > action's configuration.
> > Please let us know how it goes. One important observation is : it is not
> > ClassNotFoundException rather NoClassDefFoundError.
> > Regards,Mohammad
> >
> >
> >
> > On Thursday, May 14, 2015 8:53 PM, Gautam <ga...@gmail.com>
> > wrote:
> >
> >
> > Hello,
> > I'm trying to submit a hadoop jar job via Oozie workflow. In
> > particular, trying to run it using the shell action by invoking the
> > /usr/bin/hadoop jar <jar_file> <MRClass>..
> >
> >
> > I tested this job separately from the command line and it works when I
> just
> > submit using "hadoop jar <jar_file> <class_name> " . But the same job
> fails
> > with the following error from within Oozie flow step with this ..
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError: Could not
> > initialize class org.apache.log4j.LogManager
> > at org.apache.log4j.Logger.getLogger(Logger.java:117)
> > at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:128)
> > at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:124)
> > at
> > com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:19)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >
> >
> >
> > I'v made sure there's a log4j.xml and the LogManager impl in my uber jar.
> > I'd imagine others have tried submitting a MR job with "hadoop jar "
> > runtime. Although, I'm also aware of the <map-reduce> action but it's too
> > cumbersome for me since i set a lot of params, counters in my job client
> > class (which implements the Tool abstract class).
> >
> >
> >
> > Here's my workflow step config:
> >
> > <action name="phShimDump">
> > <shell xmlns="uri:oozie:shell-action:0.3">
> > <job-tracker>${jobTracker}</job-tracker>
> > <name-node>${nameNode}</name-node>
> > <exec>/usr/bin/hadoop</exec>
> > <argument>jar</argument>
> > <argument>kafka-hdfs-etl/jar/kafka-hdfs-etl.jar</argument>
> >
> > <argument>com.upsight.valhalla.kafka.dumper.KafkaDumper</argument>
> >
> > <!--<argument>kafka-hdfs-etl/script/kafka-etl-console</argument>-->
> > <argument>-P</argument>
> >
> > <argument>kafka-hdfs-etl/script/oozie/${phShimDumpConfig}</argument>
> > <argument>-D</argument>
> > <argument>camus.job.name=ph_up_shim_data</argument>
> > <argument>-D</argument>
> > <argument>zookeeper.hosts=${zookeeperHosts}</argument>
> > <argument>-D</argument>
> > <argument>kafka.brokers=${kafkaBrokers}</argument>
> >
> >
> <archive>/opt/kafka-hdfs-etl/kafka-hdfs-etl.tar.gz#kafka-hdfs-etl</archive>
> > <capture-output/>
> > </shell>
> > <ok to="end" />
> > <error to="kill" />
> > </action>
> >
> >
> > thanks and regards,
> > -Gautam.
> >
> >
> >
> >
>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>
>
>
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."