You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Matteo Luzzi <ma...@gmail.com> on 2015/07/21 13:33:37 UTC

java.lang.NoSuchMethodError while running a spark job

Hi again!

As I previously wrote in here (
http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E)
I'm running an oozie distro 4.2.0 built against hadoop 2.7.0.

I'm trying to execute a fat spark jar that I developed using oozie spark
action. The jar itself is perfectly running using a local/standalone/yarn
master mode via normal spark-submit.

My workflow is pretty simple

<workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkBatch'>
<start to='spark-node' />

<action name='spark-node'>
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>${master}</master>
            <name>sparkbatch</name>
          <class>com.sparkBatch.storageArchitecture.App</class>

<jar>${nameNode}/user/${wf:user()}/${batchAppFolder}/lib/${jarName}</jar>
           <!-- opts when using standalone/yarn -->
            <spark-opts>--num-executors 1  --driver-memory 2g
--executor-memory 3g --executor-cores 4 --queue default</spark-opts>
</spark>
<ok to="end"/>
        <error to="fail"/>
</action>
 <kill name="fail">
        <message>Spark Job failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

I can submit the workflow on oozie server but the job status is stuck on
RUNNING, no errors in the oozie job log.

When I open the application console on hadoop I have this error:

Application application_1437460801014_0011 failed 2 times due to AM
Container for appattempt_1437460801014_0011_000002 exited with exitCode: 1
For more detailed output, check application tracking page:
http://Matteos-MBP.local:8088/cluster/app/application_1437460801014_0011Then,
click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1437460801014_0011_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.


and checking the yarn log via yarn logs -applicationId appId I have the
following:

2015-07-20 22:51:36,794 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
application appattempt_1437424593766_0001_000002
2015-07-20 22:51:36,938 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError:
org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364)
2015-07-20 22:51:36,940 INFO [Thread-1]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a
signal. Signaling RMCommunicator and JobHistoryEventHandler.

>From the last error it seems that some component is invoking a method which
has been removed since hadoop 2.3.0 and it is not found on the current
version that I'm using (2.7.0)

P.S I can correctly execute the oozie spark example which works with a
local master

I think it's still a problem of configuration/version between oozie and
hadoop but after two days I can't figure out what is the problem. Did
somebody already face this weird error?


-- 
Matteo Remo Luzzi

RE: java.lang.NoSuchMethodError while running a spark job

Posted by Oussama Chougna <ou...@hotmail.com>.
To avoid dependency problems, make sure the dependency versions in your java project match with the versions on your hadoop cluster and also scope them as "provided".
And there should be a log file for the job that executes the oozie action, please check your job history server for jobs with a name like: oozie:launcher:T=shell:W=Some workflow:A=some-action:ID=0000019-150721095438256-oozie-oozi-W

Cheers,

Oussama Chougna



> From: matteo.luzzi@gmail.com
> Date: Tue, 21 Jul 2015 15:50:12 +0200
> Subject: Re: java.lang.NoSuchMethodError while running a spark job
> To: user@oozie.apache.org
> 
> Hi, thanks for you reply
> 
> I noticed that I can check the classpath only on logs of successful jobs
> (oozie spark examples). In the application folders that refer to my custom
> workflow/spark jar I have only the few lines about the NoSuchMethodError.
> Anyway in the logs with the class path I found that the version of
> haddop-common is 2.7.0 according to the hadoop version used.
> 
> Checking the maven dependency tree I found that the spark core artifact I
> found that the version of spark I'm using (1.4.0) has a nested dependency
> on hadoop-client:2.2.0 which has a dependency on
> hadoop-mapreduce-client:2.2.0 which has actually the method
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main invoking the missing
> org.apache.hadoop.http.HttpConfig.setPolicy.
> 
> I tried to exclude that dependency and submit the oozie job again and now I
> get a different error when I check the yarn log:
> 
> INFO [main] org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED;
> cause: java.lang.UnsupportedOperationException: Not implemented by the TFS
> FileSystem implementation
> java.lang.UnsupportedOperationException: Not implemented by the TFS
> FileSystem implementation
> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217)
> at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1496)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1493)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1426)
> 
> I'm afraid that, as you said, I might be using a incompatible version of
> hadoop or oozie or spark. Everytime I fix an error I just get a new one and
> I believe that things should have been smother.
> 
> 2015-07-21 14:35 GMT+02:00 Oussama Chougna <ou...@hotmail.com>:
> 
> > Hello,
> > Check which version of hadoop-common is on your classpath.You can check
> > this in the logs of the oozie action job. Each oozie action is executed via
> > a Mapper and thus submitted as a Hadoop job.
> > The log ouput looks like:
> > Files in current
> > dir:/yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002/.
> > ======================
> > File: .launch_container.sh.crc
> > File: launch_container.sh
> > File: .action.xml.crc
> > File: .container_tokens.crc
> > File: oozie-hadoop-utils-2.6.0-cdh5.4.2.oozie-4.1.0-cdh5.4.2.jar
> > File: .default_container_executor_session.sh.crc
> > File: oozie-sharelib-oozie-4.1.0-cdh5.4.2.jar
> > File: action.xml
> > File: job.xml
> > File: default_container_executor_session.sh
> > File: json-simple-1.1.jar
> > File: container_tokens
> > File: default_container_executor.sh
> > Dir: tmp
> > File: ftp.sh
> > File: .default_container_executor.sh.crc
> > File: .job.xml.crc
> >
> > Oozie Java/Map-Reduce/Pig action launcher-job configuration
> > =================================================================
> > Workflow job id   : 0000015-150721095438256-oozie-oozi-W
> > Workflow action id: 0000015-150721095438256-oozie-oozi-W@foobar
> >
> > Classpath         :
> > ------------------------
> >
> > /yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002
> >   /etc/hadoop/conf.cloudera.yarn
> >   /var/run/cloudera-scm-agent/process/293-yarn-NODEMANAGER
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-protobuf.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-common.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format-javadoc.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-jackson.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs-2.6.0-cdh5.4.2.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common.jar
> >   /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-annotations.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-hadoop.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-pig-bundle.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-scala_2.10.jar
> >
> > /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2-tests.jar.......................
> > I guess you are using an incompatible version with Yarn.
> >
> >
> >
> > > From: matteo.luzzi@gmail.com
> > > Date: Tue, 21 Jul 2015 13:33:37 +0200
> > > Subject: java.lang.NoSuchMethodError while running a spark job
> > > To: user@oozie.apache.org
> > >
> > > Hi again!
> > >
> > > As I previously wrote in here (
> > >
> > http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E
> > )
> > > I'm running an oozie distro 4.2.0 built against hadoop 2.7.0.
> > >
> > > I'm trying to execute a fat spark jar that I developed using oozie spark
> > > action. The jar itself is perfectly running using a local/standalone/yarn
> > > master mode via normal spark-submit.
> > >
> > > My workflow is pretty simple
> > >
> > > <workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkBatch'>
> > > <start to='spark-node' />
> > >
> > > <action name='spark-node'>
> > > <spark xmlns="uri:oozie:spark-action:0.1">
> > > <job-tracker>${jobTracker}</job-tracker>
> > >             <name-node>${nameNode}</name-node>
> > >             <master>${master}</master>
> > >             <name>sparkbatch</name>
> > >           <class>com.sparkBatch.storageArchitecture.App</class>
> > >
> > > <jar>${nameNode}/user/${wf:user()}/${batchAppFolder}/lib/${jarName}</jar>
> > >            <!-- opts when using standalone/yarn -->
> > >             <spark-opts>--num-executors 1  --driver-memory 2g
> > > --executor-memory 3g --executor-cores 4 --queue default</spark-opts>
> > > </spark>
> > > <ok to="end"/>
> > >         <error to="fail"/>
> > > </action>
> > >  <kill name="fail">
> > >         <message>Spark Job failed, error
> > > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > >     </kill>
> > >     <end name="end"/>
> > > </workflow-app>
> > >
> > > I can submit the workflow on oozie server but the job status is stuck on
> > > RUNNING, no errors in the oozie job log.
> > >
> > > When I open the application console on hadoop I have this error:
> > >
> > > Application application_1437460801014_0011 failed 2 times due to AM
> > > Container for appattempt_1437460801014_0011_000002 exited with exitCode:
> > 1
> > > For more detailed output, check application tracking page:
> > >
> > http://Matteos-MBP.local:8088/cluster/app/application_1437460801014_0011Then
> > ,
> > > click on links to logs of each attempt.
> > > Diagnostics: Exception from container-launch.
> > > Container id: container_1437460801014_0011_02_000001
> > > Exit code: 1
> > > Stack trace: ExitCodeException exitCode=1:
> > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
> > > at org.apache.hadoop.util.Shell.run(Shell.java:456)
> > > at
> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
> > > at
> > >
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> > > at
> > >
> > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> > > at
> > >
> > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > > at java.lang.Thread.run(Thread.java:745)
> > > Container exited with a non-zero exit code 1
> > > Failing this attempt. Failing the application.
> > >
> > >
> > > and checking the yarn log via yarn logs -applicationId appId I have the
> > > following:
> > >
> > > 2015-07-20 22:51:36,794 INFO [main]
> > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> > > application appattempt_1437424593766_0001_000002
> > > 2015-07-20 22:51:36,938 ERROR [main]
> > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting
> > MRAppMaster
> > > java.lang.NoSuchMethodError:
> > >
> > org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V
> > > at
> > >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364)
> > > 2015-07-20 22:51:36,940 INFO [Thread-1]
> > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a
> > > signal. Signaling RMCommunicator and JobHistoryEventHandler.
> > >
> > > From the last error it seems that some component is invoking a method
> > which
> > > has been removed since hadoop 2.3.0 and it is not found on the current
> > > version that I'm using (2.7.0)
> > >
> > > P.S I can correctly execute the oozie spark example which works with a
> > > local master
> > >
> > > I think it's still a problem of configuration/version between oozie and
> > > hadoop but after two days I can't figure out what is the problem. Did
> > > somebody already face this weird error?
> > >
> > >
> > > --
> > > Matteo Remo Luzzi
> >
> >
> 
> 
> 
> -- 
> Matteo Remo Luzzi

Re: java.lang.NoSuchMethodError while running a spark job

Posted by Matteo Luzzi <ma...@gmail.com>.
Hi, thanks for you reply

I noticed that I can check the classpath only on logs of successful jobs
(oozie spark examples). In the application folders that refer to my custom
workflow/spark jar I have only the few lines about the NoSuchMethodError.
Anyway in the logs with the class path I found that the version of
haddop-common is 2.7.0 according to the hadoop version used.

Checking the maven dependency tree I found that the spark core artifact I
found that the version of spark I'm using (1.4.0) has a nested dependency
on hadoop-client:2.2.0 which has a dependency on
hadoop-mapreduce-client:2.2.0 which has actually the method
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main invoking the missing
org.apache.hadoop.http.HttpConfig.setPolicy.

I tried to exclude that dependency and submit the oozie job again and now I
get a different error when I check the yarn log:

INFO [main] org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED;
cause: java.lang.UnsupportedOperationException: Not implemented by the TFS
FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS
FileSystem implementation
at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1496)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1493)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1426)

I'm afraid that, as you said, I might be using a incompatible version of
hadoop or oozie or spark. Everytime I fix an error I just get a new one and
I believe that things should have been smother.

2015-07-21 14:35 GMT+02:00 Oussama Chougna <ou...@hotmail.com>:

> Hello,
> Check which version of hadoop-common is on your classpath.You can check
> this in the logs of the oozie action job. Each oozie action is executed via
> a Mapper and thus submitted as a Hadoop job.
> The log ouput looks like:
> Files in current
> dir:/yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002/.
> ======================
> File: .launch_container.sh.crc
> File: launch_container.sh
> File: .action.xml.crc
> File: .container_tokens.crc
> File: oozie-hadoop-utils-2.6.0-cdh5.4.2.oozie-4.1.0-cdh5.4.2.jar
> File: .default_container_executor_session.sh.crc
> File: oozie-sharelib-oozie-4.1.0-cdh5.4.2.jar
> File: action.xml
> File: job.xml
> File: default_container_executor_session.sh
> File: json-simple-1.1.jar
> File: container_tokens
> File: default_container_executor.sh
> Dir: tmp
> File: ftp.sh
> File: .default_container_executor.sh.crc
> File: .job.xml.crc
>
> Oozie Java/Map-Reduce/Pig action launcher-job configuration
> =================================================================
> Workflow job id   : 0000015-150721095438256-oozie-oozi-W
> Workflow action id: 0000015-150721095438256-oozie-oozi-W@foobar
>
> Classpath         :
> ------------------------
>
> /yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002
>   /etc/hadoop/conf.cloudera.yarn
>   /var/run/cloudera-scm-agent/process/293-yarn-NODEMANAGER
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-protobuf.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-common.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format-javadoc.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-jackson.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs-2.6.0-cdh5.4.2.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common.jar
>   /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-annotations.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-hadoop.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-pig-bundle.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-scala_2.10.jar
>
> /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2-tests.jar.......................
> I guess you are using an incompatible version with Yarn.
>
>
>
> > From: matteo.luzzi@gmail.com
> > Date: Tue, 21 Jul 2015 13:33:37 +0200
> > Subject: java.lang.NoSuchMethodError while running a spark job
> > To: user@oozie.apache.org
> >
> > Hi again!
> >
> > As I previously wrote in here (
> >
> http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E
> )
> > I'm running an oozie distro 4.2.0 built against hadoop 2.7.0.
> >
> > I'm trying to execute a fat spark jar that I developed using oozie spark
> > action. The jar itself is perfectly running using a local/standalone/yarn
> > master mode via normal spark-submit.
> >
> > My workflow is pretty simple
> >
> > <workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkBatch'>
> > <start to='spark-node' />
> >
> > <action name='spark-node'>
> > <spark xmlns="uri:oozie:spark-action:0.1">
> > <job-tracker>${jobTracker}</job-tracker>
> >             <name-node>${nameNode}</name-node>
> >             <master>${master}</master>
> >             <name>sparkbatch</name>
> >           <class>com.sparkBatch.storageArchitecture.App</class>
> >
> > <jar>${nameNode}/user/${wf:user()}/${batchAppFolder}/lib/${jarName}</jar>
> >            <!-- opts when using standalone/yarn -->
> >             <spark-opts>--num-executors 1  --driver-memory 2g
> > --executor-memory 3g --executor-cores 4 --queue default</spark-opts>
> > </spark>
> > <ok to="end"/>
> >         <error to="fail"/>
> > </action>
> >  <kill name="fail">
> >         <message>Spark Job failed, error
> > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> >     </kill>
> >     <end name="end"/>
> > </workflow-app>
> >
> > I can submit the workflow on oozie server but the job status is stuck on
> > RUNNING, no errors in the oozie job log.
> >
> > When I open the application console on hadoop I have this error:
> >
> > Application application_1437460801014_0011 failed 2 times due to AM
> > Container for appattempt_1437460801014_0011_000002 exited with exitCode:
> 1
> > For more detailed output, check application tracking page:
> >
> http://Matteos-MBP.local:8088/cluster/app/application_1437460801014_0011Then
> ,
> > click on links to logs of each attempt.
> > Diagnostics: Exception from container-launch.
> > Container id: container_1437460801014_0011_02_000001
> > Exit code: 1
> > Stack trace: ExitCodeException exitCode=1:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
> > at org.apache.hadoop.util.Shell.run(Shell.java:456)
> > at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Container exited with a non-zero exit code 1
> > Failing this attempt. Failing the application.
> >
> >
> > and checking the yarn log via yarn logs -applicationId appId I have the
> > following:
> >
> > 2015-07-20 22:51:36,794 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> > application appattempt_1437424593766_0001_000002
> > 2015-07-20 22:51:36,938 ERROR [main]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting
> MRAppMaster
> > java.lang.NoSuchMethodError:
> >
> org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V
> > at
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364)
> > 2015-07-20 22:51:36,940 INFO [Thread-1]
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a
> > signal. Signaling RMCommunicator and JobHistoryEventHandler.
> >
> > From the last error it seems that some component is invoking a method
> which
> > has been removed since hadoop 2.3.0 and it is not found on the current
> > version that I'm using (2.7.0)
> >
> > P.S I can correctly execute the oozie spark example which works with a
> > local master
> >
> > I think it's still a problem of configuration/version between oozie and
> > hadoop but after two days I can't figure out what is the problem. Did
> > somebody already face this weird error?
> >
> >
> > --
> > Matteo Remo Luzzi
>
>



-- 
Matteo Remo Luzzi

RE: java.lang.NoSuchMethodError while running a spark job

Posted by Oussama Chougna <ou...@hotmail.com>.
Hello,
Check which version of hadoop-common is on your classpath.You can check this in the logs of the oozie action job. Each oozie action is executed via a Mapper and thus submitted as a Hadoop job.
The log ouput looks like:
Files in current dir:/yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002/.
======================
File: .launch_container.sh.crc
File: launch_container.sh
File: .action.xml.crc
File: .container_tokens.crc
File: oozie-hadoop-utils-2.6.0-cdh5.4.2.oozie-4.1.0-cdh5.4.2.jar
File: .default_container_executor_session.sh.crc
File: oozie-sharelib-oozie-4.1.0-cdh5.4.2.jar
File: action.xml
File: job.xml
File: default_container_executor_session.sh
File: json-simple-1.1.jar
File: container_tokens
File: default_container_executor.sh
Dir: tmp
File: ftp.sh
File: .default_container_executor.sh.crc
File: .job.xml.crc

Oozie Java/Map-Reduce/Pig action launcher-job configuration
=================================================================
Workflow job id   : 0000015-150721095438256-oozie-oozi-W
Workflow action id: 0000015-150721095438256-oozie-oozi-W@foobar

Classpath         :
------------------------
  /yarn/nm/usercache/chougnaoa/appcache/application_1437465272054_0011/container_1437465272054_0011_01_000002
  /etc/hadoop/conf.cloudera.yarn
  /var/run/cloudera-scm-agent/process/293-yarn-NODEMANAGER
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-protobuf.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-common.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format-javadoc.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-jackson.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-format.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs-2.6.0-cdh5.4.2.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-nfs.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-annotations.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-hadoop.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-pig-bundle.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/parquet-scala_2.10.jar
  /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop/hadoop-common-2.6.0-cdh5.4.2-tests.jar.......................
I guess you are using an incompatible version with Yarn.



> From: matteo.luzzi@gmail.com
> Date: Tue, 21 Jul 2015 13:33:37 +0200
> Subject: java.lang.NoSuchMethodError while running a spark job
> To: user@oozie.apache.org
> 
> Hi again!
> 
> As I previously wrote in here (
> http://mail-archives.apache.org/mod_mbox/oozie-user/201507.mbox/%3CCALBGZ8o4n27S8w6fn3HFxfzJmZbA9Gsz71Ewg%2Br6XEFCZTFpPQ%40mail.gmail.com%3E)
> I'm running an oozie distro 4.2.0 built against hadoop 2.7.0.
> 
> I'm trying to execute a fat spark jar that I developed using oozie spark
> action. The jar itself is perfectly running using a local/standalone/yarn
> master mode via normal spark-submit.
> 
> My workflow is pretty simple
> 
> <workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkBatch'>
> <start to='spark-node' />
> 
> <action name='spark-node'>
> <spark xmlns="uri:oozie:spark-action:0.1">
> <job-tracker>${jobTracker}</job-tracker>
>             <name-node>${nameNode}</name-node>
>             <master>${master}</master>
>             <name>sparkbatch</name>
>           <class>com.sparkBatch.storageArchitecture.App</class>
> 
> <jar>${nameNode}/user/${wf:user()}/${batchAppFolder}/lib/${jarName}</jar>
>            <!-- opts when using standalone/yarn -->
>             <spark-opts>--num-executors 1  --driver-memory 2g
> --executor-memory 3g --executor-cores 4 --queue default</spark-opts>
> </spark>
> <ok to="end"/>
>         <error to="fail"/>
> </action>
>  <kill name="fail">
>         <message>Spark Job failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>     </kill>
>     <end name="end"/>
> </workflow-app>
> 
> I can submit the workflow on oozie server but the job status is stuck on
> RUNNING, no errors in the oozie job log.
> 
> When I open the application console on hadoop I have this error:
> 
> Application application_1437460801014_0011 failed 2 times due to AM
> Container for appattempt_1437460801014_0011_000002 exited with exitCode: 1
> For more detailed output, check application tracking page:
> http://Matteos-MBP.local:8088/cluster/app/application_1437460801014_0011Then,
> click on links to logs of each attempt.
> Diagnostics: Exception from container-launch.
> Container id: container_1437460801014_0011_02_000001
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
> at org.apache.hadoop.util.Shell.run(Shell.java:456)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1
> Failing this attempt. Failing the application.
> 
> 
> and checking the yarn log via yarn logs -applicationId appId I have the
> following:
> 
> 2015-07-20 22:51:36,794 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1437424593766_0001_000002
> 2015-07-20 22:51:36,938 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.NoSuchMethodError:
> org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364)
> 2015-07-20 22:51:36,940 INFO [Thread-1]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a
> signal. Signaling RMCommunicator and JobHistoryEventHandler.
> 
> From the last error it seems that some component is invoking a method which
> has been removed since hadoop 2.3.0 and it is not found on the current
> version that I'm using (2.7.0)
> 
> P.S I can correctly execute the oozie spark example which works with a
> local master
> 
> I think it's still a problem of configuration/version between oozie and
> hadoop but after two days I can't figure out what is the problem. Did
> somebody already face this weird error?
> 
> 
> -- 
> Matteo Remo Luzzi