You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by xeonmailinglist <xe...@gmail.com> on 2015/02/25 13:47:48 UTC
Couldn't load classpath in a remote mapreduce job execution
Hi,
I am trying to launch a remote mapreduce job, but I get these 2 errors
below. The first error is the output of the execution. The second error
happens in the tasks.
Looking to the error, and my code, it seems that the job couldn’t load
|yarn.application.classpath| defined in the code. The path that I define
there is set in the |.bashrc| file (I show it below).
It seems that this error is related to the env vars set in my job, but
this vars are set in the |.bashrc|. Any help to understand how I fix
this error?
*Output of execution:*
|For more detailed output, check application tracking page:http://hadoop-coc-1:8088/proxy/application_1424867716027_0002/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1424867716027_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
|
*Task error*
|
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
|
*My code*
|String tempPath = "temp-" + System.currentTimeMillis();
for (String host : remoteHosts) {
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":8032");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.application.classpath ","$HADOOP_CONF_DIR, $HADOOP_HOME/share/hadoop/common/*, $HADOOP_HOME/share/hadoop/common/lib/*, "
+ "$HADOOP_HOME/share/hadoop/hdfs/*, $HADOOP_HOME/share/hadoop/hdfs/lib/*,"
+ "$HADOOP_HOME/share/hadoop/mapreduce/*, $HADOOP_HOME/share/hadoop/mapreduce/lib/*, $HADOOP_HOME/share/hadoop/yarn/*,"
+ "$HADOOP_HOME/share/hadoop/yarn/lib/*, $HADOOP_HOME/*"
);
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
// FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileInputFormat.addInputPath(job, new Path(inputPath.toString()));
// FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
FileOutputFormat.setOutputPath(job, new Path(tempPath));
job.waitForCompletion(true);
}
|
*My .bashrc*
| JAVA_HOME=/usr/lib/jvm/java-8-oracle
HADOOP_HOME=$HOME/Programs/hadoop
HADOOP_CONF_HOME=$HADOOP_HOME/etc/hadoop
export HADOOP_MANAGER=$HOME/Programs/hadoop-mapreduce-manager-python
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
|
Thanks,