Couldn't load classpath in a remote mapreduce job execution

[xeonmailinglist <xe...@gmail.com>]

Hi,

I am trying to launch a remote mapreduce job, but I get these 2 errors 
below. The first error is the output of the execution. The second error 
happens in the tasks.

Looking to the error, and my code, it seems that the job couldn’t load 
|yarn.application.classpath| defined in the code. The path that I define 
there is set in the |.bashrc| file (I show it below).

It seems that this error is related to the env vars set in my job, but 
this vars are set in the |.bashrc|. Any help to understand how I fix 
this error?

*Output of execution:*

|For more detailed output, check application tracking page:http://hadoop-coc-1:8088/proxy/application_1424867716027_0002/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1424867716027_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
     at org.apache.hadoop.util.Shell.run(Shell.java:455)
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
     at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)
|

*Task error*

|
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
|

*My code*

|String tempPath = "temp-" + System.currentTimeMillis();
         for (String host : remoteHosts) {
             Configuration conf = job.getConfiguration();
             // this should be like defined in your yarn-site.xml
             conf.set("yarn.resourcemanager.address", host + ":8032");

             // framework is now "yarn", should be defined like this in mapred-site.xm
             conf.set("mapreduce.framework.name", "yarn");
             conf.set("yarn.application.classpath ","$HADOOP_CONF_DIR, $HADOOP_HOME/share/hadoop/common/*, $HADOOP_HOME/share/hadoop/common/lib/*, "
                     + "$HADOOP_HOME/share/hadoop/hdfs/*, $HADOOP_HOME/share/hadoop/hdfs/lib/*,"
                     + "$HADOOP_HOME/share/hadoop/mapreduce/*, $HADOOP_HOME/share/hadoop/mapreduce/lib/*, $HADOOP_HOME/share/hadoop/yarn/*,"
                     + "$HADOOP_HOME/share/hadoop/yarn/lib/*, $HADOOP_HOME/*"
                     );

             // like defined in hdfs-site.xml
             conf.set("fs.defaultFS", "hdfs://" + host + ":9000");

             for (Path inputPath : inputPaths)
                 //                FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
                 FileInputFormat.addInputPath(job, new Path(inputPath.toString()));

             //            FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
             FileOutputFormat.setOutputPath(job, new Path(tempPath));

             job.waitForCompletion(true);
}
|

*My .bashrc*

|   JAVA_HOME=/usr/lib/jvm/java-8-oracle
     HADOOP_HOME=$HOME/Programs/hadoop
     HADOOP_CONF_HOME=$HADOOP_HOME/etc/hadoop
     export HADOOP_MANAGER=$HOME/Programs/hadoop-mapreduce-manager-python
     export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
|

Thanks,

​