Container Classpath w/ MiniYarnCluster

[Abraham Elmahrek <ab...@cloudera.com>]

Hello,

I was trying to use YarnMiniCluster in some of my test cases, but ran into
some problems. From YARN-129, it seems the classpath of all Yarn attempts
should be set to the java classpath when executing YARNRunner, but it
doesn't seem to be the case. I had consistently found the following error
in my attempt logs:

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/service/CompositeService
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.service.CompositeService
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 12 more

Could not find the main class:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.  Program will exit.

The work around was to change "yarn.application.classpath".

For my purposes, setting "Yarn.application.classpath" seems fine. In the
case where I have an MR job that uses a JAR file in distributed cache, then
the attempt will definitely fail. Any thoughts on how this should be
addressed?

-Abe