You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@twill.apache.org by Chris Hebert <ch...@digitalreasoning.com> on 2017/06/12 20:22:49 UTC
HelloWorld Struggle
I hate to ask this here, but it won't work, so whatever.
I followed the HelloWorld section of the Getting Started guide <
http://twill.apache.org/GettingStarted.html> on my cluster with Hadoop and
Zookeeper set up and functioning properly.
git clone https://github.com/apache/twill.git
cd twill
mvn clean install -DskipTests
export
CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-SNAPSHOT.jar:`hadoop
classpath`
java -cp $CP org.apache.twill.example.yarn.HelloWorld
my.zookeeper.domain:2181
Yes, `hadoop classpath` echoes all the relevant jar directories.
The command runs well for a bit with multiple:
[ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application
status for HelloWorldRunnable application_1111111111111_0001: ACCEPTED
until:
[ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
[ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client from
cache: org.apache.hadoop.ipc.Client@4d465b11
[ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application
status for HelloWorldRunnable application_1111111111111_0001: FAILED
...
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Yarn
application completed with failure HelloWorldRunnable...
The ResourceManager reveals:
Application application_1111111111111_0001 failed 2 times due to AM
Container for appattempt_1111111111111_0001_000002 exited with exitCode: 1
...
Diagnostics: Exception from container-launch.
The corresponding YARN logs for each DataNode reveal:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
Launch class (org.apache.twill.internal.appmaster.ApplicationMasterMain)
using classloader java.net.URLClassLoader with classpath: [
*A list of several classpaths like
"file:/some/path/yarn/local/usercache/my.username/appcache/application_1111111111111_0001/container_e10_
1111111111111
_0001_02_000001/application.jar/lib/twill-examples-yarn-0.12.0-SNAPSHOT.jar"
But none of which are paths to any Hadoop jars of the sort that are
referenced in $CP*
]
What am I missing?
I've spent an embarrassingly large amount of time on this fiddling with
environment variables and Hadoop configuration. (I'm an intern learning
this stuff the hard way, so it's not really embarrassing, just substantial.)
Re: HelloWorld Struggle
Posted by Chris Hebert <ch...@digitalreasoning.com>.
Thank you for following up. My project moved in another direction shortly
after this email exchange, so I didn't take the time to resolve it fully.
My difficulties were probably related to some configuration conflict like
the Hadoop version, as you pointed out.
Cheers!
On Wed, Jul 12, 2017 at 11:04 AM, Terence Yim <ch...@gmail.com> wrote:
> Hi Chris,
>
> Are you able to get pass the issue? One thing I noticed recently is that if
> your cluster is having different hadoop version that the one used to
> compile the hello world example (default is 2.3.0), it might not work. You
> can try to compile the example by `mvn clean package -DskipTests
> -Dhadoop.version=[cluster_hadoop_version]`.
>
> Terence
>
> On Tue, Jun 13, 2017 at 8:16 PM, Terence Yim <ch...@gmail.com> wrote:
>
> > Hi Chris,
> >
> > Are you able to get the stdout from the failure AM container? It shows
> > exactly what classpath it uses. Also, just to confirm, the jar paths that
> > you see when you run `hadoop classpath` from the client side are actually
> > exist, right?
> >
> > Terence
> >
> > On Tue, Jun 13, 2017 at 9:26 AM, Chris Hebert <chris.hebert-int@
> > digitalreasoning.com> wrote:
> >
> >> Yes. I verified that that all those environment variables resolve
> >> correctly
> >> on all the relevant nodes for all users and also in all the
> hadoop-env.sh
> >> files.
> >>
> >> Is there anything else I may be missing?
> >>
> >> On Mon, Jun 12, 2017 at 4:27 PM, Yuliya Feldman <yu...@dremio.com>
> >> wrote:
> >>
> >> > Do those env vars resolve on the NodeManager nodes correctly?
> >> >
> >> > On Mon, Jun 12, 2017 at 2:05 PM, Chris Hebert <
> >> > chris.hebert-int@digitalreasoning.com> wrote:
> >> >
> >> > > Other YARN apps work fine. For example, I just successfully ran the
> >> stock
> >> > > MapReduce wordcount example (and of course, MapReduce is a YARN
> >> > > application).
> >> > >
> >> > > I ran HelloWorld in debug mode earlier and found that yarnClasspath
> >> > > contains the following:
> >> > > $HADOOP_CONF_DIR
> >> > > $HADOOP_COMMON_HOME/*
> >> > > $HADOOP_COMMON_HOME/lib/*
> >> > > $HADOOP_HDFS_HOME/*
> >> > > $HADOOP_HDFS_HOME/lib/*
> >> > > $HADOOP_MAPRED_HOME/*
> >> > > $HADOOP_MAPRED_HOME/lib/*
> >> > > $HADOOP_YARN_HOME/*
> >> > > $HADOOP_YARN_HOME/lib/*
> >> > >
> >> > > I don't know whether these environment variables are supposed to be
> >> > > resolved to path variables already or if that happens later. I also
> >> don't
> >> > > know if I'm supposed to explicitly declare these environment
> variables
> >> > > somewhere, and if so, I do not know where in the Hadoop
> configuration
> >> it
> >> > is
> >> > > ideal for me to declare them. I tried declaring them in
> hadoop_env.sh
> >> and
> >> > > core-site.xml, but I'm not sure I did it right, and at least in my
> >> > initial
> >> > > efforts declaring these variables did not seem to prevent the error.
> >> If
> >> > it
> >> > > is the correct thing for me to set these variables appropriately
> >> > somewhere,
> >> > > then I will continue to try to do so.
> >> > >
> >> > > On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yu...@dremio.com>
> >> > wrote:
> >> > >
> >> > > > I don't think it is an issue with classpath on the client - since
> it
> >> > gets
> >> > > > to start AppMaster container
> >> > > >
> >> > > > Is any other YARN app runs OK?
> >> > > >
> >> > > > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not
> >> producing
> >> > > right
> >> > > > jars
> >> > > >
> >> > > > Look at HelloWorld code for yarnClasspath
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert <
> >> > > > chris.hebert-int@digitalreasoning.com> wrote:
> >> > > >
> >> > > > > I hate to ask this here, but it won't work, so whatever.
> >> > > > >
> >> > > > > I followed the HelloWorld section of the Getting Started guide <
> >> > > > > http://twill.apache.org/GettingStarted.html> on my cluster with
> >> > Hadoop
> >> > > > and
> >> > > > > Zookeeper set up and functioning properly.
> >> > > > >
> >> > > > > git clone https://github.com/apache/twill.git
> >> > > > > cd twill
> >> > > > > mvn clean install -DskipTests
> >> > > > >
> >> > > > > export
> >> > > > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-
> >> > > > > SNAPSHOT.jar:`hadoop
> >> > > > > classpath`
> >> > > > > java -cp $CP org.apache.twill.example.yarn.HelloWorld
> >> > > > > my.zookeeper.domain:2181
> >> > > > >
> >> > > > > Yes, `hadoop classpath` echoes all the relevant jar directories.
> >> > > > >
> >> > > > > The command runs well for a bit with multiple:
> >> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> >> > > application
> >> > > > > status for HelloWorldRunnable application_1111111111111_0001:
> >> > ACCEPTED
> >> > > > >
> >> > > > > until:
> >> > > > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
> >> > > > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered
> >> state
> >> > > > > STOPPED
> >> > > > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client
> >> from
> >> > > > > cache: org.apache.hadoop.ipc.Client@4d465b11
> >> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> >> > > application
> >> > > > > status for HelloWorldRunnable application_1111111111111_0001:
> >> FAILED
> >> > > > > ...
> >> > > > > java.util.concurrent.ExecutionException:
> >> java.lang.RuntimeException:
> >> > > > Yarn
> >> > > > > application completed with failure HelloWorldRunnable...
> >> > > > >
> >> > > > > The ResourceManager reveals:
> >> > > > > Application application_1111111111111_0001 failed 2 times due to
> >> AM
> >> > > > > Container for appattempt_1111111111111_0001_000002 exited with
> >> > > > exitCode: 1
> >> > > > > ...
> >> > > > > Diagnostics: Exception from container-launch.
> >> > > > >
> >> > > > > The corresponding YARN logs for each DataNode reveal:
> >> > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> >> > > > > org/apache/hadoop/conf/Configuration
> >> > > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> >> > > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> >> > > > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> >> > > > > at java.lang.Class.getMethod0(Class.java:3018)
> >> > > > > at java.lang.Class.getMethod(Class.java:1784)
> >> > > > > at org.apache.twill.launcher.TwillLauncher.main(
> >> > TwillLauncher.java:70)
> >> > > > > Caused by: java.lang.ClassNotFoundException:
> >> > > > > org.apache.hadoop.conf.Configuration
> >> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >> > > > > ... 6 more
> >> > > > > Launch class (org.apache.twill.internal.appmaster.
> >> > > ApplicationMasterMain)
> >> > > > > using classloader java.net.URLClassLoader with classpath: [
> >> > > > > *A list of several classpaths like
> >> > > > > "file:/some/path/yarn/local/usercache/my.username/
> >> > > appcache/application_
> >> > > > > 1111111111111_0001/container_e10_
> >> > > > > 1111111111111
> >> > > > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0.
> >> > > > > 12.0-SNAPSHOT.jar"
> >> > > > > But none of which are paths to any Hadoop jars of the sort that
> >> are
> >> > > > > referenced in $CP*
> >> > > > > ]
> >> > > > >
> >> > > > > What am I missing?
> >> > > > >
> >> > > > > I've spent an embarrassingly large amount of time on this
> fiddling
> >> > with
> >> > > > > environment variables and Hadoop configuration. (I'm an intern
> >> > learning
> >> > > > > this stuff the hard way, so it's not really embarrassing, just
> >> > > > > substantial.)
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: HelloWorld Struggle
Posted by Terence Yim <ch...@gmail.com>.
Hi Chris,
Are you able to get pass the issue? One thing I noticed recently is that if
your cluster is having different hadoop version that the one used to
compile the hello world example (default is 2.3.0), it might not work. You
can try to compile the example by `mvn clean package -DskipTests
-Dhadoop.version=[cluster_hadoop_version]`.
Terence
On Tue, Jun 13, 2017 at 8:16 PM, Terence Yim <ch...@gmail.com> wrote:
> Hi Chris,
>
> Are you able to get the stdout from the failure AM container? It shows
> exactly what classpath it uses. Also, just to confirm, the jar paths that
> you see when you run `hadoop classpath` from the client side are actually
> exist, right?
>
> Terence
>
> On Tue, Jun 13, 2017 at 9:26 AM, Chris Hebert <chris.hebert-int@
> digitalreasoning.com> wrote:
>
>> Yes. I verified that that all those environment variables resolve
>> correctly
>> on all the relevant nodes for all users and also in all the hadoop-env.sh
>> files.
>>
>> Is there anything else I may be missing?
>>
>> On Mon, Jun 12, 2017 at 4:27 PM, Yuliya Feldman <yu...@dremio.com>
>> wrote:
>>
>> > Do those env vars resolve on the NodeManager nodes correctly?
>> >
>> > On Mon, Jun 12, 2017 at 2:05 PM, Chris Hebert <
>> > chris.hebert-int@digitalreasoning.com> wrote:
>> >
>> > > Other YARN apps work fine. For example, I just successfully ran the
>> stock
>> > > MapReduce wordcount example (and of course, MapReduce is a YARN
>> > > application).
>> > >
>> > > I ran HelloWorld in debug mode earlier and found that yarnClasspath
>> > > contains the following:
>> > > $HADOOP_CONF_DIR
>> > > $HADOOP_COMMON_HOME/*
>> > > $HADOOP_COMMON_HOME/lib/*
>> > > $HADOOP_HDFS_HOME/*
>> > > $HADOOP_HDFS_HOME/lib/*
>> > > $HADOOP_MAPRED_HOME/*
>> > > $HADOOP_MAPRED_HOME/lib/*
>> > > $HADOOP_YARN_HOME/*
>> > > $HADOOP_YARN_HOME/lib/*
>> > >
>> > > I don't know whether these environment variables are supposed to be
>> > > resolved to path variables already or if that happens later. I also
>> don't
>> > > know if I'm supposed to explicitly declare these environment variables
>> > > somewhere, and if so, I do not know where in the Hadoop configuration
>> it
>> > is
>> > > ideal for me to declare them. I tried declaring them in hadoop_env.sh
>> and
>> > > core-site.xml, but I'm not sure I did it right, and at least in my
>> > initial
>> > > efforts declaring these variables did not seem to prevent the error.
>> If
>> > it
>> > > is the correct thing for me to set these variables appropriately
>> > somewhere,
>> > > then I will continue to try to do so.
>> > >
>> > > On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yu...@dremio.com>
>> > wrote:
>> > >
>> > > > I don't think it is an issue with classpath on the client - since it
>> > gets
>> > > > to start AppMaster container
>> > > >
>> > > > Is any other YARN app runs OK?
>> > > >
>> > > > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not
>> producing
>> > > right
>> > > > jars
>> > > >
>> > > > Look at HelloWorld code for yarnClasspath
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert <
>> > > > chris.hebert-int@digitalreasoning.com> wrote:
>> > > >
>> > > > > I hate to ask this here, but it won't work, so whatever.
>> > > > >
>> > > > > I followed the HelloWorld section of the Getting Started guide <
>> > > > > http://twill.apache.org/GettingStarted.html> on my cluster with
>> > Hadoop
>> > > > and
>> > > > > Zookeeper set up and functioning properly.
>> > > > >
>> > > > > git clone https://github.com/apache/twill.git
>> > > > > cd twill
>> > > > > mvn clean install -DskipTests
>> > > > >
>> > > > > export
>> > > > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-
>> > > > > SNAPSHOT.jar:`hadoop
>> > > > > classpath`
>> > > > > java -cp $CP org.apache.twill.example.yarn.HelloWorld
>> > > > > my.zookeeper.domain:2181
>> > > > >
>> > > > > Yes, `hadoop classpath` echoes all the relevant jar directories.
>> > > > >
>> > > > > The command runs well for a bit with multiple:
>> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
>> > > application
>> > > > > status for HelloWorldRunnable application_1111111111111_0001:
>> > ACCEPTED
>> > > > >
>> > > > > until:
>> > > > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
>> > > > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered
>> state
>> > > > > STOPPED
>> > > > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client
>> from
>> > > > > cache: org.apache.hadoop.ipc.Client@4d465b11
>> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
>> > > application
>> > > > > status for HelloWorldRunnable application_1111111111111_0001:
>> FAILED
>> > > > > ...
>> > > > > java.util.concurrent.ExecutionException:
>> java.lang.RuntimeException:
>> > > > Yarn
>> > > > > application completed with failure HelloWorldRunnable...
>> > > > >
>> > > > > The ResourceManager reveals:
>> > > > > Application application_1111111111111_0001 failed 2 times due to
>> AM
>> > > > > Container for appattempt_1111111111111_0001_000002 exited with
>> > > > exitCode: 1
>> > > > > ...
>> > > > > Diagnostics: Exception from container-launch.
>> > > > >
>> > > > > The corresponding YARN logs for each DataNode reveal:
>> > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
>> > > > > org/apache/hadoop/conf/Configuration
>> > > > > at java.lang.Class.getDeclaredMethods0(Native Method)
>> > > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
>> > > > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
>> > > > > at java.lang.Class.getMethod0(Class.java:3018)
>> > > > > at java.lang.Class.getMethod(Class.java:1784)
>> > > > > at org.apache.twill.launcher.TwillLauncher.main(
>> > TwillLauncher.java:70)
>> > > > > Caused by: java.lang.ClassNotFoundException:
>> > > > > org.apache.hadoop.conf.Configuration
>> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> > > > > ... 6 more
>> > > > > Launch class (org.apache.twill.internal.appmaster.
>> > > ApplicationMasterMain)
>> > > > > using classloader java.net.URLClassLoader with classpath: [
>> > > > > *A list of several classpaths like
>> > > > > "file:/some/path/yarn/local/usercache/my.username/
>> > > appcache/application_
>> > > > > 1111111111111_0001/container_e10_
>> > > > > 1111111111111
>> > > > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0.
>> > > > > 12.0-SNAPSHOT.jar"
>> > > > > But none of which are paths to any Hadoop jars of the sort that
>> are
>> > > > > referenced in $CP*
>> > > > > ]
>> > > > >
>> > > > > What am I missing?
>> > > > >
>> > > > > I've spent an embarrassingly large amount of time on this fiddling
>> > with
>> > > > > environment variables and Hadoop configuration. (I'm an intern
>> > learning
>> > > > > this stuff the hard way, so it's not really embarrassing, just
>> > > > > substantial.)
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: HelloWorld Struggle
Posted by Terence Yim <ch...@gmail.com>.
Hi Chris,
Are you able to get the stdout from the failure AM container? It shows
exactly what classpath it uses. Also, just to confirm, the jar paths that
you see when you run `hadoop classpath` from the client side are actually
exist, right?
Terence
On Tue, Jun 13, 2017 at 9:26 AM, Chris Hebert <
chris.hebert-int@digitalreasoning.com> wrote:
> Yes. I verified that that all those environment variables resolve correctly
> on all the relevant nodes for all users and also in all the hadoop-env.sh
> files.
>
> Is there anything else I may be missing?
>
> On Mon, Jun 12, 2017 at 4:27 PM, Yuliya Feldman <yu...@dremio.com> wrote:
>
> > Do those env vars resolve on the NodeManager nodes correctly?
> >
> > On Mon, Jun 12, 2017 at 2:05 PM, Chris Hebert <
> > chris.hebert-int@digitalreasoning.com> wrote:
> >
> > > Other YARN apps work fine. For example, I just successfully ran the
> stock
> > > MapReduce wordcount example (and of course, MapReduce is a YARN
> > > application).
> > >
> > > I ran HelloWorld in debug mode earlier and found that yarnClasspath
> > > contains the following:
> > > $HADOOP_CONF_DIR
> > > $HADOOP_COMMON_HOME/*
> > > $HADOOP_COMMON_HOME/lib/*
> > > $HADOOP_HDFS_HOME/*
> > > $HADOOP_HDFS_HOME/lib/*
> > > $HADOOP_MAPRED_HOME/*
> > > $HADOOP_MAPRED_HOME/lib/*
> > > $HADOOP_YARN_HOME/*
> > > $HADOOP_YARN_HOME/lib/*
> > >
> > > I don't know whether these environment variables are supposed to be
> > > resolved to path variables already or if that happens later. I also
> don't
> > > know if I'm supposed to explicitly declare these environment variables
> > > somewhere, and if so, I do not know where in the Hadoop configuration
> it
> > is
> > > ideal for me to declare them. I tried declaring them in hadoop_env.sh
> and
> > > core-site.xml, but I'm not sure I did it right, and at least in my
> > initial
> > > efforts declaring these variables did not seem to prevent the error. If
> > it
> > > is the correct thing for me to set these variables appropriately
> > somewhere,
> > > then I will continue to try to do so.
> > >
> > > On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yu...@dremio.com>
> > wrote:
> > >
> > > > I don't think it is an issue with classpath on the client - since it
> > gets
> > > > to start AppMaster container
> > > >
> > > > Is any other YARN app runs OK?
> > > >
> > > > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not producing
> > > right
> > > > jars
> > > >
> > > > Look at HelloWorld code for yarnClasspath
> > > >
> > > >
> > > >
> > > > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert <
> > > > chris.hebert-int@digitalreasoning.com> wrote:
> > > >
> > > > > I hate to ask this here, but it won't work, so whatever.
> > > > >
> > > > > I followed the HelloWorld section of the Getting Started guide <
> > > > > http://twill.apache.org/GettingStarted.html> on my cluster with
> > Hadoop
> > > > and
> > > > > Zookeeper set up and functioning properly.
> > > > >
> > > > > git clone https://github.com/apache/twill.git
> > > > > cd twill
> > > > > mvn clean install -DskipTests
> > > > >
> > > > > export
> > > > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-
> > > > > SNAPSHOT.jar:`hadoop
> > > > > classpath`
> > > > > java -cp $CP org.apache.twill.example.yarn.HelloWorld
> > > > > my.zookeeper.domain:2181
> > > > >
> > > > > Yes, `hadoop classpath` echoes all the relevant jar directories.
> > > > >
> > > > > The command runs well for a bit with multiple:
> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> > > application
> > > > > status for HelloWorldRunnable application_1111111111111_0001:
> > ACCEPTED
> > > > >
> > > > > until:
> > > > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
> > > > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered
> state
> > > > > STOPPED
> > > > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client
> from
> > > > > cache: org.apache.hadoop.ipc.Client@4d465b11
> > > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> > > application
> > > > > status for HelloWorldRunnable application_1111111111111_0001:
> FAILED
> > > > > ...
> > > > > java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> > > > Yarn
> > > > > application completed with failure HelloWorldRunnable...
> > > > >
> > > > > The ResourceManager reveals:
> > > > > Application application_1111111111111_0001 failed 2 times due to AM
> > > > > Container for appattempt_1111111111111_0001_000002 exited with
> > > > exitCode: 1
> > > > > ...
> > > > > Diagnostics: Exception from container-launch.
> > > > >
> > > > > The corresponding YARN logs for each DataNode reveal:
> > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > org/apache/hadoop/conf/Configuration
> > > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> > > > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> > > > > at java.lang.Class.getMethod0(Class.java:3018)
> > > > > at java.lang.Class.getMethod(Class.java:1784)
> > > > > at org.apache.twill.launcher.TwillLauncher.main(
> > TwillLauncher.java:70)
> > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > org.apache.hadoop.conf.Configuration
> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > ... 6 more
> > > > > Launch class (org.apache.twill.internal.appmaster.
> > > ApplicationMasterMain)
> > > > > using classloader java.net.URLClassLoader with classpath: [
> > > > > *A list of several classpaths like
> > > > > "file:/some/path/yarn/local/usercache/my.username/
> > > appcache/application_
> > > > > 1111111111111_0001/container_e10_
> > > > > 1111111111111
> > > > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0.
> > > > > 12.0-SNAPSHOT.jar"
> > > > > But none of which are paths to any Hadoop jars of the sort that are
> > > > > referenced in $CP*
> > > > > ]
> > > > >
> > > > > What am I missing?
> > > > >
> > > > > I've spent an embarrassingly large amount of time on this fiddling
> > with
> > > > > environment variables and Hadoop configuration. (I'm an intern
> > learning
> > > > > this stuff the hard way, so it's not really embarrassing, just
> > > > > substantial.)
> > > > >
> > > >
> > >
> >
>
Re: HelloWorld Struggle
Posted by Chris Hebert <ch...@digitalreasoning.com>.
Yes. I verified that that all those environment variables resolve correctly
on all the relevant nodes for all users and also in all the hadoop-env.sh
files.
Is there anything else I may be missing?
On Mon, Jun 12, 2017 at 4:27 PM, Yuliya Feldman <yu...@dremio.com> wrote:
> Do those env vars resolve on the NodeManager nodes correctly?
>
> On Mon, Jun 12, 2017 at 2:05 PM, Chris Hebert <
> chris.hebert-int@digitalreasoning.com> wrote:
>
> > Other YARN apps work fine. For example, I just successfully ran the stock
> > MapReduce wordcount example (and of course, MapReduce is a YARN
> > application).
> >
> > I ran HelloWorld in debug mode earlier and found that yarnClasspath
> > contains the following:
> > $HADOOP_CONF_DIR
> > $HADOOP_COMMON_HOME/*
> > $HADOOP_COMMON_HOME/lib/*
> > $HADOOP_HDFS_HOME/*
> > $HADOOP_HDFS_HOME/lib/*
> > $HADOOP_MAPRED_HOME/*
> > $HADOOP_MAPRED_HOME/lib/*
> > $HADOOP_YARN_HOME/*
> > $HADOOP_YARN_HOME/lib/*
> >
> > I don't know whether these environment variables are supposed to be
> > resolved to path variables already or if that happens later. I also don't
> > know if I'm supposed to explicitly declare these environment variables
> > somewhere, and if so, I do not know where in the Hadoop configuration it
> is
> > ideal for me to declare them. I tried declaring them in hadoop_env.sh and
> > core-site.xml, but I'm not sure I did it right, and at least in my
> initial
> > efforts declaring these variables did not seem to prevent the error. If
> it
> > is the correct thing for me to set these variables appropriately
> somewhere,
> > then I will continue to try to do so.
> >
> > On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yu...@dremio.com>
> wrote:
> >
> > > I don't think it is an issue with classpath on the client - since it
> gets
> > > to start AppMaster container
> > >
> > > Is any other YARN app runs OK?
> > >
> > > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not producing
> > right
> > > jars
> > >
> > > Look at HelloWorld code for yarnClasspath
> > >
> > >
> > >
> > > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert <
> > > chris.hebert-int@digitalreasoning.com> wrote:
> > >
> > > > I hate to ask this here, but it won't work, so whatever.
> > > >
> > > > I followed the HelloWorld section of the Getting Started guide <
> > > > http://twill.apache.org/GettingStarted.html> on my cluster with
> Hadoop
> > > and
> > > > Zookeeper set up and functioning properly.
> > > >
> > > > git clone https://github.com/apache/twill.git
> > > > cd twill
> > > > mvn clean install -DskipTests
> > > >
> > > > export
> > > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-
> > > > SNAPSHOT.jar:`hadoop
> > > > classpath`
> > > > java -cp $CP org.apache.twill.example.yarn.HelloWorld
> > > > my.zookeeper.domain:2181
> > > >
> > > > Yes, `hadoop classpath` echoes all the relevant jar directories.
> > > >
> > > > The command runs well for a bit with multiple:
> > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> > application
> > > > status for HelloWorldRunnable application_1111111111111_0001:
> ACCEPTED
> > > >
> > > > until:
> > > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
> > > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
> > > > STOPPED
> > > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client from
> > > > cache: org.apache.hadoop.ipc.Client@4d465b11
> > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> > application
> > > > status for HelloWorldRunnable application_1111111111111_0001: FAILED
> > > > ...
> > > > java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> > > Yarn
> > > > application completed with failure HelloWorldRunnable...
> > > >
> > > > The ResourceManager reveals:
> > > > Application application_1111111111111_0001 failed 2 times due to AM
> > > > Container for appattempt_1111111111111_0001_000002 exited with
> > > exitCode: 1
> > > > ...
> > > > Diagnostics: Exception from container-launch.
> > > >
> > > > The corresponding YARN logs for each DataNode reveal:
> > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > org/apache/hadoop/conf/Configuration
> > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> > > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> > > > at java.lang.Class.getMethod0(Class.java:3018)
> > > > at java.lang.Class.getMethod(Class.java:1784)
> > > > at org.apache.twill.launcher.TwillLauncher.main(
> TwillLauncher.java:70)
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.hadoop.conf.Configuration
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > ... 6 more
> > > > Launch class (org.apache.twill.internal.appmaster.
> > ApplicationMasterMain)
> > > > using classloader java.net.URLClassLoader with classpath: [
> > > > *A list of several classpaths like
> > > > "file:/some/path/yarn/local/usercache/my.username/
> > appcache/application_
> > > > 1111111111111_0001/container_e10_
> > > > 1111111111111
> > > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0.
> > > > 12.0-SNAPSHOT.jar"
> > > > But none of which are paths to any Hadoop jars of the sort that are
> > > > referenced in $CP*
> > > > ]
> > > >
> > > > What am I missing?
> > > >
> > > > I've spent an embarrassingly large amount of time on this fiddling
> with
> > > > environment variables and Hadoop configuration. (I'm an intern
> learning
> > > > this stuff the hard way, so it's not really embarrassing, just
> > > > substantial.)
> > > >
> > >
> >
>
Re: HelloWorld Struggle
Posted by Yuliya Feldman <yu...@dremio.com>.
Do those env vars resolve on the NodeManager nodes correctly?
On Mon, Jun 12, 2017 at 2:05 PM, Chris Hebert <
chris.hebert-int@digitalreasoning.com> wrote:
> Other YARN apps work fine. For example, I just successfully ran the stock
> MapReduce wordcount example (and of course, MapReduce is a YARN
> application).
>
> I ran HelloWorld in debug mode earlier and found that yarnClasspath
> contains the following:
> $HADOOP_CONF_DIR
> $HADOOP_COMMON_HOME/*
> $HADOOP_COMMON_HOME/lib/*
> $HADOOP_HDFS_HOME/*
> $HADOOP_HDFS_HOME/lib/*
> $HADOOP_MAPRED_HOME/*
> $HADOOP_MAPRED_HOME/lib/*
> $HADOOP_YARN_HOME/*
> $HADOOP_YARN_HOME/lib/*
>
> I don't know whether these environment variables are supposed to be
> resolved to path variables already or if that happens later. I also don't
> know if I'm supposed to explicitly declare these environment variables
> somewhere, and if so, I do not know where in the Hadoop configuration it is
> ideal for me to declare them. I tried declaring them in hadoop_env.sh and
> core-site.xml, but I'm not sure I did it right, and at least in my initial
> efforts declaring these variables did not seem to prevent the error. If it
> is the correct thing for me to set these variables appropriately somewhere,
> then I will continue to try to do so.
>
> On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yu...@dremio.com> wrote:
>
> > I don't think it is an issue with classpath on the client - since it gets
> > to start AppMaster container
> >
> > Is any other YARN app runs OK?
> >
> > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not producing
> right
> > jars
> >
> > Look at HelloWorld code for yarnClasspath
> >
> >
> >
> > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert <
> > chris.hebert-int@digitalreasoning.com> wrote:
> >
> > > I hate to ask this here, but it won't work, so whatever.
> > >
> > > I followed the HelloWorld section of the Getting Started guide <
> > > http://twill.apache.org/GettingStarted.html> on my cluster with Hadoop
> > and
> > > Zookeeper set up and functioning properly.
> > >
> > > git clone https://github.com/apache/twill.git
> > > cd twill
> > > mvn clean install -DskipTests
> > >
> > > export
> > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-
> > > SNAPSHOT.jar:`hadoop
> > > classpath`
> > > java -cp $CP org.apache.twill.example.yarn.HelloWorld
> > > my.zookeeper.domain:2181
> > >
> > > Yes, `hadoop classpath` echoes all the relevant jar directories.
> > >
> > > The command runs well for a bit with multiple:
> > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> application
> > > status for HelloWorldRunnable application_1111111111111_0001: ACCEPTED
> > >
> > > until:
> > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
> > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
> > > STOPPED
> > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client from
> > > cache: org.apache.hadoop.ipc.Client@4d465b11
> > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn
> application
> > > status for HelloWorldRunnable application_1111111111111_0001: FAILED
> > > ...
> > > java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> > Yarn
> > > application completed with failure HelloWorldRunnable...
> > >
> > > The ResourceManager reveals:
> > > Application application_1111111111111_0001 failed 2 times due to AM
> > > Container for appattempt_1111111111111_0001_000002 exited with
> > exitCode: 1
> > > ...
> > > Diagnostics: Exception from container-launch.
> > >
> > > The corresponding YARN logs for each DataNode reveal:
> > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/conf/Configuration
> > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> > > at java.lang.Class.getMethod0(Class.java:3018)
> > > at java.lang.Class.getMethod(Class.java:1784)
> > > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70)
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.hadoop.conf.Configuration
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > ... 6 more
> > > Launch class (org.apache.twill.internal.appmaster.
> ApplicationMasterMain)
> > > using classloader java.net.URLClassLoader with classpath: [
> > > *A list of several classpaths like
> > > "file:/some/path/yarn/local/usercache/my.username/
> appcache/application_
> > > 1111111111111_0001/container_e10_
> > > 1111111111111
> > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0.
> > > 12.0-SNAPSHOT.jar"
> > > But none of which are paths to any Hadoop jars of the sort that are
> > > referenced in $CP*
> > > ]
> > >
> > > What am I missing?
> > >
> > > I've spent an embarrassingly large amount of time on this fiddling with
> > > environment variables and Hadoop configuration. (I'm an intern learning
> > > this stuff the hard way, so it's not really embarrassing, just
> > > substantial.)
> > >
> >
>
Re: HelloWorld Struggle
Posted by Chris Hebert <ch...@digitalreasoning.com>.
Other YARN apps work fine. For example, I just successfully ran the stock
MapReduce wordcount example (and of course, MapReduce is a YARN
application).
I ran HelloWorld in debug mode earlier and found that yarnClasspath
contains the following:
$HADOOP_CONF_DIR
$HADOOP_COMMON_HOME/*
$HADOOP_COMMON_HOME/lib/*
$HADOOP_HDFS_HOME/*
$HADOOP_HDFS_HOME/lib/*
$HADOOP_MAPRED_HOME/*
$HADOOP_MAPRED_HOME/lib/*
$HADOOP_YARN_HOME/*
$HADOOP_YARN_HOME/lib/*
I don't know whether these environment variables are supposed to be
resolved to path variables already or if that happens later. I also don't
know if I'm supposed to explicitly declare these environment variables
somewhere, and if so, I do not know where in the Hadoop configuration it is
ideal for me to declare them. I tried declaring them in hadoop_env.sh and
core-site.xml, but I'm not sure I did it right, and at least in my initial
efforts declaring these variables did not seem to prevent the error. If it
is the correct thing for me to set these variables appropriately somewhere,
then I will continue to try to do so.
On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yu...@dremio.com> wrote:
> I don't think it is an issue with classpath on the client - since it gets
> to start AppMaster container
>
> Is any other YARN app runs OK?
>
> May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not producing right
> jars
>
> Look at HelloWorld code for yarnClasspath
>
>
>
> On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert <
> chris.hebert-int@digitalreasoning.com> wrote:
>
> > I hate to ask this here, but it won't work, so whatever.
> >
> > I followed the HelloWorld section of the Getting Started guide <
> > http://twill.apache.org/GettingStarted.html> on my cluster with Hadoop
> and
> > Zookeeper set up and functioning properly.
> >
> > git clone https://github.com/apache/twill.git
> > cd twill
> > mvn clean install -DskipTests
> >
> > export
> > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-
> > SNAPSHOT.jar:`hadoop
> > classpath`
> > java -cp $CP org.apache.twill.example.yarn.HelloWorld
> > my.zookeeper.domain:2181
> >
> > Yes, `hadoop classpath` echoes all the relevant jar directories.
> >
> > The command runs well for a bit with multiple:
> > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application
> > status for HelloWorldRunnable application_1111111111111_0001: ACCEPTED
> >
> > until:
> > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
> > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
> > STOPPED
> > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client from
> > cache: org.apache.hadoop.ipc.Client@4d465b11
> > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application
> > status for HelloWorldRunnable application_1111111111111_0001: FAILED
> > ...
> > java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> Yarn
> > application completed with failure HelloWorldRunnable...
> >
> > The ResourceManager reveals:
> > Application application_1111111111111_0001 failed 2 times due to AM
> > Container for appattempt_1111111111111_0001_000002 exited with
> exitCode: 1
> > ...
> > Diagnostics: Exception from container-launch.
> >
> > The corresponding YARN logs for each DataNode reveal:
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/hadoop/conf/Configuration
> > at java.lang.Class.getDeclaredMethods0(Native Method)
> > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> > at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> > at java.lang.Class.getMethod0(Class.java:3018)
> > at java.lang.Class.getMethod(Class.java:1784)
> > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.conf.Configuration
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > ... 6 more
> > Launch class (org.apache.twill.internal.appmaster.ApplicationMasterMain)
> > using classloader java.net.URLClassLoader with classpath: [
> > *A list of several classpaths like
> > "file:/some/path/yarn/local/usercache/my.username/appcache/application_
> > 1111111111111_0001/container_e10_
> > 1111111111111
> > _0001_02_000001/application.jar/lib/twill-examples-yarn-0.
> > 12.0-SNAPSHOT.jar"
> > But none of which are paths to any Hadoop jars of the sort that are
> > referenced in $CP*
> > ]
> >
> > What am I missing?
> >
> > I've spent an embarrassingly large amount of time on this fiddling with
> > environment variables and Hadoop configuration. (I'm an intern learning
> > this stuff the hard way, so it's not really embarrassing, just
> > substantial.)
> >
>
Re: HelloWorld Struggle
Posted by Yuliya Feldman <yu...@dremio.com>.
I don't think it is an issue with classpath on the client - since it gets
to start AppMaster container
Is any other YARN app runs OK?
May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not producing right
jars
Look at HelloWorld code for yarnClasspath
On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert <
chris.hebert-int@digitalreasoning.com> wrote:
> I hate to ask this here, but it won't work, so whatever.
>
> I followed the HelloWorld section of the Getting Started guide <
> http://twill.apache.org/GettingStarted.html> on my cluster with Hadoop and
> Zookeeper set up and functioning properly.
>
> git clone https://github.com/apache/twill.git
> cd twill
> mvn clean install -DskipTests
>
> export
> CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0-
> SNAPSHOT.jar:`hadoop
> classpath`
> java -cp $CP org.apache.twill.example.yarn.HelloWorld
> my.zookeeper.domain:2181
>
> Yes, `hadoop classpath` echoes all the relevant jar directories.
>
> The command runs well for a bit with multiple:
> [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application
> status for HelloWorldRunnable application_1111111111111_0001: ACCEPTED
>
> until:
> [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service:
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
> STOPPED
> [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client from
> cache: org.apache.hadoop.ipc.Client@4d465b11
> [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application
> status for HelloWorldRunnable application_1111111111111_0001: FAILED
> ...
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Yarn
> application completed with failure HelloWorldRunnable...
>
> The ResourceManager reveals:
> Application application_1111111111111_0001 failed 2 times due to AM
> Container for appattempt_1111111111111_0001_000002 exited with exitCode: 1
> ...
> Diagnostics: Exception from container-launch.
>
> The corresponding YARN logs for each DataNode reveal:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/conf/Configuration
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.conf.Configuration
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 6 more
> Launch class (org.apache.twill.internal.appmaster.ApplicationMasterMain)
> using classloader java.net.URLClassLoader with classpath: [
> *A list of several classpaths like
> "file:/some/path/yarn/local/usercache/my.username/appcache/application_
> 1111111111111_0001/container_e10_
> 1111111111111
> _0001_02_000001/application.jar/lib/twill-examples-yarn-0.
> 12.0-SNAPSHOT.jar"
> But none of which are paths to any Hadoop jars of the sort that are
> referenced in $CP*
> ]
>
> What am I missing?
>
> I've spent an embarrassingly large amount of time on this fiddling with
> environment variables and Hadoop configuration. (I'm an intern learning
> this stuff the hard way, so it's not really embarrassing, just
> substantial.)
>