You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Izhar ul Hassan <ez...@gmail.com> on 2013/12/27 14:40:17 UTC

Errors with spark-0.8.1 hadoop-yarn 2.2.0

Hi,

I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed
spark-0.8.1 with support for spark enabled. I get the following errors when
trying to run the examples:

SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar
\
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar
\
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1


Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program
will exit.

spark-0.8.0 with hadooop 2.0.5-alpha works fine.
-- 
/Izhar

Re: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Posted by Azuryy Yu <az...@gmail.com>.
Maybe you don't enabled Yarn when build Spark.


On Mon, Dec 30, 2013 at 8:36 AM, Liu, Raymond <ra...@intel.com> wrote:

> Hi Izhar
>
>         Is that the exact command you are running? Say with 0.8.0 instead
> of 0.8.1 in the cmd?
>
> Raymond Liu
>
> From: Izhar ul Hassan [mailto:ezhaar@gmail.com]
> Sent: Friday, December 27, 2013 9:40 PM
> To: user@spark.incubator.apache.org
> Subject: Errors with spark-0.8.1 hadoop-yarn 2.2.0
>
> Hi,
> I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed
> spark-0.8.1 with support for spark enabled. I get the following errors when
> trying to run the examples:
> SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar
> \
>     ./spark-class org.apache.spark.deploy.yarn.Client \
>       --jar
> examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
>       --class org.apache.spark.examples.SparkPi \
>       --args yarn-standalone \
>       --num-workers 3 \
>       --master-memory 4g \
>       --worker-memory 2g \
>       --worker-cores 1
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/deploy/yarn/Client
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.Client
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
> Could not find the main class: org.apache.spark.deploy.yarn.Client.
> Program will exit.
> spark-0.8.0 with hadooop 2.0.5-alpha works fine.
> --
> /Izhar
>

Re: RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Posted by "leosandylh@gmail.com" <le...@gmail.com>.
What is your classpath ?
Had you builded your spark when you changed the new version and with yarn?
Have you find your jar under the $SPARK_HOME/assembly/target/scala-2.9.3  ? or there is not just only one ?




leosandylh@gmail.com

From: Liu, Raymond
Date: 2013-12-30 08:36
To: user@spark.incubator.apache.org
Subject: RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0
Hi Izhar

Is that the exact command you are running? Say with 0.8.0 instead of 0.8.1 in the cmd?

Raymond Liu

From: Izhar ul Hassan [mailto:ezhaar@gmail.com] 
Sent: Friday, December 27, 2013 9:40 PM
To: user@spark.incubator.apache.org
Subject: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Hi,
I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed spark-0.8.1 with support for spark enabled. I get the following errors when trying to run the examples:
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program will exit.
spark-0.8.0 with hadooop 2.0.5-alpha works fine.
-- 
/Izhar 

RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Posted by "Liu, Raymond" <ra...@intel.com>.
Hi Izhar

	Is that the exact command you are running? Say with 0.8.0 instead of 0.8.1 in the cmd?

Raymond Liu

From: Izhar ul Hassan [mailto:ezhaar@gmail.com] 
Sent: Friday, December 27, 2013 9:40 PM
To: user@spark.incubator.apache.org
Subject: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Hi,
I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed spark-0.8.1 with support for spark enabled. I get the following errors when trying to run the examples:
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program will exit.
spark-0.8.0 with hadooop 2.0.5-alpha works fine.
-- 
/Izhar