You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Gino Mathews <gi...@thinkpalm.com> on 2014/03/11 14:02:22 UTC
Spark stand alone cluster mode
Hi,
I am new to spark.
I would like to run jobs in Spark stand alone cluster mode.
No cluser managers other than spark is used. (https://spark.apache.org/docs/0.9.0/spark-standalone.html)
I have tried wordcount from spark shell and stand alone scala app.
The code reads input from HDFS and writes the results to HDFS. uses 2 worker nodes.
In spark-shell the wordcount is successful, how ever my effort to run stand alone programmes are in vain.
My environement
Ubuntu 12.04 - 32 bit
JAVA 1.7.0_51
I have installed spark @ $HOME/Downloads/spark-0.9.0-incubating
installed hadoop 2.2.0 as separate hduser and given permission to other users.
installed scala 2.10.3
installed sbt 0.13.1
Spark master act as HDFS master
I have one master and 2 worker nodes and HDFS is accessible in all nodes.
I downloaded example project and modified to use my spark cluster.
I started the sparkcluster at spark://192.168.0.138:7077
and hdfs://master:9000/
When I run the project as SPARK_HADOOP_VERSION=2.2.0 sbt run, I get following error
gino@master:~/Test/spark-example-project$ SPARK_HADOOP_VERSION=2.2.0 sbt run
[info] Loading project definition from /home/gino/Test/spark-example-project/project
[info] Set current project to spark-example-project (in build file:/home/gino/Test/spark-example-project/)
[info] Running com.Thinkpalm.spark.WordCountHDFS
[error] (run-main-0) java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 0 s, completed Mar 11, 2014 2:54:54 PM
Could anyone give some pointers ... I have attached the project for reference.
Thanks and regards
Gino Mathews
Re: Spark stand alone cluster mode
Posted by Yana Kadiyska <ya...@gmail.com>.
does sbt "show full-classpath" show spark-core on the classpath? I am
still pretty new to scala but it seems like you have val sparkCore
= "org.apache.spark" %% "spark-core" % V.spark %
"provided" -- I believe the "provided" part means it's in your
classpath. Spark-shell script sets up a lot of stuff for you so...
On Tue, Mar 11, 2014 at 9:02 AM, Gino Mathews <gi...@thinkpalm.com> wrote:
> Hi,
>
> I am new to spark.
> I would like to run jobs in Spark stand alone cluster mode.
>
> No cluser managers other than spark is used.
> (https://spark.apache.org/docs/0.9.0/spark-standalone.html)
> I have tried wordcount from spark shell and stand alone scala app.
>
> The code reads input from HDFS and writes the results to HDFS. uses 2 worker
> nodes.
>
> In spark-shell the wordcount is successful, how ever my effort to run stand
> alone programmes are in vain.
>
>
>
> My environement
>
> Ubuntu 12.04 - 32 bit
>
> JAVA 1.7.0_51
>
> I have installed spark @ $HOME/Downloads/spark-0.9.0-incubating
> installed hadoop 2.2.0 as separate hduser and given permission to other
> users.
> installed scala 2.10.3
> installed sbt 0.13.1
>
> Spark master act as HDFS master
> I have one master and 2 worker nodes and HDFS is accessible in all nodes.
> I downloaded example project and modified to use my spark cluster.
> I started the sparkcluster at spark://192.168.0.138:7077
> and hdfs://master:9000/
> When I run the project as SPARK_HADOOP_VERSION=2.2.0 sbt run, I get
> following error
>
> gino@master:~/Test/spark-example-project$ SPARK_HADOOP_VERSION=2.2.0 sbt run
> [info] Loading project definition from
> /home/gino/Test/spark-example-project/project
> [info] Set current project to spark-example-project (in build
> file:/home/gino/Test/spark-example-project/)
> [info] Running com.Thinkpalm.spark.WordCountHDFS
> [error] (run-main-0) java.lang.NoClassDefFoundError:
> org/apache/spark/SparkContext
> java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
> at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
> at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
> at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> [trace] Stack trace suppressed: run last compile:run for the full output.
> java.lang.RuntimeException: Nonzero exit code: 1
> at scala.sys.package$.error(package.scala:27)
> [trace] Stack trace suppressed: run last compile:run for the full output.
> [error] (compile:run) Nonzero exit code: 1
> [error] Total time: 0 s, completed Mar 11, 2014 2:54:54 PM
>
> Could anyone give some pointers ... I have attached the project for
> reference.
>
>
>
> Thanks and regards
>
> Gino Mathews