You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Harold Nguyen <ha...@nexgate.com> on 2014/10/28 19:08:08 UTC
Including jars in Spark-shell vs Spark-submit
Hi all,
The following works fine when submitting dependency jars through
Spark-Shell:
./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars
/home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
But not through spark-submit:
./bin/spark-submit --class
"org.apache.spark.examples.streaming.CassandraSave" --master
spark://ip-172-31-38-112:7077
streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar --jars
local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
Am I issuing the spark-submit command incorrectly ? Each of the workers has
that built jar in their respective directories
(spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar)
Thanks,
Harold
Re: Including jars in Spark-shell vs Spark-submit
Posted by Helena Edelson <he...@datastax.com>.
Ah excellent! I will be sure to check if we need to update our documentation based on your feedback :)
Cheers,
Helena
On Oct 28, 2014, at 3:03 PM, Harold Nguyen <ha...@nexgate.com> wrote:
> Hi Helena,
>
> It's great to e-meet you! I've actually been following along your blogs and talks trying to get this to work.
>
> I just solved it, and you were absolutely correct. I've been using 1.1.0-alpha3 as my dependency, but my assembly is the 1.2.0-SNAPSHOT.
>
> Thanks for looking through all my other threads to piece together the problem.
>
> So the solution is that I just included the spark-cassandra-connector-assembly-1.2.0-SNAPSHOT straight into a "lib" folder, removed the line from 1.1.0-alpha3 line from libraryDepenencies, and re-packaged it.
>
> It all works great now, and I couldn't be happier. I had to piece together 6 different forums and sites to get that working (being absolutely new to Spark and Scala and sbt). I'll write a blog post on how to get this working later, in case it can help someone.
>
> I really appreciate the help!
>
> Harold
>
> On Tue, Oct 28, 2014 at 11:55 AM, Helena Edelson <he...@datastax.com> wrote:
> Hi Harold,
> It seems like, based on your previous post, you are using one version of the connector as a dependency yet building the assembly jar from master? You were using 1.1.0-alpha3 (you can upgrade to alpha4, beta coming this week) yet your assembly is spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.
>
> libraryDependencies ++= Seq(
> "org.apache.spark" % "spark-streaming_2.10" % "1.1.0",
> "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3" withSources() withJavadoc(),
> "org.apache.spark" %% "spark-sql" % "1.1.0"
> )
>
> - Helena
>
> On Oct 28, 2014, at 2:08 PM, Harold Nguyen <ha...@nexgate.com> wrote:
>
> > Hi all,
> >
> > The following works fine when submitting dependency jars through Spark-Shell:
> >
> > ./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars /home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> >
> > But not through spark-submit:
> >
> > ./bin/spark-submit --class "org.apache.spark.examples.streaming.CassandraSave" --master spark://ip-172-31-38-112:7077 streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar --jars local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> >
> > Am I issuing the spark-submit command incorrectly ? Each of the workers has that built jar in their respective directories (spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar)
> >
> > Thanks,
> >
> > Harold
>
>
Re: Including jars in Spark-shell vs Spark-submit
Posted by Harold Nguyen <ha...@nexgate.com>.
Hi Helena,
It's great to e-meet you! I've actually been following along your blogs and
talks trying to get this to work.
I just solved it, and you were absolutely correct. I've been using
1.1.0-alpha3 as my dependency, but my assembly is the 1.2.0-SNAPSHOT.
Thanks for looking through all my other threads to piece together the
problem.
So the solution is that I just included the
spark-cassandra-connector-assembly-1.2.0-SNAPSHOT
straight into a "lib" folder, removed the line from 1.1.0-alpha3 line from
libraryDepenencies, and re-packaged it.
It all works great now, and I couldn't be happier. I had to piece together
6 different forums and sites to get that working (being absolutely new to
Spark and Scala and sbt). I'll write a blog post on how to get this working
later, in case it can help someone.
I really appreciate the help!
Harold
On Tue, Oct 28, 2014 at 11:55 AM, Helena Edelson <
helena.edelson@datastax.com> wrote:
> Hi Harold,
> It seems like, based on your previous post, you are using one version of
> the connector as a dependency yet building the assembly jar from master?
> You were using 1.1.0-alpha3 (you can upgrade to alpha4, beta coming this
> week) yet your assembly is
> spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.
>
> libraryDependencies ++= Seq(
> "org.apache.spark" % "spark-streaming_2.10" % "1.1.0",
> "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3"
> withSources() withJavadoc(),
> "org.apache.spark" %% "spark-sql" % "1.1.0"
> )
>
> - Helena
>
> On Oct 28, 2014, at 2:08 PM, Harold Nguyen <ha...@nexgate.com> wrote:
>
> > Hi all,
> >
> > The following works fine when submitting dependency jars through
> Spark-Shell:
> >
> > ./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars
> /home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> >
> > But not through spark-submit:
> >
> > ./bin/spark-submit --class
> "org.apache.spark.examples.streaming.CassandraSave" --master
> spark://ip-172-31-38-112:7077
> streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar --jars
> local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> >
> > Am I issuing the spark-submit command incorrectly ? Each of the workers
> has that built jar in their respective directories
> (spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar)
> >
> > Thanks,
> >
> > Harold
>
>
Re: Including jars in Spark-shell vs Spark-submit
Posted by Helena Edelson <he...@datastax.com>.
Hi Harold,
It seems like, based on your previous post, you are using one version of the connector as a dependency yet building the assembly jar from master? You were using 1.1.0-alpha3 (you can upgrade to alpha4, beta coming this week) yet your assembly is spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-streaming_2.10" % "1.1.0",
"com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3" withSources() withJavadoc(),
"org.apache.spark" %% "spark-sql" % "1.1.0"
)
- Helena
On Oct 28, 2014, at 2:08 PM, Harold Nguyen <ha...@nexgate.com> wrote:
> Hi all,
>
> The following works fine when submitting dependency jars through Spark-Shell:
>
> ./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars /home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
>
> But not through spark-submit:
>
> ./bin/spark-submit --class "org.apache.spark.examples.streaming.CassandraSave" --master spark://ip-172-31-38-112:7077 streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar --jars local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
>
> Am I issuing the spark-submit command incorrectly ? Each of the workers has that built jar in their respective directories (spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar)
>
> Thanks,
>
> Harold
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org