You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nick Chammas <ni...@gmail.com> on 2014/07/09 20:44:27 UTC
How should I add a jar?
I’m just starting to use the Scala version of Spark’s shell, and I’d like
to add in a jar I believe I need to access Twitter data live, twitter4j
<http://twitter4j.org/en/index.html>. I’m confused over where and how to
add this jar in.
SPARK-1089 <https://issues.apache.org/jira/browse/SPARK-1089> mentions two
environment variables, SPARK_CLASSPATH and ADD_JARS. SparkContext also has
an addJar method and a jars property, the latter of which does not have an
associated doc
<http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext>
.
What’s the difference between all these jar-related things, and what do I
need to do to add this Twitter jar in correctly?
Nick
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-should-I-add-a-jar-tp9224.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How should I add a jar?
Posted by Nicholas Chammas <ni...@gmail.com>.
Public service announcement:
If you're trying to do some stream processing on Twitter data, you'll need
version 3.0.6 of twitter4j <http://twitter4j.org/archive/>. That should
work with the Spark Streaming 1.0.0 Twitter library.
The latest version of twitter4j, 4.0.2, appears to have breaking changes in
its API for us.
Nick
On Wed, Jul 9, 2014 at 3:34 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:
> Awww yessss. That worked! Thank you Sameer.
>
> Is this documented somewhere? I feel there there's a slight doc deficiency
> here.
>
> Nick
>
>
> On Wed, Jul 9, 2014 at 2:50 PM, Sameer Tilak <ss...@live.com> wrote:
>
>> Hi Nicholas,
>>
>> I am using Spark 1.0 and I use this method to specify the additional
>> jars. First jar is the dependency and the second one is my application.
>> Hope this will work for you.
>>
>> ./spark-shell --jars
>> /apps/software/secondstring/secondstring/dist/lib/secondstring-20140630.jar,/apps/software/scala-approsstrmatch/approxstrmatch.jar
>>
>>
>>
>> ------------------------------
>> Date: Wed, 9 Jul 2014 11:44:27 -0700
>> From: nicholas.chammas@gmail.com
>> To: user@spark.incubator.apache.org
>> Subject: How should I add a jar?
>>
>>
>> I’m just starting to use the Scala version of Spark’s shell, and I’d like
>> to add in a jar I believe I need to access Twitter data live, twitter4j
>> <http://twitter4j.org/en/index.html>. I’m confused over where and how to
>> add this jar in.
>>
>> SPARK-1089 <https://issues.apache.org/jira/browse/SPARK-1089> mentions
>> two environment variables, SPARK_CLASSPATH and ADD_JARS. SparkContext
>> also has an addJar method and a jars property, the latter of which does
>> not have an associated doc
>> <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext>
>> .
>>
>> What’s the difference between all these jar-related things, and what do I
>> need to do to add this Twitter jar in correctly?
>>
>> Nick
>>
>>
>> ------------------------------
>> View this message in context: How should I add a jar?
>> <http://apache-spark-user-list.1001560.n3.nabble.com/How-should-I-add-a-jar-tp9224.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>
>
Re: How should I add a jar?
Posted by Nicholas Chammas <ni...@gmail.com>.
Awww yessss. That worked! Thank you Sameer.
Is this documented somewhere? I feel there there's a slight doc deficiency
here.
Nick
On Wed, Jul 9, 2014 at 2:50 PM, Sameer Tilak <ss...@live.com> wrote:
> Hi Nicholas,
>
> I am using Spark 1.0 and I use this method to specify the additional jars.
> First jar is the dependency and the second one is my application. Hope this
> will work for you.
>
> ./spark-shell --jars
> /apps/software/secondstring/secondstring/dist/lib/secondstring-20140630.jar,/apps/software/scala-approsstrmatch/approxstrmatch.jar
>
>
>
> ------------------------------
> Date: Wed, 9 Jul 2014 11:44:27 -0700
> From: nicholas.chammas@gmail.com
> To: user@spark.incubator.apache.org
> Subject: How should I add a jar?
>
>
> I’m just starting to use the Scala version of Spark’s shell, and I’d like
> to add in a jar I believe I need to access Twitter data live, twitter4j
> <http://twitter4j.org/en/index.html>. I’m confused over where and how to
> add this jar in.
>
> SPARK-1089 <https://issues.apache.org/jira/browse/SPARK-1089> mentions
> two environment variables, SPARK_CLASSPATH and ADD_JARS. SparkContext
> also has an addJar method and a jars property, the latter of which does
> not have an associated doc
> <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext>
> .
>
> What’s the difference between all these jar-related things, and what do I
> need to do to add this Twitter jar in correctly?
>
> Nick
>
>
> ------------------------------
> View this message in context: How should I add a jar?
> <http://apache-spark-user-list.1001560.n3.nabble.com/How-should-I-add-a-jar-tp9224.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
RE: How should I add a jar?
Posted by Sameer Tilak <ss...@live.com>.
Hi Nicholas,
I am using Spark 1.0 and I use this method to specify the additional jars. First jar is the dependency and the second one is my application. Hope this will work for you.
./spark-shell --jars /apps/software/secondstring/secondstring/dist/lib/secondstring-20140630.jar,/apps/software/scala-approsstrmatch/approxstrmatch.jar
Date: Wed, 9 Jul 2014 11:44:27 -0700
From: nicholas.chammas@gmail.com
To: user@spark.incubator.apache.org
Subject: How should I add a jar?
I’m just starting to use the Scala version of Spark’s shell, and I’d like to add in a jar I believe I need to access Twitter data live, twitter4j. I’m confused over where and how to add this jar in.
SPARK-1089 mentions two environment variables, SPARK_CLASSPATH and ADD_JARS. SparkContext also has an addJar method and a jars property, the latter of which does not have an associated doc.
What’s the difference between all these jar-related things, and what do I need to do to add this Twitter jar in correctly?
Nick
View this message in context: How should I add a jar?
Sent from the Apache Spark User List mailing list archive at Nabble.com.