You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sadaf <sa...@platalytics.com> on 2015/07/30 12:49:27 UTC
Twitter Connector-Spark Streaming
Hi.
I am writing twitter connector using spark streaming. but it fetched the
random tweets.
Is there any way to receive the tweets of a particular account?
I made an app on twitter and used the credentials as given below.
def managingCredentials(): Option[twitter4j.auth.Authorization]=
{
object auth{
val config = new twitter4j.conf.ConfigurationBuilder()
.setOAuthConsumerKey("****")
.setOAuthConsumerSecret("****")
.setOAuthAccessToken("****")
.setOAuthAccessTokenSecret("****")
.build
}
val twitter_auth = new TwitterFactory(auth.config)
val a = new twitter4j.auth.OAuthAuthorization(auth.config)
val atwitter : Option[twitter4j.auth.Authorization] =
Some(twitter_auth.getInstance(a).getAuthorization())
atwitter
}
Thanks :)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Twitter Connector-Spark Streaming
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
> Hi Sadaf
>
> Which version of spark are you using? And whats in the spark-env.sh file?
> I think you are using both SPARK_CLASSPATH (which is deprecated) and
> spark.executor.extraClasspath (may be set in spark-defaults.sh file).
>
> Thanks
> Best Regards
>
> On Wed, Aug 5, 2015 at 6:22 PM, Sadaf Khan <sa...@platalytics.com> wrote:
>
>> Hi Akhil,
>>
>> I know you are a big knowledge base of spark streaming.
>> I wan your little help more.
>> I've done the twitter streaming using twitter's streaming user api and
>> spark streaming. this runs successfully on my local machine. but when i run
>> this program on cluster in local mode. it just run successfully for the
>> very first time. later on it gives the following exception.
>>
>> "Exception in thread "main" org.apache.spark.SparkException: Found both
>> spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former."
>>
>> and spark class path is unset already!!
>> I have to make a new checkpoint directory each time to make it run
>> successfully. otherwise it shows above exception.
>>
>> Can you please again help me to resolve this issue?
>> Thanks :)
>>
>> On Tue, Aug 4, 2015 at 4:33 PM, Sadaf Khan <sa...@platalytics.com> wrote:
>>
>>> thanks for being a helping hand before. :)
>>>
>>> one more point i want to discuss is that i now used Twitter Rest Api and
>>> successfully fetch the home timeline. but now it is showing 20 recent
>>> tweets,
>>> Is there any way to get 3200 (max rate limit) last tweets?
>>>
>>> Thanks in anticipation :)
>>>
>>>
>>> On Tue, Aug 4, 2015 at 2:05 PM, Sadaf Khan <sa...@platalytics.com>
>>> wrote:
>>>
>>>> thanks alot
>>>>
>>>> On Tue, Aug 4, 2015 at 2:00 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>>> wrote:
>>>>
>>>>> You will have to write your own consumer for pulling your custom
>>>>> feeds, and then you can do a union (customfeedDstream.union(twitterStream))
>>>>> with the twitter stream api.
>>>>>
>>>>> Thanks
>>>>> Best Regards
>>>>>
>>>>> On Tue, Aug 4, 2015 at 2:28 PM, Sadaf Khan <sa...@platalytics.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks alot :)
>>>>>>
>>>>>> One more thing that i want to ask is that i have used twitters
>>>>>> streaming api.and it seems that the above solution uses rest api. how can i
>>>>>> used both simultaneously ?
>>>>>>
>>>>>> Any response will be much appreciated :)
>>>>>> Regards
>>>>>>
>>>>>> On Tue, Aug 4, 2015 at 1:51 PM, Akhil Das <akhil@sigmoidanalytics.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Yes you can, when you start the application in the first batch you
>>>>>>> just need to pull all the tweets from your account. You need to look into
>>>>>>> the API for that. Have a look at this
>>>>>>> https://dev.twitter.com/rest/reference/get/statuses/user_timeline
>>>>>>>
>>>>>>> Thanks
>>>>>>> Best Regards
>>>>>>>
>>>>>>> On Tue, Aug 4, 2015 at 1:47 PM, Sadaf Khan <sa...@platalytics.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi.
>>>>>>>> You were really helpful for me last time :) and i have done with
>>>>>>>> the last problem.
>>>>>>>> I wanna ask you one more question. Now my connector is showing the
>>>>>>>> tweets that occurs after running the program. Is there any way to fetch all
>>>>>>>> old tweets since when the account was created?
>>>>>>>>
>>>>>>>> I will be thankful to you for you kind response.
>>>>>>>>
>>>>>>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan <sa...@platalytics.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> thanks alot for this help :)
>>>>>>>>>
>>>>>>>>> On Thu, Jul 30, 2015 at 6:41 PM, Akhil Das <
>>>>>>>>> akhil@sigmoidanalytics.com> wrote:
>>>>>>>>>
>>>>>>>>>> You can create a custom receiver and then inside it you can write
>>>>>>>>>> yourown piece of code to receive data, filter them etc before giving it to
>>>>>>>>>> spark.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Best Regards
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan <
>>>>>>>>>> sadaf@platalytics.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> okay :)
>>>>>>>>>>>
>>>>>>>>>>> then is there anyway to fetch the tweets specific to my account?
>>>>>>>>>>>
>>>>>>>>>>> Thanks in anticipation :)
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das <
>>>>>>>>>>> akhil@sigmoidanalytics.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Owh, this one fetches the public tweets, not the one specific
>>>>>>>>>>>> to your account.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan <
>>>>>>>>>>>> sadaf@platalytics.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> yes. but can you please tell me how to mention a specific user
>>>>>>>>>>>>> account in filter?
>>>>>>>>>>>>> I want to fetch my tweets, tweets of my followers and the
>>>>>>>>>>>>> tweets of those whom i followed.
>>>>>>>>>>>>> So in short i want to fatch the tweets of my account only.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Recently i have used
>>>>>>>>>>>>> val tweets
>>>>>>>>>>>>> =TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any response will be very much appreciated. :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das <
>>>>>>>>>>>>> akhil@sigmoidanalytics.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> TwitterUtils.createStream takes a 3rd argument which is a
>>>>>>>>>>>>>> filter, once you provide these, it will only fetch tweets of such.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jul 30, 2015 at 4:19 PM, Sadaf <sadaf@platalytics.com
>>>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>> I am writing twitter connector using spark streaming. but it
>>>>>>>>>>>>>>> fetched the
>>>>>>>>>>>>>>> random tweets.
>>>>>>>>>>>>>>> Is there any way to receive the tweets of a particular
>>>>>>>>>>>>>>> account?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I made an app on twitter and used the credentials as given
>>>>>>>>>>>>>>> below.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> def managingCredentials():
>>>>>>>>>>>>>>> Option[twitter4j.auth.Authorization]=
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> object auth{
>>>>>>>>>>>>>>> val config = new twitter4j.conf.ConfigurationBuilder()
>>>>>>>>>>>>>>> .setOAuthConsumerKey("****")
>>>>>>>>>>>>>>> .setOAuthConsumerSecret("****")
>>>>>>>>>>>>>>> .setOAuthAccessToken("****")
>>>>>>>>>>>>>>> .setOAuthAccessTokenSecret("****")
>>>>>>>>>>>>>>> .build
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> val twitter_auth = new TwitterFactory(auth.config)
>>>>>>>>>>>>>>> val a = new
>>>>>>>>>>>>>>> twitter4j.auth.OAuthAuthorization(auth.config)
>>>>>>>>>>>>>>> val atwitter : Option[twitter4j.auth.Authorization] =
>>>>>>>>>>>>>>> Some(twitter_auth.getInstance(a).getAuthorization())
>>>>>>>>>>>>>>> atwitter
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
>>>>>>>>>>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Twitter Connector-Spark Streaming
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You will have to write your own consumer for pulling your custom feeds, and
then you can do a union (customfeedDstream.union(twitterStream)) with the
twitter stream api.
Thanks
Best Regards
On Tue, Aug 4, 2015 at 2:28 PM, Sadaf Khan <sa...@platalytics.com> wrote:
> Thanks alot :)
>
> One more thing that i want to ask is that i have used twitters streaming
> api.and it seems that the above solution uses rest api. how can i used both
> simultaneously ?
>
> Any response will be much appreciated :)
> Regards
>
> On Tue, Aug 4, 2015 at 1:51 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Yes you can, when you start the application in the first batch you just
>> need to pull all the tweets from your account. You need to look into the
>> API for that. Have a look at this
>> https://dev.twitter.com/rest/reference/get/statuses/user_timeline
>>
>> Thanks
>> Best Regards
>>
>> On Tue, Aug 4, 2015 at 1:47 PM, Sadaf Khan <sa...@platalytics.com> wrote:
>>
>>> Hi.
>>> You were really helpful for me last time :) and i have done with the
>>> last problem.
>>> I wanna ask you one more question. Now my connector is showing the
>>> tweets that occurs after running the program. Is there any way to fetch all
>>> old tweets since when the account was created?
>>>
>>> I will be thankful to you for you kind response.
>>>
>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan <sa...@platalytics.com>
>>> wrote:
>>>
>>>> thanks alot for this help :)
>>>>
>>>> On Thu, Jul 30, 2015 at 6:41 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>>> wrote:
>>>>
>>>>> You can create a custom receiver and then inside it you can write
>>>>> yourown piece of code to receive data, filter them etc before giving it to
>>>>> spark.
>>>>>
>>>>> Thanks
>>>>> Best Regards
>>>>>
>>>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan <sa...@platalytics.com>
>>>>> wrote:
>>>>>
>>>>>> okay :)
>>>>>>
>>>>>> then is there anyway to fetch the tweets specific to my account?
>>>>>>
>>>>>> Thanks in anticipation :)
>>>>>>
>>>>>> On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das <
>>>>>> akhil@sigmoidanalytics.com> wrote:
>>>>>>
>>>>>>> Owh, this one fetches the public tweets, not the one specific to
>>>>>>> your account.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Best Regards
>>>>>>>
>>>>>>> On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan <sa...@platalytics.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> yes. but can you please tell me how to mention a specific user
>>>>>>>> account in filter?
>>>>>>>> I want to fetch my tweets, tweets of my followers and the tweets of
>>>>>>>> those whom i followed.
>>>>>>>> So in short i want to fatch the tweets of my account only.
>>>>>>>>
>>>>>>>> Recently i have used
>>>>>>>> val tweets
>>>>>>>> =TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2)
>>>>>>>>
>>>>>>>>
>>>>>>>> Any response will be very much appreciated. :)
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das <
>>>>>>>> akhil@sigmoidanalytics.com> wrote:
>>>>>>>>
>>>>>>>>> TwitterUtils.createStream takes a 3rd argument which is a filter,
>>>>>>>>> once you provide these, it will only fetch tweets of such.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Best Regards
>>>>>>>>>
>>>>>>>>> On Thu, Jul 30, 2015 at 4:19 PM, Sadaf <sa...@platalytics.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>> I am writing twitter connector using spark streaming. but it
>>>>>>>>>> fetched the
>>>>>>>>>> random tweets.
>>>>>>>>>> Is there any way to receive the tweets of a particular account?
>>>>>>>>>>
>>>>>>>>>> I made an app on twitter and used the credentials as given below.
>>>>>>>>>>
>>>>>>>>>> def managingCredentials(): Option[twitter4j.auth.Authorization]=
>>>>>>>>>> {
>>>>>>>>>> object auth{
>>>>>>>>>> val config = new twitter4j.conf.ConfigurationBuilder()
>>>>>>>>>> .setOAuthConsumerKey("****")
>>>>>>>>>> .setOAuthConsumerSecret("****")
>>>>>>>>>> .setOAuthAccessToken("****")
>>>>>>>>>> .setOAuthAccessTokenSecret("****")
>>>>>>>>>> .build
>>>>>>>>>> }
>>>>>>>>>> val twitter_auth = new TwitterFactory(auth.config)
>>>>>>>>>> val a = new twitter4j.auth.OAuthAuthorization(auth.config)
>>>>>>>>>> val atwitter : Option[twitter4j.auth.Authorization] =
>>>>>>>>>> Some(twitter_auth.getInstance(a).getAuthorization())
>>>>>>>>>> atwitter
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Thanks :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
>>>>>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>>>>>> Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>>>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Twitter Connector-Spark Streaming
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can create a custom receiver and then inside it you can write yourown
piece of code to receive data, filter them etc before giving it to spark.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan <sa...@platalytics.com> wrote:
> okay :)
>
> then is there anyway to fetch the tweets specific to my account?
>
> Thanks in anticipation :)
>
> On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Owh, this one fetches the public tweets, not the one specific to your
>> account.
>>
>> Thanks
>> Best Regards
>>
>> On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan <sa...@platalytics.com>
>> wrote:
>>
>>> yes. but can you please tell me how to mention a specific user account
>>> in filter?
>>> I want to fetch my tweets, tweets of my followers and the tweets of
>>> those whom i followed.
>>> So in short i want to fatch the tweets of my account only.
>>>
>>> Recently i have used
>>> val tweets
>>> =TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2)
>>>
>>>
>>> Any response will be very much appreciated. :)
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> TwitterUtils.createStream takes a 3rd argument which is a filter, once
>>>> you provide these, it will only fetch tweets of such.
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Thu, Jul 30, 2015 at 4:19 PM, Sadaf <sa...@platalytics.com> wrote:
>>>>
>>>>> Hi.
>>>>> I am writing twitter connector using spark streaming. but it fetched
>>>>> the
>>>>> random tweets.
>>>>> Is there any way to receive the tweets of a particular account?
>>>>>
>>>>> I made an app on twitter and used the credentials as given below.
>>>>>
>>>>> def managingCredentials(): Option[twitter4j.auth.Authorization]=
>>>>> {
>>>>> object auth{
>>>>> val config = new twitter4j.conf.ConfigurationBuilder()
>>>>> .setOAuthConsumerKey("****")
>>>>> .setOAuthConsumerSecret("****")
>>>>> .setOAuthAccessToken("****")
>>>>> .setOAuthAccessTokenSecret("****")
>>>>> .build
>>>>> }
>>>>> val twitter_auth = new TwitterFactory(auth.config)
>>>>> val a = new twitter4j.auth.OAuthAuthorization(auth.config)
>>>>> val atwitter : Option[twitter4j.auth.Authorization] =
>>>>> Some(twitter_auth.getInstance(a).getAuthorization())
>>>>> atwitter
>>>>> }
>>>>>
>>>>> Thanks :)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Twitter Connector-Spark Streaming
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
TwitterUtils.createStream takes a 3rd argument which is a filter, once you
provide these, it will only fetch tweets of such.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 4:19 PM, Sadaf <sa...@platalytics.com> wrote:
> Hi.
> I am writing twitter connector using spark streaming. but it fetched the
> random tweets.
> Is there any way to receive the tweets of a particular account?
>
> I made an app on twitter and used the credentials as given below.
>
> def managingCredentials(): Option[twitter4j.auth.Authorization]=
> {
> object auth{
> val config = new twitter4j.conf.ConfigurationBuilder()
> .setOAuthConsumerKey("****")
> .setOAuthConsumerSecret("****")
> .setOAuthAccessToken("****")
> .setOAuthAccessTokenSecret("****")
> .build
> }
> val twitter_auth = new TwitterFactory(auth.config)
> val a = new twitter4j.auth.OAuthAuthorization(auth.config)
> val atwitter : Option[twitter4j.auth.Authorization] =
> Some(twitter_auth.getInstance(a).getAuthorization())
> atwitter
> }
>
> Thanks :)
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>