You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sadaf <sa...@platalytics.com> on 2015/08/04 10:29:26 UTC
Twitter live Streaming
Hi
Is there any way to get all old tweets since when the account was created
using spark streaming and twitters api? Currently my connector is showing
those tweets that get posted after the program runs. I've done this task
using spark streaming and a custom receiver using "twitter user api".
Thanks in anticipation.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-live-Streaming-tp24124.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Twitter live Streaming
Posted by Enno Shioji <es...@gmail.com>.
If you want to do it through streaming API you have to pay Gnip; it's not free. You can go through non-streaming Twitter API and convert it to stream yourself though.
> On 4 Aug 2015, at 09:29, Sadaf <sa...@platalytics.com> wrote:
>
> Hi
> Is there any way to get all old tweets since when the account was created
> using spark streaming and twitters api? Currently my connector is showing
> those tweets that get posted after the program runs. I've done this task
> using spark streaming and a custom receiver using "twitter user api".
>
> Thanks in anticipation.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-live-Streaming-tp24124.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> Q
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Twitter live Streaming
Posted by pradyumnad <pr...@gmail.com>.
Streaming API, as in the name, gives out the live stream of tweets which are
posted right then.
If you would like to get the old tweets use the rest API from Twitter.
Twitter4j is the twitter library that I use and suggest for the task.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-live-Streaming-tp24124p24202.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
AW: Twitter live Streaming
Posted by Filli Alem <Al...@ti8m.ch>.
Hi Sadaf,
Im currently struggling with Twitter Streaming as well. I cant get it working using the simple setup bellow. I use spark 1.2 and I replaced twitter4j v3 with v4. Am I doing something wrong? How are you doing this?
twitter4j.conf.Configuration conf = new twitter4j.conf.ConfigurationBuilder()
.setOAuthConsumerKey("********************")
.setOAuthConsumerSecret("*********************")
.setOAuthAccessToken("*****************")
.setOAuthAccessTokenSecret("**************************").build();
TwitterFactory tf =new TwitterFactory(conf);
Authorization a = new OAuthAuthorization(conf);
Authorization a2 = tf.getInstance(a).getAuthorization();
SparkConf sparkConf = new SparkConf().setAppName("TwitterStreamJob");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,Durations.seconds(30));
String[] filter = new String[]{"football"};
JavaReceiverInputDStream<Status> receiverStream = TwitterUtils.createStream(jssc, a2, filter);
JavaDStream<String> tweets= receiverStream.map(new Function<Status, String>() {
@Override
public String call(Status tweet) throws Exception {
return tweet.getUser().getName() +"_" + tweet.getText() +"_" + tweet.getCreatedAt().getTime();
}
});
tweets.foreachRDD(new Function<JavaRDD<String>, Void>() {
@Override
public Void call(JavaRDD<String> arg0) throws Exception {
arg0.saveAsTextFile("hdfs://myhost/results/twitter_" + UUID.randomUUID().toString());
return null;
}
});
jssc.start();
jssc.awaitTermination();
-----Ursprüngliche Nachricht-----
Von: Sadaf [mailto:sadaf@platalytics.com]
Gesendet: Dienstag, 4. August 2015 10:29
An: user@spark.apache.org
Betreff: Twitter live Streaming
Hi
Is there any way to get all old tweets since when the account was created using spark streaming and twitters api? Currently my connector is showing those tweets that get posted after the program runs. I've done this task using spark streaming and a custom receiver using "twitter user api".
Thanks in anticipation.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-live-Streaming-tp24124.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
[http://www.ti8m.ch/fileadmin/daten/ti8m/Bilder/footer/Footer_Paymit_klein.jpg]<https://www.ti8m.ch/competences/garage.html>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org