You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by SK <sk...@gmail.com> on 2014/10/10 02:22:58 UTC

getting tweets for a specified handle

Hi,
    I am using Spark 1.1.0.  Is there a way to get the complete tweets
corresponding to a handle (for e.g. @Delta)? I tried using the following
example that extracts just the hashtags and replaced the "#" with "@" as
follows.  I need the complete tweet and not just the tags. 

  //  val hashTags = stream.flatMap(status => status.getText.split("
").filter(_.startsWith("#")))

  // replaced the above with the following:
  val tweets = stream.map(status =>
status.getText.filter(_.contains("@Delta")))

  But I get an error:  " value contains is not a member of Char"

  I am trying to find out if there a better  Spark API to get the tweets for
a handle so that we dont have to do the filtering -  Something along the
lines of searchTwitter(topic, number_of_tags) API that is provided by the
twitteR package in R would be appropriate. 

thanks








--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/getting-tweets-for-a-specified-handle-tp16085.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: getting tweets for a specified handle

Posted by SK <sk...@gmail.com>.

Thanks. I made the change and ran the code. But I dont get any tweets for my
handle, although I do see the tweets when I search for it on twitter. Does
Spark allow us to get the tweets from the past (say the last 100 tweets?
tweets that appeared in the last 10 minutes)?

thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/getting-tweets-for-a-specified-handle-tp16085p16180.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: getting tweets for a specified handle

Posted by Frank Austin Nothaft <fn...@berkeley.edu>.

Hi SK,

In the code you reference, 

> status => status.getText.split("
> ").filter(_.startsWith("#")))

gets the text of the tweet and splits it at whitespace. Once the tweet is split into words, it then filters out all words that don’t start with ‘#’ (e.g., that aren’t hashtags), and returns those words back to the flatMap transformation.

In your code, you don’t want to do a map, since you’re not changing the tweets. Instead, you want to do a filter, which will keep all tweets that pass a predicate function, i.e., 

> stream.map(status =>
> status.getText.filter(_.contains("@Delta")))

should be:

> stream.filter(status =>
> status.getText.contains("@Delta"))


Regards,

Frank Austin Nothaft
fnothaft@berkeley.edu
fnothaft@eecs.berkeley.edu
202-340-0466

On Oct 9, 2014, at 5:22 PM, SK <sk...@gmail.com> wrote:

> Hi,
>    I am using Spark 1.1.0.  Is there a way to get the complete tweets
> corresponding to a handle (for e.g. @Delta)? I tried using the following
> example that extracts just the hashtags and replaced the "#" with "@" as
> follows.  I need the complete tweet and not just the tags. 
> 
>  //  val hashTags = stream.flatMap(status => status.getText.split("
> ").filter(_.startsWith("#")))
> 
>  // replaced the above with the following:
>  val tweets = stream.map(status =>
> status.getText.filter(_.contains("@Delta")))
> 
>  But I get an error:  " value contains is not a member of Char"
> 
>  I am trying to find out if there a better  Spark API to get the tweets for
> a handle so that we dont have to do the filtering -  Something along the
> lines of searchTwitter(topic, number_of_tags) API that is provided by the
> twitteR package in R would be appropriate. 
> 
> thanks
> 
> 
> 
> 
> 
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/getting-tweets-for-a-specified-handle-tp16085.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org