Tweets Stream input for Flink

[Mustafa Elbehery <el...@gmail.com>]

Hi All,

We have created a Java model for the Twitter API, and are looking for a way
to integrate it with Flink.

*aalexandrov <https://dimatub.slack.com/team/aalexandrov>**[8:44 PM]*We
have a sequence of tweets serialized as JSON objects and need to parse them
accordingly in order to convert them to our Java model.

*aalexandrov <https://dimatub.slack.com/team/aalexandrov>**[8:45 PM]*Is
there an InputReader that can parse a JSON array into a sequence of strings
(each string representing a single array element)?
Regards.

-- 
Mustafa Elbehery
EIT ICT Labs Master School <http://www.masterschool.eitictlabs.eu/home/>
+49(0)16 2819 2148
skype: mustafaelbehery87

Re: Tweets Stream input for Flink

[Márton Balassi <ba...@gmail.com>]

Dear Mustafa,

Flink has built-in support for stream processing providing connectors for a
number of input sources, the Twitter Streaming API amongst others. We
provide an example of reading from the publicly available tweets and
parsing the flat JSON objects to build our internal DataStream
representation containing the relevant fields of the JSON.

Documentation:
http://flink.incubator.apache.org/docs/0.7-incubating/streaming_guide.html#twitter-streaming-api

Relevant example code:
https://github.com/apache/incubator-flink/blob/master/flink-addons/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterLocal.java

Operator parsing the JSON into a DataStream:
https://github.com/apache/incubator-flink/blob/master/flink-addons/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/json/JSONParseFlatMap.java

Cheers,

Marton

On Tue, Oct 28, 2014 at 8:47 PM, Mustafa Elbehery <elbeherymustafa@gmail.com
> wrote:

> Hi All,
>
> We have created a Java model for the Twitter API, and are looking for a
> way to integrate it with Flink.
>
> *aalexandrov <https://dimatub.slack.com/team/aalexandrov>**[8:44 PM]*We
> have a sequence of tweets serialized as JSON objects and need to parse them
> accordingly in order to convert them to our Java model.
>
> *aalexandrov <https://dimatub.slack.com/team/aalexandrov>**[8:45 PM]*Is
> there an InputReader that can parse a JSON array into a sequence of strings
> (each string representing a single array element)?
> Regards.
>
> --
> Mustafa Elbehery
> EIT ICT Labs Master School <http://www.masterschool.eitictlabs.eu/home/>
> +49(0)16 2819 2148
> skype: mustafaelbehery87
>
>