UDP support in Streaming API

[Janani Chakkaradhari <ja...@gmail.com>]

​Hi,

Does Flink's streaming api has support for
​ reading streams of data via Java UDP (DatagramSocket)? If so kindly
advise me in which release of Flink I can find it.

Thanks,
Janani

Re: UDP support in Streaming API

[Márton Balassi <mb...@apache.org>]

Dear Janani,

Apache Kafka as a source is supported by our system, check out the
documentation for details. [1]

You can use UDP as a source if you wish, just bear in mind the standard
disadvantages of it: the possibility of losing messages and that you will
have to manually deal with the serialization of datagrams. For adding your
custom source all you need to implement is an instance of the
SourceFunction interface. [2]

[1]
http://ci.apache.org/projects/flink/flink-docs-master/streaming_guide.html#apache-kafka
[2]
http://ci.apache.org/projects/flink/flink-docs-master/streaming_guide.html#sources

Best,

Marton

On Thu, Mar 26, 2015 at 1:15 PM, Janani Chakkaradhari <
janani.cse05@gmail.com> wrote:

> Hi Stephan,
>
> Yes, you are right. I will try writing a custom data source as you
> mentioned. Also, I still need to check the possibility of our system to use
> Apache Kafka as a broker. Could it be possible for you to point out here
> the downsides of using UDP as a source for streaming data?
>
> Thanks,
> Janani
>
> On Wed, Mar 25, 2015 at 9:44 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > Hi Janani!
> >
> > Do I understand you correctly in that you want a Flink stream source that
> > receives UDP datagrams and turns them into Flink DataStream?
> >
> > Such a thing is not in there, yet. The interface to define custom data
> > sources is rather simple, though, it should be possible to add something
> > like this.
> >
> > Two things are not clear to me however:
> >
> > 1) Where are the datagrams sent to? That would need to be the point where
> > the data source runs.
> >
> > 2) You need a way of turning the datagrams (which are just bytes) into
> > records. Would that be hard-wired in your case?
> >
> > So, while probably possible, I would guess a UDP source has quite a few
> > downsides over a message queue as a source. Using something like Kafka to
> > communicate the source data is easier and better recoverable.
> >
> > Does your setup allow to put the data into Kafka and having Flink read
> the
> > streams from Kafka?
> >
> > Greetings,
> > Stephan
> >  Am 25.03.2015 19:20 schrieb "Janani Chakkaradhari" <
> > janani.cse05@gmail.com
> > >:
> >
> > > ​Hi,
> > >
> > > Does Flink's streaming api has support for
> > > ​ reading streams of data via Java UDP (DatagramSocket)? If so kindly
> > > advise me in which release of Flink I can find it.
> > >
> > > Thanks,
> > > Janani
> > >
> >
>

Re: UDP support in Streaming API

[Janani Chakkaradhari <ja...@gmail.com>]

Hi Stephan,

Yes, you are right. I will try writing a custom data source as you
mentioned. Also, I still need to check the possibility of our system to use
Apache Kafka as a broker. Could it be possible for you to point out here
the downsides of using UDP as a source for streaming data?

Thanks,
Janani

On Wed, Mar 25, 2015 at 9:44 PM, Stephan Ewen <se...@apache.org> wrote:

> Hi Janani!
>
> Do I understand you correctly in that you want a Flink stream source that
> receives UDP datagrams and turns them into Flink DataStream?
>
> Such a thing is not in there, yet. The interface to define custom data
> sources is rather simple, though, it should be possible to add something
> like this.
>
> Two things are not clear to me however:
>
> 1) Where are the datagrams sent to? That would need to be the point where
> the data source runs.
>
> 2) You need a way of turning the datagrams (which are just bytes) into
> records. Would that be hard-wired in your case?
>
> So, while probably possible, I would guess a UDP source has quite a few
> downsides over a message queue as a source. Using something like Kafka to
> communicate the source data is easier and better recoverable.
>
> Does your setup allow to put the data into Kafka and having Flink read the
> streams from Kafka?
>
> Greetings,
> Stephan
>  Am 25.03.2015 19:20 schrieb "Janani Chakkaradhari" <
> janani.cse05@gmail.com
> >:
>
> > ​Hi,
> >
> > Does Flink's streaming api has support for
> > ​ reading streams of data via Java UDP (DatagramSocket)? If so kindly
> > advise me in which release of Flink I can find it.
> >
> > Thanks,
> > Janani
> >
>

Re: UDP support in Streaming API

[Stephan Ewen <se...@apache.org>]

Hi Janani!

Do I understand you correctly in that you want a Flink stream source that
receives UDP datagrams and turns them into Flink DataStream?

Such a thing is not in there, yet. The interface to define custom data
sources is rather simple, though, it should be possible to add something
like this.

Two things are not clear to me however:

1) Where are the datagrams sent to? That would need to be the point where
the data source runs.

2) You need a way of turning the datagrams (which are just bytes) into
records. Would that be hard-wired in your case?

So, while probably possible, I would guess a UDP source has quite a few
downsides over a message queue as a source. Using something like Kafka to
communicate the source data is easier and better recoverable.

Does your setup allow to put the data into Kafka and having Flink read the
streams from Kafka?

Greetings,
Stephan
 Am 25.03.2015 19:20 schrieb "Janani Chakkaradhari" <janani.cse05@gmail.com
>:

> ​Hi,
>
> Does Flink's streaming api has support for
> ​ reading streams of data via Java UDP (DatagramSocket)? If so kindly
> advise me in which release of Flink I can find it.
>
> Thanks,
> Janani
>

Re: UDP support in Streaming API

[Sarvani <sa...@gmail.com>]

Hi Janani,

Have you implemented UDP source using the SourceFunction interface?



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/UDP-support-in-Streaming-API-tp4715p12182.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.