consume kafka stream with flink

[Tanguy Racinet <ta...@gmail.com>]

Hello,

I am currently trying to develop am algorithm mining frequent item sets over a data stream.
I am using kafka to generate the stream, however I cannot manage to link Flink to Kafka.

The code presented here is working but only using Flink version 0.9.1
https://github.com/dataArtisans/kafka-example/blob/master/src/main/java/com/dataartisans/ReadFromKafka.java
It doesn’t work for me when using FLink 10-SNAPSHOT or the latest release.

I tried several other things and was hoping someone could help me understand the differences between kafkaSource, FlinkKafkaConsummer, FlinkKafkaConsummer081 and FlinkKafkaConsummer082.

Which one is the latest API ? Is there another way to consume Kafka steam with Flink that is even more up to date ? Or should I use FlinkKafkaConsummer082 and stay with Flink version 9 ?

Thank you for your help.
Regards,
Tanguy R.

Re: consume kafka stream with flink

[Robert Metzger <rm...@apache.org>]

Hi Tanguy,

I would recommend to refer to the documentation of the specific Flink
version you are using.

This is the documentation for 1.0-SNAPSHOT:
https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/connectors/kafka.html
and this is the doc for 0.10.x:
https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#apache-kafka

I would recommend to use Flink 0.10.x and the FlinkKafkaConsumer082. With
Flink 1.0 the consumer will be renamed to FlinkKafkaConsumer08.

Let me know if you have further questions.


Regards,
Robert


On Fri, Feb 12, 2016 at 12:07 PM, Tanguy Racinet <ta...@gmail.com>
wrote:

> Hello,
>
> I am currently trying to develop am algorithm mining frequent item sets
> over a data stream.
> I am using kafka to generate the stream, however I cannot manage to link
> Flink to Kafka.
>
> The code presented here is working but only using Flink version 0.9.1
>
> https://github.com/dataArtisans/kafka-example/blob/master/src/main/java/com/dataartisans/ReadFromKafka.java
> It doesn’t work for me when using FLink 10-SNAPSHOT or the latest release.
>
> I tried several other things and was hoping someone could help me
> understand the differences between kafkaSource, FlinkKafkaConsummer,
> FlinkKafkaConsummer081 and FlinkKafkaConsummer082.
>
> Which one is the latest API ? Is there another way to consume Kafka steam
> with Flink that is even more up to date ? Or should I use
> FlinkKafkaConsummer082 and stay with Flink version 9 ?
>
> Thank you for your help.
> Regards,
> Tanguy R.
>