Question about the need of consumer groups from kafka

[Ricardo Dinis <di...@gmail.com>]

Hi,

If flink can manage kafka without using its own's mechanism (kafka
consumers) it still need the consumer group defined from kafka?

And why the examples is needed to use the plugin for shadding creating a
fat-jar?

Thanks

Re: Question about the need of consumer groups from kafka

[Till Rohrmann <tr...@apache.org>]

Hi Ricardo,

could you please give a bit more details what you mean with "not using its
own mechanism"? Flink's Kafka connector uses the Kafka consumer and
producer (to some extent) API to talk to Kafka. The consumer groups are a
central concept of Kafka and as such, the Flink Kafka connector has to know
it in order to consume from the right group.

Creating a fat-jar will make sure that the jar includes additional
dependencies such as the flink-kafka-connector which is not present on the
Flink cluster.

Cheers,
Till

On Fri, Feb 16, 2018 at 2:05 PM, Ricardo Dinis <di...@gmail.com> wrote:

> Hi,
>
> If flink can manage kafka without using its own's mechanism (kafka
> consumers) it still need the consumer group defined from kafka?
>
> And why the examples is needed to use the plugin for shadding creating a
> fat-jar?
>
> Thanks
>