You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Hironori Ogibayashi <og...@gmail.com> on 2016/09/27 10:17:02 UTC
FlinkKafkaConsumer and Kafka topic/partition change
Hello,
I want FlinkKafkaConsumer to follow changes in Kafka topic/partition change.
This means:
- When we add partitions to a topic, we want FlinkKafkaConsumer to
start reading added partitions.
- We want to specify topics by pattern (e.g accesslog.*), and want
FlinkKafkaConsumer to start reading new topics if they appeared after
starting job.
As long as reading source code and my experiment, FlinkKafkaConsumer
uses KafkaConsumer.assign() instead of subscribe(), so partitions are
assigned to each KafkaConsumer instance just once at job starting
time.
Is there any way to let FlinkKafkaConsumer follow topic/partition change?
Regards,
Hironori Ogibayashi
Re: FlinkKafkaConsumer and Kafka topic/partition change
Posted by Hironori Ogibayashi <og...@gmail.com>.
Gordon,
Thank you for your quick response!
I am looking forward to that feature. I will periodically check that JIRA.
I am also interested in the Robert's implementation because my use
current case is
system monitoring and scalability has higher priority than correctness.
Regards,
Hironori
2016-09-27 19:53 GMT+09:00 Tzu-Li (Gordon) Tai <tz...@apache.org>:
> Hi!
>
> This is definitely a planned feature for the Kafka connectors, there’s a
> JIRA exactly for this [1].
> We’re currently going through some blocking tasks to make this happen, I
> also hope to speed up things over there :)
>
> Your observation is correct that the Kaka consumer uses “assign()” instead
> of “subscribe()”.
> This is due to the fact that the partition-to-subtask assignment needs to be
> determinate in Flink
> for exactly-once semantics.
> If you’re not concerned about exactly-once and want to experiment around for
> now before [1] comes around,
> I believe Robert has recently implemented a Kafka consumer that uses
> “subscribe()”, so the Kafka
> topics can scale (looping in Robert to provide more info about this one).
>
> Best Regards,
> Gordon
>
> [1] https://issues.apache.org/jira/browse/FLINK-4022
>
>
> On September 27, 2016 at 6:17:06 PM, Hironori Ogibayashi
> (ogibayashi@gmail.com) wrote:
>
> Hello,
>
> I want FlinkKafkaConsumer to follow changes in Kafka topic/partition change.
> This means:
> - When we add partitions to a topic, we want FlinkKafkaConsumer to
> start reading added partitions.
> - We want to specify topics by pattern (e.g accesslog.*), and want
> FlinkKafkaConsumer to start reading new topics if they appeared after
> starting job.
>
> As long as reading source code and my experiment, FlinkKafkaConsumer
> uses KafkaConsumer.assign() instead of subscribe(), so partitions are
> assigned to each KafkaConsumer instance just once at job starting
> time.
>
> Is there any way to let FlinkKafkaConsumer follow topic/partition change?
>
> Regards,
> Hironori Ogibayashi
Re: FlinkKafkaConsumer and Kafka topic/partition change
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi!
This is definitely a planned feature for the Kafka connectors, there’s a JIRA exactly for this [1].
We’re currently going through some blocking tasks to make this happen, I also hope to speed up things over there :)
Your observation is correct that the Kaka consumer uses “assign()” instead of “subscribe()”.
This is due to the fact that the partition-to-subtask assignment needs to be determinate in Flink
for exactly-once semantics.
If you’re not concerned about exactly-once and want to experiment around for now before [1] comes around,
I believe Robert has recently implemented a Kafka consumer that uses “subscribe()”, so the Kafka
topics can scale (looping in Robert to provide more info about this one).
Best Regards,
Gordon
[1] https://issues.apache.org/jira/browse/FLINK-4022
On September 27, 2016 at 6:17:06 PM, Hironori Ogibayashi (ogibayashi@gmail.com) wrote:
Hello,
I want FlinkKafkaConsumer to follow changes in Kafka topic/partition change.
This means:
- When we add partitions to a topic, we want FlinkKafkaConsumer to
start reading added partitions.
- We want to specify topics by pattern (e.g accesslog.*), and want
FlinkKafkaConsumer to start reading new topics if they appeared after
starting job.
As long as reading source code and my experiment, FlinkKafkaConsumer
uses KafkaConsumer.assign() instead of subscribe(), so partitions are
assigned to each KafkaConsumer instance just once at job starting
time.
Is there any way to let FlinkKafkaConsumer follow topic/partition change?
Regards,
Hironori Ogibayashi