Flink job is not stops reading or stops reading from certain kafka topic partitions

[vivekyadav01 <vi...@gmail.com>]

We have a Flink stream job that uses FlinkKafkaConsumer010. We keep the
kafkaconsumer operator's parallelism same as the number of partition in
kafka topic so that each partition is attached to one subtask.
Quite frequently the job stops reading from certain partitions. On
investigating under job metrics tab(committed offsets & current offset
metrics), few partitions are not associated with any subtask while few
partitions are associated with more than 1 subtasks(which seems wrong
behavior). I have debugged it locally and seen that initially the consumer
starts fine with uniform association with partitions but over the time some
partitions disassociate from the subtask.

Here is our kafka consumer config
-checkpoint disabled.
-enable.auto.commit true
-auto.commit.interval.ms 5 minutes
-request.timeout.ms 3 minutes

Kafka Version: 0.10.1.0
Flink version: 1.3.1

Has someone else faced this issue? Any help or pointers would be much
appreciated.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/