You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Mason Chen <ma...@gmail.com> on 2022/03/02 22:16:58 UTC

KafkaPartitionSplitReader handleSplitsChanges

Hi all,

I noticed in the javadocs that SplitReaders should not have a blocking
handleSplitsChanges implementation:
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/splitreader/SplitReader.java#L55

However, for KafkaPartitionSplitReader, the implementation is blocking due
to potential IO to retrieve committed offsets. I just wanted to double
check: is this javadoc accurate or should the KafkaPartitionSplitReader
implementation be optimized?

Best,
Mason

Re: KafkaPartitionSplitReader handleSplitsChanges

Posted by Mason Chen <ma...@gmail.com>.

Or is the motivation that resolving the committed/latest offsets is an
infrequent event (and only for bounded read) so the optimization is not
worth it?

On Wed, Mar 2, 2022 at 2:16 PM Mason Chen <ma...@gmail.com> wrote:

> Hi all,
>
> I noticed in the javadocs that SplitReaders should not have a blocking
> handleSplitsChanges implementation:
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/splitreader/SplitReader.java#L55
>
> However, for KafkaPartitionSplitReader, the implementation is blocking due
> to potential IO to retrieve committed offsets. I just wanted to double
> check: is this javadoc accurate or should the KafkaPartitionSplitReader
> implementation be optimized?
>
> Best,
> Mason
>
>