[jira] [Created] (FLINK-37004) Implement Multithreaded SourceReader

["Hong Liang Teoh (Jira)" <ji...@apache.org>]

Hong Liang Teoh created FLINK-37004:
---------------------------------------

             Summary: Implement Multithreaded SourceReader
                 Key: FLINK-37004
                 URL: https://issues.apache.org/jira/browse/FLINK-37004
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kinesis
            Reporter: Hong Liang Teoh


Currently the *KinesisStreamsSourceReader* extends {*}SingleThreadMultiplexSourceReaderBase{*}, which has a single thread per subtask.

 

Since multiple shards can be assigned to the same subtask, this can limit the maximum throughput of the source, unless users increase the parallelism of the source.

 

However, this solution doesn't work when Kinesis streams have on-demand mode. Since the stream can shard up / down dynamically.

 

In this case, it might make sense to have a multithread source reader, and some form of setting to limit the maximum number of shards handled by each thread.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)