[jira] [Created] (FLINK-6503) Refactor KinesisDataFetcher to separate concerns for shard discovery

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

Tzu-Li (Gordon) Tai created FLINK-6503:
------------------------------------------

             Summary: Refactor KinesisDataFetcher to separate concerns for shard discovery
                 Key: FLINK-6503
                 URL: https://issues.apache.org/jira/browse/FLINK-6503
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
            Reporter: Tzu-Li (Gordon) Tai


Currently, shard discovery is done within the `KinesisDataFetcher`'s loop. I propose to extract that from the fetcher and encapsulate that in a separate class to separate concerns.

As can be seen in https://github.com/apache/flink/pull/3001/files#diff-0b702244caed9a73b7b3e1dc8a9cbaebR461, one downside if we do not do this refactor is apparent when adding rescalability of the Kinesis consumer. That required shards lookup operations prioir to running the fetcher, which resulted in the need to expose shard fetching logic external to the `KinesisDataFetcher`. This should be properly done be separating concerns of shard discovery from the fetcher.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)