You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Arian Rohani (Jira)" <ji...@apache.org> on 2022/03/30 15:17:00 UTC
[jira] [Created] (FLINK-26933) FlinkKinesisConsumer incorrectly determines shards as newly discovered when tested against Kinesalite when consuming DynamoDB streams
Arian Rohani created FLINK-26933:
------------------------------------
Summary: FlinkKinesisConsumer incorrectly determines shards as newly discovered when tested against Kinesalite when consuming DynamoDB streams
Key: FLINK-26933
URL: https://issues.apache.org/jira/browse/FLINK-26933
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Reporter: Arian Rohani
This ticket is related to https://issues.apache.org/jira/browse/FLINK-5075
The kinesalite (mock implementation of kinesis) does not take exclusiveShardStartId into account when performing a DescribeStream operation. This causes the FlinkKinesisConsumer to resubscribe to already subscribed shards and reconsume the records when consuming a DynamoDB stream. A fix was implemented for the Kinesis stream inside of the listShards(...) method, but this logic (see [here|https://github.com/apache/flink/blame/b2ca390d478aa855eb0f2028d0ed965803a98af1/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java#L568]) is not executed when connecting to a DynamoDB stream.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)