[jira] [Created] (FLINK-26890) [Kinesis][Consumer] DynamoDB consumer error consuming partitions close to retention

["Danny Cranmer (Jira)" <ji...@apache.org>]

Danny Cranmer created FLINK-26890:
-------------------------------------

             Summary: [Kinesis][Consumer] DynamoDB consumer error consuming partitions close to retention
                 Key: FLINK-26890
                 URL: https://issues.apache.org/jira/browse/FLINK-26890
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kinesis
    Affects Versions: 1.15.0
            Reporter: Danny Cranmer
             Fix For: 1.15.0, 1.16.0, 1.13.7, 1.14.5


*Background*

The Amazon Kinesis Data Streams consumer supports consuming from Amazon DynamoDB via the [DynamoDB Streams Kinesis Adapter|https://github.com/awslabs/dynamodb-streams-kinesis-adapter]. 

*Problem*

We have seen instances of consumer throwing {{ResouceNotFoundException}} when attempting to invoke {{GetShardIterator}}.

{code}
com.amazonaws.services.kinesis.model.ResourceNotFoundException: Requested resource not found: Shard does not exist 
{code}

According to the DynamoDB team, the {{DescribeStream}} call may return shard IDs that are no longer valid, and this exception needs to be handled by the client. 

*Solution*

Modify the DynamoDB consumer to treat {{ResourceNotFoundException}} as a shard closed signal.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)