You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Hong Liang Teoh (Jira)" <ji...@apache.org> on 2025/05/28 07:16:00 UTC

[jira] [Updated] (FLINK-36296) Add incremental shard discovery for DDB Streams source

     [ https://issues.apache.org/jira/browse/FLINK-36296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Liang Teoh updated FLINK-36296:
------------------------------------
    Fix Version/s: aws-connector-5.1.0

> Add incremental shard discovery for DDB Streams source
> ------------------------------------------------------
>
>                 Key: FLINK-36296
>                 URL: https://issues.apache.org/jira/browse/FLINK-36296
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / DynamoDB
>            Reporter: Abhi Gupta
>            Assignee: Abhi Gupta
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: aws-connector-5.1.0
>
>
> DDB Streams source does a full shard discovery every minute, on large streams, this discovery can take more than 20 minutes to discover all shards, we can optimize this to do an incremental shard discovery periodically while also doing periodic shard discovery. This makes having the children shards faster and reduces the millisBehindLatest



--
This message was sent by Atlassian Jira
(v8.20.10#820010)