You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Hong Liang Teoh (Jira)" <ji...@apache.org> on 2025/05/28 07:13:00 UTC

[jira] [Updated] (FLINK-37273) Dynamodb Streams getRecords call adaptive delay

     [ https://issues.apache.org/jira/browse/FLINK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Liang Teoh updated FLINK-37273:
------------------------------------
    Fix Version/s: aws-connector-5.1.0

> Dynamodb Streams getRecords call adaptive delay
> -----------------------------------------------
>
>                 Key: FLINK-37273
>                 URL: https://issues.apache.org/jira/browse/FLINK-37273
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / DynamoDB
>    Affects Versions: 2.1.0
>            Reporter: Pranav Wadekar
>            Priority: Major
>             Fix For: aws-connector-5.1.0, 2.1.0
>
>
> I am trying to read the data from DynamoDBStreams which is introduced in Flink v18.0. It is working fine for large tables (10K RPM), and the efficiency is also good.
> However, the dynamodb stream cost for {{getRecords}} AWS requests. Although it's not a problem for big tables, for small tables even if the records are not committed Flink is continuously polling the records.
> With the above behaviour, we are getting an unnecessary cost of 130$ per day. Ideally, we need to scan the stream at specific intervals with some backoff time.
> I did not find anything regarding this.
> Can we please implement the adaptive delay if there is any idleness in incoming records? I can also contribute to the code.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)