You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Weijie Guo (Jira)" <ji...@apache.org> on 2025/03/18 09:49:02 UTC

[jira] [Updated] (FLINK-37273) Dynamodb Streams getRecords call adaptive delay

     [ https://issues.apache.org/jira/browse/FLINK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Weijie Guo updated FLINK-37273:
-------------------------------
    Affects Version/s: 2.1.0
                           (was: aws-connector-5.0.0)

> Dynamodb Streams getRecords call adaptive delay
> -----------------------------------------------
>
>                 Key: FLINK-37273
>                 URL: https://issues.apache.org/jira/browse/FLINK-37273
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / DynamoDB
>    Affects Versions: 2.1.0
>            Reporter: Pranav Wadekar
>            Priority: Major
>             Fix For: 2.0.0
>
>
> I am trying to read the data from DynamoDBStreams which is introduced in Flink v18.0. It is working fine for large tables (10K RPM), and the efficiency is also good.
> However, the dynamodb stream cost for {{getRecords}} AWS requests. Although it's not a problem for big tables, for small tables even if the records are not committed Flink is continuously polling the records.
> With the above behaviour, we are getting an unnecessary cost of 130$ per day. Ideally, we need to scan the stream at specific intervals with some backoff time.
> I did not find anything regarding this.
> Can we please implement the adaptive delay if there is any idleness in incoming records? I can also contribute to the code.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)