You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Abhi Gupta (Jira)" <ji...@apache.org> on 2025/04/04 08:15:00 UTC

[jira] [Commented] (FLINK-37273) Dynamodb Streams getRecords call adaptive delay

    [ https://issues.apache.org/jira/browse/FLINK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17940903#comment-17940903 ] 

Abhi Gupta commented on FLINK-37273:
------------------------------------

[~pranavwadekar] this has been fixed as part of this PR: https://github.com/apache/flink-connector-aws/pull/190

> Dynamodb Streams getRecords call adaptive delay
> -----------------------------------------------
>
>                 Key: FLINK-37273
>                 URL: https://issues.apache.org/jira/browse/FLINK-37273
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / DynamoDB
>    Affects Versions: 2.1.0
>            Reporter: Pranav Wadekar
>            Priority: Major
>             Fix For: 2.1.0
>
>
> I am trying to read the data from DynamoDBStreams which is introduced in Flink v18.0. It is working fine for large tables (10K RPM), and the efficiency is also good.
> However, the dynamodb stream cost for {{getRecords}} AWS requests. Although it's not a problem for big tables, for small tables even if the records are not committed Flink is continuously polling the records.
> With the above behaviour, we are getting an unnecessary cost of 130$ per day. Ideally, we need to scan the stream at specific intervals with some backoff time.
> I did not find anything regarding this.
> Can we please implement the adaptive delay if there is any idleness in incoming records? I can also contribute to the code.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)