[jira] [Resolved] (FLINK-36270) DDB Streams Connector performance issue due to splitsAvailableForAssignment function

["Hong Liang Teoh (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-36270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Liang Teoh resolved FLINK-36270.
-------------------------------------
    Resolution: Fixed

 merged commit [{{9b0d983}}|https://github.com/apache/flink-connector-aws/commit/9b0d98343a892be64d25d26064108aa27a255bd0] into   apache:main

> DDB Streams Connector performance issue due to splitsAvailableForAssignment function
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-36270
>                 URL: https://issues.apache.org/jira/browse/FLINK-36270
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / DynamoDB
>            Reporter: Abhi Gupta
>            Assignee: Abhi Gupta
>            Priority: Major
>              Labels: pull-request-available
>
> In DDB Streams connector, while testing we found out that when we are spending a lot of time in markAsFinished function because we are calling splitsAvailableForAssignment which is O(N), and given n shards can be marked as finished concurrently, the algorithm becomes O(n^2). Change the algo to assign only child shards when a parent is finished. We can start tracking child shards of a shard in SplitTracker



--
This message was sent by Atlassian Jira
(v8.20.10#820010)