[jira] [Created] (FLINK-36270) DDB Streams Connector performance issue due to splitsAvailableForAssignment function

["Abhi Gupta (Jira)" <ji...@apache.org>]

Abhi Gupta created FLINK-36270:
----------------------------------

             Summary: DDB Streams Connector performance issue due to splitsAvailableForAssignment function
                 Key: FLINK-36270
                 URL: https://issues.apache.org/jira/browse/FLINK-36270
             Project: Flink
          Issue Type: Bug
          Components: Connectors / DynamoDB
            Reporter: Abhi Gupta


In DDB Streams connector, while testing we found out that when we are spending a lot of time in markAsFinished function because we are calling splitsAvailableForAssignment which is O(n), and given n shards can be marked as finished concurrently, the algorithm becomes O(n^2). Change the algo to assign only child shards when a parent is finished. We can start tracking child shards of a shard in SplitTracker



--
This message was sent by Atlassian Jira
(v8.20.10#820010)