You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gguptp (via GitHub)" <gi...@apache.org> on 2024/09/12 12:18:56 UTC

[PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

gguptp opened a new pull request, #164:
URL: https://github.com/apache/flink-connector-aws/pull/164

   
   
   <!--
   *Thank you for contributing to Apache Flink AWS Connectors - we are happy that you want to help us improve our Flink connectors. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   ## Contribution Checklist
   
   - The name of the pull request should correspond to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
   - Commits should be in the form of "[FLINK-XXXX][component] Title of the pull request", where [FLINK-XXXX] should be replaced by the actual issue number. 
       Generally, [component] should be the connector you are working on.
       For example: "[FLINK-XXXX][Connectors/Kinesis] XXXX" if you are working on the Kinesis connector or "[FLINK-XXXX][Connectors/AWS] XXXX" if you are working on components shared among all the connectors.
   - Each pull request should only have one JIRA issue.
   - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   -->
   
   ## Purpose of the change
   
   When a split is marked as finished, we are iterating over all splits and assigning available splits to readers. On a large enough stream with enough shards, we'd be spending a lot of time in finishedSplits because we are always calling `getSplitsForAssignment` function which is O(n). This leads to performance issues. Making this an O(1) algo to solve the issues
   
   ## Verifying this change
   
   Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
   
   *(Please pick either of the following options)*
   
   - This change is already covered by existing UTs
   
   
   ## Significant changes
   *(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)*
   - [ ] Dependencies have been added or upgraded
   - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`)
   - [ ] Serializers have been changed
   - [ ] New feature has been introduced
     - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "gguptp (via GitHub)" <gi...@apache.org>.
gguptp commented on PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#issuecomment-2348537801

   > Add some pictures for the performance bottleneck!
   > ### We see the checkpoints taking a long time to complete
   > 
   > ![image](https://private-user-images.githubusercontent.com/35062175/367238974-5ad6b70f-d46c-4cb1-beb3-1ea1dc52b78c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYyMjEwMzMsIm5iZiI6MTcyNjIyMDczMywicGF0aCI6Ii8zNTA2MjE3NS8zNjcyMzg5NzQtNWFkNmI3MGYtZDQ2Yy00Y2IxLWJlYjMtMWVhMWRjNTJiNzhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEzVDA5NDUzM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU3ZTZjNzYzMGRjZDgwY2NmNTk5NmNiZTkzYzViZTliYmJjMDhjNjQxNjE4YmIwODFhNTQzMWM4M2Y0NTliZDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.E4OvH7JjRuO_QntXNrJTQ4XYoUlM96TGN5-h0LGsmVU)
   > ### We see that the time is spent on "Start Delay"
   > 
   > ![image](https://private-user-images.githubusercontent.com/35062175/367239202-4a813286-7874-435a-845c-53d4658d43ac.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYyMjEwMzMsIm5iZiI6MTcyNjIyMDczMywicGF0aCI6Ii8zNTA2MjE3NS8zNjcyMzkyMDItNGE4MTMyODYtNzg3NC00MzVhLTg0NWMtNTNkNDY1OGQ0M2FjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEzVDA5NDUzM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThmMjYwMDgwMzllNTg3NTgzYjI3MTFjN2YzMzJmMjM2NGQ5NDUyNDMxMzgzMDI2MzA0YmM3M2MwNjRkNDA5MjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0._H29iWVbqPk7n7wk96_fH4u8J1msKVZJl3rVfdckb78)
   > ### We also see new checkpoints being stuck in an "unacknowledged" state
   > 
   > ![image](https://private-user-images.githubusercontent.com/35062175/367239332-1412df54-1584-4ac5-830a-635dccc9ce5d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYyMjEwMzMsIm5iZiI6MTcyNjIyMDczMywicGF0aCI6Ii8zNTA2MjE3NS8zNjcyMzkzMzItMTQxMmRmNTQtMTU4NC00YWM1LTgzMGEtNjM1ZGNjYzljZTVkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEzVDA5NDUzM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFkZDg0MzhiYWZiNWYzNGI0MzMzMmNhOWEwNWU3YjgzY2U4MDY0MDQ0ZDU2ZGQ4ZDFlMWMyODk3ZTg1OTlmNDAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.LSKIyCQejEsDjPBEmu9liWg0LhBPgiJozn08Csjtevk)
   
   Updated the PR overview


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "hlteoh37 (via GitHub)" <gi...@apache.org>.
hlteoh37 commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1758604229


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,24 +173,63 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> getUnassignedChildSplits(Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(
+                        splitId -> {
+                            if (!parentChildSplitMap.containsKey(splitId)) {
+                                LOG.warn(
+                                        "splitId: {} is not present in parent-child relationship map. "
+                                                + "This indicates that there might be some data loss in the application",
+                                        splitId);
+                            }
+                            return parentChildSplitMap.containsKey(splitId);
+                        })
+                .map(parentChildSplitMap::get)
+                .flatMap(Set::stream)
+                .filter(this::checkIfSplitCanBeAssigned)
+                .map(knownSplits::get)
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Tells whether a split can be assigned or not. Conditions which it checks:
+     *
+     * <p>- Split should not already be assigned.
+     *
+     * <p>- Split should not be already finished.
+     *
+     * <p>- The parent splits should either be finished or no longer be present in knownSplits.
+     */
+    private boolean checkIfSplitCanBeAssigned(String splitId) {
+        boolean splitIsNotAssigned = !isAssigned(splitId);
+        DynamoDbStreamsShardSplit split = knownSplits.get(splitId);
+        return splitIsNotAssigned
+                && !isFinished(splitId)
+                && verifyParentIsEitherFinishedOrCleanedUp(split);
+    }

Review Comment:
   Actaully can we make this take `Split` as argument? Otherwise we need to do null checks on the result of `knownSplits`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "gguptp (via GitHub)" <gi...@apache.org>.
gguptp commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1757215863


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/DynamoDbStreamsSourceEnumerator.java:
##########
@@ -201,8 +202,21 @@ private SplitGraphInconsistencyTracker trackSplitsAndResolveInconsistencies(
     }
 
     private void assignSplits() {
+        List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment =
+                splitTracker.splitsAvailableForAssignment();
+        assignSplitsInternal(splitsAvailableForAssignment);
+    }
+
+    private void assignSplits(Set<String> finishedSplitIds) {
+        List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment =
+                splitTracker.splitsAvailableForAssignment(finishedSplitIds);
+        assignSplitsInternal(splitsAvailableForAssignment);
+    }

Review Comment:
   yeah makes sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "gguptp (via GitHub)" <gi...@apache.org>.
gguptp commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1758619467


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,24 +173,63 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> getUnassignedChildSplits(Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(
+                        splitId -> {
+                            if (!parentChildSplitMap.containsKey(splitId)) {
+                                LOG.warn(
+                                        "splitId: {} is not present in parent-child relationship map. "
+                                                + "This indicates that there might be some data loss in the application",
+                                        splitId);
+                            }
+                            return parentChildSplitMap.containsKey(splitId);
+                        })
+                .map(parentChildSplitMap::get)
+                .flatMap(Set::stream)
+                .filter(this::checkIfSplitCanBeAssigned)
+                .map(knownSplits::get)
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Tells whether a split can be assigned or not. Conditions which it checks:
+     *
+     * <p>- Split should not already be assigned.
+     *
+     * <p>- Split should not be already finished.
+     *
+     * <p>- The parent splits should either be finished or no longer be present in knownSplits.
+     */
+    private boolean checkIfSplitCanBeAssigned(String splitId) {
+        boolean splitIsNotAssigned = !isAssigned(splitId);
+        DynamoDbStreamsShardSplit split = knownSplits.get(splitId);
+        return splitIsNotAssigned
+                && !isFinished(splitId)
+                && verifyParentIsEitherFinishedOrCleanedUp(split);
+    }

Review Comment:
   updated this to take split as input



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "gguptp (via GitHub)" <gi...@apache.org>.
gguptp commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1757216391


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -86,17 +87,27 @@ public SplitTracker(
      */
     public void addSplits(Collection<Shard> shardsToAdd) {
         Set<String> discoveredShardIds =
-                shardsToAdd.stream().map(Shard::shardId).collect(Collectors.toSet());
+                shardsToAdd.parallelStream().map(Shard::shardId).collect(Collectors.toSet());

Review Comment:
   lol you're right 😂 i made everything as parallelstream



##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,12 +170,35 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment(
+            Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(parentChildSplitMap::containsKey)

Review Comment:
   yeah will do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "gguptp (via GitHub)" <gi...@apache.org>.
gguptp commented on PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#issuecomment-2347138069

   Thanks for the review @hlteoh37! The review comments have been addressed now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "hlteoh37 (via GitHub)" <gi...@apache.org>.
hlteoh37 commented on PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#issuecomment-2348506206

   Let's add some pictures for the performance bottleneck we found!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "hlteoh37 (via GitHub)" <gi...@apache.org>.
hlteoh37 commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1756874586


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -86,17 +87,27 @@ public SplitTracker(
      */
     public void addSplits(Collection<Shard> shardsToAdd) {
         Set<String> discoveredShardIds =
-                shardsToAdd.stream().map(Shard::shardId).collect(Collectors.toSet());
+                shardsToAdd.parallelStream().map(Shard::shardId).collect(Collectors.toSet());

Review Comment:
   nit: Using `parallelStream` for this is a little bit much 🤣 



##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/DynamoDbStreamsSourceEnumerator.java:
##########
@@ -201,8 +202,21 @@ private SplitGraphInconsistencyTracker trackSplitsAndResolveInconsistencies(
     }
 
     private void assignSplits() {
+        List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment =
+                splitTracker.splitsAvailableForAssignment();
+        assignSplitsInternal(splitsAvailableForAssignment);
+    }
+
+    private void assignSplits(Set<String> finishedSplitIds) {
+        List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment =
+                splitTracker.splitsAvailableForAssignment(finishedSplitIds);
+        assignSplitsInternal(splitsAvailableForAssignment);
+    }

Review Comment:
   can we name these methods something like `assignChildSplits()` and `assignAllAvailableSplits()`



##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,12 +170,35 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment(
+            Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(parentChildSplitMap::containsKey)

Review Comment:
   can we log a warn here ?
   
   If the `parentChildSplitMap` doesn't have the key, this indicates there has been some issue and potential loss of data



##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/DynamoDbStreamsSourceEnumerator.java:
##########
@@ -201,8 +202,21 @@ private SplitGraphInconsistencyTracker trackSplitsAndResolveInconsistencies(
     }
 
     private void assignSplits() {
+        List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment =
+                splitTracker.splitsAvailableForAssignment();
+        assignSplitsInternal(splitsAvailableForAssignment);
+    }
+
+    private void assignSplits(Set<String> finishedSplitIds) {
+        List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment =
+                splitTracker.splitsAvailableForAssignment(finishedSplitIds);
+        assignSplitsInternal(splitsAvailableForAssignment);
+    }

Review Comment:
   `assignSplitsInternal` can then be `assignSplits`



##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,12 +170,35 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment(

Review Comment:
   nit: can we name something meaningful like `getUnassignedChildSplits`



##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,12 +170,35 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment(
+            Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(parentChildSplitMap::containsKey)
+                .filter(
+                        splitId -> {
+                            boolean splitIsNotAssigned = !isAssigned(splitId);
+                            DynamoDbStreamsShardSplit split = knownSplits.get(splitId);
+                            return splitIsNotAssigned
+                                    && !isFinished(splitId)
+                                    && verifyParentIsEitherFinishedOrCleanedUp(split);

Review Comment:
   can we extract this to a private method? Prevent code duplication for this essential logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "hlteoh37 (via GitHub)" <gi...@apache.org>.
hlteoh37 commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1756892805


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,12 +170,35 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment(

Review Comment:
   Let's also add some unit tests for these public methods!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "gguptp (via GitHub)" <gi...@apache.org>.
gguptp commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1757216895


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,12 +170,35 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment(

Review Comment:
   will do



##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,12 +170,35 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> splitsAvailableForAssignment(
+            Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(parentChildSplitMap::containsKey)
+                .filter(
+                        splitId -> {
+                            boolean splitIsNotAssigned = !isAssigned(splitId);
+                            DynamoDbStreamsShardSplit split = knownSplits.get(splitId);
+                            return splitIsNotAssigned
+                                    && !isFinished(splitId)
+                                    && verifyParentIsEitherFinishedOrCleanedUp(split);

Review Comment:
   will do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "hlteoh37 (via GitHub)" <gi...@apache.org>.
hlteoh37 commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1758601446


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,24 +173,63 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> getUnassignedChildSplits(Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(
+                        splitId -> {
+                            if (!parentChildSplitMap.containsKey(splitId)) {
+                                LOG.warn(
+                                        "splitId: {} is not present in parent-child relationship map. "
+                                                + "This indicates that there might be some data loss in the application",
+                                        splitId);
+                            }
+                            return parentChildSplitMap.containsKey(splitId);
+                        })
+                .map(parentChildSplitMap::get)
+                .flatMap(Set::stream)
+                .filter(this::checkIfSplitCanBeAssigned)
+                .map(knownSplits::get)
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Tells whether a split can be assigned or not. Conditions which it checks:
+     *
+     * <p>- Split should not already be assigned.
+     *
+     * <p>- Split should not be already finished.
+     *
+     * <p>- The parent splits should either be finished or no longer be present in knownSplits.
+     */
+    private boolean checkIfSplitCanBeAssigned(String splitId) {
+        boolean splitIsNotAssigned = !isAssigned(splitId);
+        DynamoDbStreamsShardSplit split = knownSplits.get(splitId);
+        return splitIsNotAssigned
+                && !isFinished(splitId)
+                && verifyParentIsEitherFinishedOrCleanedUp(split);
+    }

Review Comment:
   very nice :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "gguptp (via GitHub)" <gi...@apache.org>.
gguptp commented on code in PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164#discussion_r1758606097


##########
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/enumerator/tracker/SplitTracker.java:
##########
@@ -159,24 +173,63 @@ public boolean isAssigned(String splitId) {
         return assignedSplits.contains(splitId);
     }
 
+    /**
+     * Function to get children splits available for given parent ids. This will ensure not to
+     * iterate all the values in knownSplits so saving compute
+     */
+    public List<DynamoDbStreamsShardSplit> getUnassignedChildSplits(Set<String> parentSplitIds) {
+        return parentSplitIds
+                .parallelStream()
+                .filter(
+                        splitId -> {
+                            if (!parentChildSplitMap.containsKey(splitId)) {
+                                LOG.warn(
+                                        "splitId: {} is not present in parent-child relationship map. "
+                                                + "This indicates that there might be some data loss in the application",
+                                        splitId);
+                            }
+                            return parentChildSplitMap.containsKey(splitId);
+                        })
+                .map(parentChildSplitMap::get)
+                .flatMap(Set::stream)
+                .filter(this::checkIfSplitCanBeAssigned)
+                .map(knownSplits::get)
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Tells whether a split can be assigned or not. Conditions which it checks:
+     *
+     * <p>- Split should not already be assigned.
+     *
+     * <p>- Split should not be already finished.
+     *
+     * <p>- The parent splits should either be finished or no longer be present in knownSplits.
+     */
+    private boolean checkIfSplitCanBeAssigned(String splitId) {
+        boolean splitIsNotAssigned = !isAssigned(splitId);
+        DynamoDbStreamsShardSplit split = knownSplits.get(splitId);
+        return splitIsNotAssigned
+                && !isFinished(splitId)
+                && verifyParentIsEitherFinishedOrCleanedUp(split);
+    }

Review Comment:
   Updating



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-36270][Connectors/DynamoDB] Assign only children splits of a split when it is marked as finished [flink-connector-aws]

Posted by "hlteoh37 (via GitHub)" <gi...@apache.org>.
hlteoh37 merged PR #164:
URL: https://github.com/apache/flink-connector-aws/pull/164


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org