You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/22 19:00:17 UTC

[GitHub] [hudi] yihua commented on a change in pull request #5090: [HUDI-3642] Handle NPE due to empty requested replacecommit metadata

yihua commented on a change in pull request #5090:
URL: https://github.com/apache/hudi/pull/5090#discussion_r832525843



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/ConcurrentOperation.java
##########
@@ -116,14 +116,27 @@ private void init(HoodieInstant instant) {
                 this.metadataWrapper.getMetadataFromTimeline().getHoodieReplaceCommitMetadata().getPartitionToWriteStats()).keySet();
             this.operationType = WriteOperationType.fromValue(this.metadataWrapper.getMetadataFromTimeline().getHoodieReplaceCommitMetadata().getOperationType());
           } else {
-            HoodieRequestedReplaceMetadata requestedReplaceMetadata = this.metadataWrapper.getMetadataFromTimeline().getHoodieRequestedReplaceMetadata();
-            this.mutatedFileIds = requestedReplaceMetadata
-                .getClusteringPlan().getInputGroups()
-                .stream()
-                .flatMap(ig -> ig.getSlices().stream())
-                .map(file -> file.getFileId())
-                .collect(Collectors.toSet());
-            this.operationType = WriteOperationType.CLUSTER;
+            // we need to different handling for requested and inflight replacecommit because
+            // for requested replacecommit, clustering will generate a plan and HoodieRequestedReplaceMetadata will not be empty, but insert_overwrite/insert_overwrite_table could have empty content
+            // for inflight replacecommit, clustering will have no content in metadata, but insert_overwrite/insert_overwrite_table will have some commit metadata
+            if (instant.isRequested()) {
+              HoodieRequestedReplaceMetadata requestedReplaceMetadata = this.metadataWrapper.getMetadataFromTimeline().getHoodieRequestedReplaceMetadata();
+              if (requestedReplaceMetadata != null) {
+                this.mutatedFileIds = requestedReplaceMetadata
+                    .getClusteringPlan().getInputGroups()
+                    .stream()
+                    .flatMap(ig -> ig.getSlices().stream())
+                    .map(file -> file.getFileId())
+                    .collect(Collectors.toSet());
+                this.operationType = WriteOperationType.CLUSTER;
+              }
+            } else {

Review comment:
       If the replacecommit from clustering is inflight, should we still read the requested HoodieRequestedReplaceMetadata?  Otherwise, we miss those stored File IDs.

##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/ConcurrentOperation.java
##########
@@ -116,14 +116,27 @@ private void init(HoodieInstant instant) {
                 this.metadataWrapper.getMetadataFromTimeline().getHoodieReplaceCommitMetadata().getPartitionToWriteStats()).keySet();
             this.operationType = WriteOperationType.fromValue(this.metadataWrapper.getMetadataFromTimeline().getHoodieReplaceCommitMetadata().getOperationType());
           } else {
-            HoodieRequestedReplaceMetadata requestedReplaceMetadata = this.metadataWrapper.getMetadataFromTimeline().getHoodieRequestedReplaceMetadata();
-            this.mutatedFileIds = requestedReplaceMetadata
-                .getClusteringPlan().getInputGroups()
-                .stream()
-                .flatMap(ig -> ig.getSlices().stream())
-                .map(file -> file.getFileId())
-                .collect(Collectors.toSet());
-            this.operationType = WriteOperationType.CLUSTER;
+            // we need to different handling for requested and inflight replacecommit because

Review comment:
       nit: `we need to different handling` -> `we need to have different handling`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org