You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2023/08/30 22:28:10 UTC

[GitHub] [hudi] nsivabalan commented on a diff in pull request #9561: [HUDI-6763] Optimize collect calls

nsivabalan commented on code in PR #9561:
URL: https://github.com/apache/hudi/pull/9561#discussion_r1310896302


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -286,24 +286,24 @@ protected String getCommitActionType() {
 
   @Override
   protected void setCommitMetadata(HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
-    result.setCommitMetadata(Option.of(CommitUtils.buildMetadata(result.getWriteStatuses().map(WriteStatus::getStat).collectAsList(),
+    List<HoodieWriteStat> writeStats = result.getWriteStatuses().map(WriteStatus::getStat).collectAsList();
+    result.setWriteStats(writeStats);
+    result.setCommitMetadata(Option.of(CommitUtils.buildMetadata(writeStats,
         result.getPartitionToReplaceFileIds(),
         extraMetadata, operationType, getSchemaToStoreInCommit(), getCommitActionType())));
   }
 
   @Override
   protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result) {

Review Comment:
   yeah, I don't see a need for it. 
   btw, in general, there are two code paths which we have been punting to refactor and fix it propery. 
   auto cmmit enabled and auto commit disabled. 
   this is auto commit enabled code path. 
   both deltastreamer and spark datasource writer goes thru auto commit disabled code path. 
   just fyi



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org