You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2023/08/30 22:28:10 UTC
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9561: [HUDI-6763] Optimize collect calls
nsivabalan commented on code in PR #9561:
URL: https://github.com/apache/hudi/pull/9561#discussion_r1310896302
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -286,24 +286,24 @@ protected String getCommitActionType() {
@Override
protected void setCommitMetadata(HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
- result.setCommitMetadata(Option.of(CommitUtils.buildMetadata(result.getWriteStatuses().map(WriteStatus::getStat).collectAsList(),
+ List<HoodieWriteStat> writeStats = result.getWriteStatuses().map(WriteStatus::getStat).collectAsList();
+ result.setWriteStats(writeStats);
+ result.setCommitMetadata(Option.of(CommitUtils.buildMetadata(writeStats,
result.getPartitionToReplaceFileIds(),
extraMetadata, operationType, getSchemaToStoreInCommit(), getCommitActionType())));
}
@Override
protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
Review Comment:
yeah, I don't see a need for it.
btw, in general, there are two code paths which we have been punting to refactor and fix it propery.
auto cmmit enabled and auto commit disabled.
this is auto commit enabled code path.
both deltastreamer and spark datasource writer goes thru auto commit disabled code path.
just fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org