You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2023/02/12 03:57:43 UTC
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7923: [HUDI-5769] Fixing deletion of metadata partitions based on explicit configs set by user
nsivabalan commented on code in PR #7923:
URL: https://github.com/apache/hudi/pull/7923#discussion_r1103730595
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -268,7 +268,7 @@ private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfi
.withAutoClean(false)
.withCleanerParallelism(parallelism)
.withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
- .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.EAGER)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY)
Review Comment:
temp fix to make tests run w/o issues. Once https://github.com/apache/hudi/pull/7921 is landed, will revert this.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2893,6 +2907,7 @@ protected void setDefaults() {
HoodieWriteCommitCallbackConfig.newBuilder().fromProperties(writeConfig.getProps()).build());
writeConfig.setDefaultOnCondition(!isPayloadConfigSet,
HoodiePayloadConfig.newBuilder().fromProperties(writeConfig.getProps()).build());
+ setMetadataConfigsSetByUser();
Review Comment:
we should call this before calling
```
writeConfig.setDefaultOnCondition(!isMetadataConfigSet,
HoodieMetadataConfig.newBuilder().withEngineType(engineType).fromProperties(writeConfig.getProps()).build());
```
if not, default value from code will be set in properties map and we can't deduce if something was explicitly set by the user or was it set as default by code.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##########
@@ -136,9 +135,9 @@ protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<
engineContext.setJobStatus(this.getClass().getName(), "Committing " + instantTime + " to metadata table " + metadataWriteConfig.getTableName());
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig)) {
// rollback partially failed writes if any.
- if (writeClient.rollbackFailedWrites()) {
+ /*if (writeClient.rollbackFailedWrites()) {
Review Comment:
temp fix to make tests run w/o issues. Once https://github.com/apache/hudi/pull/7921 is landed, will revert this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org