You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/20 02:47:18 UTC
[GitHub] [hudi] watermelon12138 commented on a diff in pull request #5199: [HUDI-3770] Delete the remaining temporary marker files after the syn…
watermelon12138 commented on code in PR #5199:
URL: https://github.com/apache/hudi/pull/5199#discussion_r853671852
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -372,6 +372,8 @@ public void completeCompaction(
} finally {
this.txnManager.endTransaction(Option.of(compactionInstant));
}
+ WriteMarkersFactory.get(config.getMarkersType(), table, compactionCommitTime)
+ .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
Review Comment:
Thank you for your review. Generally, Delete the remaining temporary marker files in postCommit stage. So, I think compaction progress also need to delete marker files after completing inflight compaction.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java:
##########
@@ -165,7 +165,7 @@ public Schema getTableAvroSchema(boolean includeMetadataFields) throws Exception
if (includeMetadataFields) {
return HoodieAvroUtils.addMetadataFields(schemaFromTableConfig.get(), hasOperationField);
} else {
- return schemaFromTableConfig.get();
+ return HoodieAvroUtils.removeMetadataFields(schemaFromTableConfig.get());
Review Comment:
Hello, When create schema already contains metadata fields but includeMetadataField is set to false, can avro schema that does not contain metadata fields be returned? Considering the above scenario, I modified the code on line 168.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java:
##########
@@ -165,7 +165,7 @@ public Schema getTableAvroSchema(boolean includeMetadataFields) throws Exception
if (includeMetadataFields) {
return HoodieAvroUtils.addMetadataFields(schemaFromTableConfig.get(), hasOperationField);
} else {
- return schemaFromTableConfig.get();
+ return HoodieAvroUtils.removeMetadataFields(schemaFromTableConfig.get());
Review Comment:
I answered your questions in the comments and hope you can review my code again.
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -372,6 +372,8 @@ public void completeCompaction(
} finally {
this.txnManager.endTransaction(Option.of(compactionInstant));
}
+ WriteMarkersFactory.get(config.getMarkersType(), table, compactionCommitTime)
+ .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
Review Comment:
I answered your questions in the comments and hope you can review my code again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org