You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/05/21 20:38:20 UTC
[GitHub] [hudi] vinothchandar commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival
vinothchandar commented on a change in pull request #2784:
URL: https://github.com/apache/hudi/pull/2784#discussion_r637218096
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java
##########
@@ -107,14 +119,25 @@ public static HoodieArchivedMetaEntry createMetaWrapper(HoodieInstant hoodieInst
return archivedMetaWrapper;
}
- public static HoodieArchivedMetaEntry createMetaWrapper(HoodieInstant hoodieInstant,
- HoodieCommitMetadata hoodieCommitMetadata) {
- HoodieArchivedMetaEntry archivedMetaWrapper = new HoodieArchivedMetaEntry();
- archivedMetaWrapper.setCommitTime(hoodieInstant.getTimestamp());
- archivedMetaWrapper.setActionState(hoodieInstant.getState().name());
- archivedMetaWrapper.setHoodieCommitMetadata(convertCommitMetadata(hoodieCommitMetadata));
- archivedMetaWrapper.setActionType(ActionType.commit.name());
- return archivedMetaWrapper;
+ public static Option<HoodieCommitMetadata> getInflightReplaceMetadata(HoodieTableMetaClient metaClient, HoodieInstant instant) throws IOException {
+ Option<byte[]> inflightContent = metaClient.getActiveTimeline().getInstantDetails(instant);
+ if (!inflightContent.isPresent() || inflightContent.get().length == 0) {
+ // inflight files can be empty in some certain cases, e.g. when users opt in clustering
Review comment:
@satishkotha what is the original reason for this?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/ReplaceArchivalHelper.java
##########
@@ -68,6 +68,11 @@
public static boolean deleteReplacedFileGroups(HoodieEngineContext context, HoodieTableMetaClient metaClient,
TableFileSystemView fileSystemView,
HoodieInstant instant, List<String> replacedPartitions) {
+ // There is no file id to be replaced in the very first replace commit file for insert overwrite operation
+ if (replacedPartitions.isEmpty()) {
+ LOG.warn("Found no partition files to replace");
Review comment:
may be add the instant time as well in the log message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org