You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/29 01:22:37 UTC
[GitHub] [hudi] alexeykudinkin opened a new pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
alexeykudinkin opened a new pull request #4716:
URL: https://github.com/apache/hudi/pull/4716
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
This change is addressing issues in regards to Metadata Table observing ingesting duplicated records leading to it persisting incorrect file-sizes for the files referred to in those records.
There are multiple issues that were leading to that:
1. [HUDI-3322] Incorrect Rollback Plan generation: Rollback Plan generated for MOR tables was overly expansively listing all log-files with the latest base-instant as the ones that have been affected by the rollback, leading to invalid MT records being ingested referring to those.
2. [HUDI-3343] Metadata Table including Uncommitted Log Files during Bootstrap: Since MT is bootstrapped at the end of the commit operation execution (after FS activity, but before committing to the timeline), it was actually incorrectly ingesting some files that were part of the intermediate state of the operation being committed.
This change will unblock Stack of PRs based off #4556
## Brief change log
- Cleaned up Rollback request generation flow
- Cleaned up superfluous `writtenLogFilesSize` payload
- Fixed RollbackUtils to always leverage appropriate instant when fetching metadata
- Propagate `HoodieWriteStat` w/in `ListingBasedRollbackRequest`
- Fixed RollbackPlan generation to properly refeference files bearing blocks
- Fixed superfluous `RollbackPlan` generation
- Fixing MT to be bootstapped _before_ carrying out any FS operations, to make sure it's not ingesting intermediate step upon bootstrapping
- Fix MarkerBasedRollbackStrategy to only ref the latest log-file as the one blocks have been appended to
## Verify this pull request
This pull request is already covered by existing tests, such as *(please describe tests)*.
This PR fixing following tests that started to fail after changes in #4556:
`TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore`
`TestHoodieSparkMergeOnReadTableRollback#testRollbackWithDeltaAndCompactionCommit`
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r796077526
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -442,8 +443,11 @@ private void updateTableMetadata(HoodieTable<T, JavaRDD<HoodieRecord<T>>, JavaRD
metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance())
.run(HoodieTableVersion.current(), instantTime);
metaClient.reloadActiveTimeline();
- initializeMetadataTable(Option.of(instantTime));
}
+ // Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
+ // if it didn't exist before
+ // See https://issues.apache.org/jira/browse/HUDI-3343 for more details
+ initializeMetadataTable(Option.of(instantTime));
Review comment:
So the thinking is like following:
1. MT should be aware of all the (relevant) files in the directory
2. MT should not be bootstrapped in the middle of the operation (to avoid it observing intermediate state of the operation that actually triggers bootstrapping)
In case of multi-writers -- there're multiple possible scenarios
1. Writers running concurrently: this should be handled by taking the mutex, to make sure MT is bootstrapped once
2. Writer bootstrapping MT running after previous Writer failed to commit (leaving intermediate state in the directory): in that case i'd still argue it's better to MT to be aware of this files than not, so that it's inline with FS-based implementation. Whether these files will be part of the output should be decided by the record-reader, not MT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024799819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024801043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589)
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r795095166
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -442,8 +443,11 @@ private void updateTableMetadata(HoodieTable<T, JavaRDD<HoodieRecord<T>>, JavaRD
metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance())
.run(HoodieTableVersion.current(), instantTime);
metaClient.reloadActiveTimeline();
- initializeMetadataTable(Option.of(instantTime));
}
+ // Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
+ // if it didn't exist before
+ // See https://issues.apache.org/jira/browse/HUDI-3343 for more details
+ initializeMetadataTable(Option.of(instantTime));
Review comment:
I get the fix here. But lets say there is a failed commit just before this current writer. Wouldn't the MDT bootstrap also include the files from the failed commit ?
Or are we gonna rely on MDT bootstrap will not happen if there is any pending operation in data table. And so unless the rollback of the failed commit completes, we may not trigger MDT bootstrap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024798480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1026246434
>Here is my understanding of the fix.
We fix the rollback plan generated in ListingBasedStrategy to just include the log files with full sizes.
For Marker bases strategy, we set file sizes to -1, but fixed the right set of log files to be included in rollback plan.
> So, we still have an issue w/ how we reconcile or merge multiple metadata records.
For eg:
Rec1: file1 delta size 100 (commit1)
Rec2: file1 deltasize 200 (commit2)
Rec3: file1 full size 350 (rollback)
> when we merge all these 3 records from metadata table, whats the final resolved record look like ?
When we do a rollback the value in the plan (carrying the mapping of path to size) is only used to determine whether we should _append Rollback Block_ or _delete files_. After we actually appended the Rollback Block, we now only modify the record related to the file we've appended the block to (previously we would also update all the log-files from the `writtenLogFileSizesMap`)
> If you don't mind, can we add tests for the fix in this patch. The tests should fail if not the fix and should pass w/ the fix.
There are already tests covering this, which were failing in #4556 (which is how i come to fix this issues).
The reason why these were not failing is simply b/c we're not using MT table when we're reading the data back using Hive's InputFormats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r794979942
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -442,8 +443,11 @@ private void updateTableMetadata(HoodieTable<T, JavaRDD<HoodieRecord<T>>, JavaRD
metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance())
.run(HoodieTableVersion.current(), instantTime);
metaClient.reloadActiveTimeline();
- initializeMetadataTable(Option.of(instantTime));
}
+ // Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
+ // if it didn't exist before
+ // See https://issues.apache.org/jira/browse/HUDI-3343 for more details
+ initializeMetadataTable(Option.of(instantTime));
Review comment:
This is addressing HUDI-3343
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
##########
@@ -264,17 +265,6 @@ private static void processRollbackMetadata(HoodieActiveTimeline metadataTableTi
partitionToAppendedFiles.get(partition).merge(new Path(path).getName(), size, fileMergeFn);
});
}
-
- if (pm.getWrittenLogFiles() != null && !pm.getWrittenLogFiles().isEmpty()) {
Review comment:
This is addressing HUDI-3322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r796077526
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -442,8 +443,11 @@ private void updateTableMetadata(HoodieTable<T, JavaRDD<HoodieRecord<T>>, JavaRD
metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance())
.run(HoodieTableVersion.current(), instantTime);
metaClient.reloadActiveTimeline();
- initializeMetadataTable(Option.of(instantTime));
}
+ // Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
+ // if it didn't exist before
+ // See https://issues.apache.org/jira/browse/HUDI-3343 for more details
+ initializeMetadataTable(Option.of(instantTime));
Review comment:
So the thinking is like following:
1. MT should be aware of all the (relevant) files in the directory
2. MT should not be bootstrapped in the middle of the operation (to avoid it seeing intermediate state)
In case of multi-writers -- there're multiple possible scenarios
1. Writers running concurrently: this should be handled by taking the mutex, to make sure MT is bootstrapped once
2. Writer bootstrapping MT running after previous Writer failed to commit (leaving intermediate state in the directory): in that case i'd still argue it's better to MT to be aware of this files than not, so that it's inline with FS-based implementation. Whether these files will be part of the output should be decided by the record-reader, not MT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r797155269
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
+ HoodieFileFormat.HOODIE_LOG.getFileExtension(), baseCommitTime).get();
+
+ // NOTE: Marker's don't carry information about the cumulative size of the blocks that have been appended,
+ // therefore we simply stub this value.
+ Map<String, Long> logFilesWithBlocsToRollback =
Review comment:
based on offline discussion, this will case an issue in storage schemes like hdfs where rollback block could be added to existing log file. will take it up as a follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024837778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1028357992
cool. thanks!
btw, can you remind me something. with this patch, whats the size we set in rollback metadata for log files where rollback block command is added?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1026235895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
}, {
"hash" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
* 550d8e3230b42b18bd7713245696f8de87d2e458 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r797153919
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -442,8 +443,11 @@ private void updateTableMetadata(HoodieTable<T, JavaRDD<HoodieRecord<T>>, JavaRD
metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance())
.run(HoodieTableVersion.current(), instantTime);
metaClient.reloadActiveTimeline();
- initializeMetadataTable(Option.of(instantTime));
}
+ // Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
+ // if it didn't exist before
+ // See https://issues.apache.org/jira/browse/HUDI-3343 for more details
+ initializeMetadataTable(Option.of(instantTime));
Review comment:
based on offline discussion, we are good here. MDT intitalization may not kick in if there are any pending operations in data table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024812581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589)
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024812581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589)
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r796085280
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
Review comment:
Correct. Without fixing the marker there's no way for us to know which file is bearing the actual block.
> incase of multi-writer use-cases, this may not be true unfortunately. one writer could fail mid way. there could be other concurrent writers. and after a timeout, some other writer will trigger the rollback of the previously failed commit. So, by this time, there could be more log blocks/files added.
Not sure i follow. Can you elaborate? Who in that scenario would be rolling back the failed commit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1026235895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
}, {
"hash" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
* 550d8e3230b42b18bd7713245696f8de87d2e458 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r796088457
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
+ HoodieFileFormat.HOODIE_LOG.getFileExtension(), baseCommitTime).get();
+
+ // NOTE: Marker's don't carry information about the cumulative size of the blocks that have been appended,
+ // therefore we simply stub this value.
+ Map<String, Long> logFilesWithBlocsToRollback =
Review comment:
There's no issues w/ `HoodieMetadataFileInfo` -- it always stores file-size. The problem that i was bringing up earlier is that we manicure `FileStatus` w/ delta file-size during append.
But since deltas are only used for log-file appends, and it isn't actually faulty here, i don't think there's need for us to drop everything and try to fix it right away, since its not an issue at the moment (and unlikely to become) and its scope is actually very limited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024798480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024832837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4716:
URL: https://github.com/apache/hudi/pull/4716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1026272781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
}, {
"hash" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5630",
"triggerID" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
* 550d8e3230b42b18bd7713245696f8de87d2e458 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5630)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1026272781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
}, {
"hash" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5630",
"triggerID" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
* 550d8e3230b42b18bd7713245696f8de87d2e458 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5630)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024848858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024801043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589)
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024848858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024837464
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r796086591
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
+ HoodieFileFormat.HOODIE_LOG.getFileExtension(), baseCommitTime).get();
+
+ // NOTE: Marker's don't carry information about the cumulative size of the blocks that have been appended,
+ // therefore we simply stub this value.
+ Map<String, Long> logFilesWithBlocsToRollback =
Review comment:
Intentionally putting invalid data in there:
1. These file-sizes aren't actually used currently.
2. There's no way we could now at this point without scanning the latest log how many bytes were actually written.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1026312765
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
}, {
"hash" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5630",
"triggerID" : "550d8e3230b42b18bd7713245696f8de87d2e458",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 550d8e3230b42b18bd7713245696f8de87d2e458 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5630)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024837778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598",
"triggerID" : "1024837464",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5598)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024832837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
}, {
"hash" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593",
"triggerID" : "bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* bb05ceb91b62d86e5f06ae6f00a28eb57c17aa6d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5593)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r795093095
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
+ HoodieFileFormat.HOODIE_LOG.getFileExtension(), baseCommitTime).get();
+
+ // NOTE: Marker's don't carry information about the cumulative size of the blocks that have been appended,
+ // therefore we simply stub this value.
+ Map<String, Long> logFilesWithBlocsToRollback =
Review comment:
I was also thinking if we should fix the way we store the file size in HoodieMetadataPayload.
```
HoodieMetadataFileInfo(long size, boolean isFullSize, boolean isDelete)
```
With CommitMetadata, these will represent delta sizes. and during rollback/restore, these will represent full sizes. We can combine multiple metadata records based on whether its delta or full size.
##########
File path: hudi-common/src/main/avro/HoodieRollbackMetadata.avsc
##########
@@ -38,14 +38,6 @@
"type": "long",
"doc": "Size of this file in bytes"
}
- }], "default":null },
Review comment:
this may not be backwards compatible while reading rollback metadata written w/ 0.10.0 or previous versions.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
Review comment:
probably we should fix marker file even for log files. If not, we have to read all log files and then determine based on the instant time in the header to know exactly which log files were touched by a given commit
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
Review comment:
incase of multi-writer use-cases, this may not be true unfortunately. one writer could fail mid way. there could be other concurrent writers. and after a timeout, some other writer will trigger the rollback of the previously failed commit. So, by this time, there could be more log blocks/files added.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/MarkerBasedRollbackStrategy.java
##########
@@ -90,42 +86,41 @@ public MarkerBasedRollbackStrategy(HoodieTable<?, ?, ?, ?> table, HoodieEngineCo
Collections.singletonList(fullDeletePath.toString()),
Collections.emptyMap());
case APPEND:
+ // NOTE: This marker file-path does NOT correspond to a log-file, but rather is a phony
+ // path serving as a "container" for the following components:
+ // - Base file's file-id
+ // - Base file's commit instant
+ // - Partition path
return getRollbackRequestForAppend(WriteMarkers.stripMarkerSuffix(markerFilePath));
default:
throw new HoodieRollbackException("Unknown marker type, during rollback of " + instantToRollback);
}
- }, parallelism).stream().collect(Collectors.toList());
+ }, parallelism);
} catch (Exception e) {
throw new HoodieRollbackException("Error rolling back using marker files written for " + instantToRollback, e);
}
}
- protected HoodieRollbackRequest getRollbackRequestForAppend(String appendBaseFilePath) throws IOException {
- Path baseFilePathForAppend = new Path(basePath, appendBaseFilePath);
+ protected HoodieRollbackRequest getRollbackRequestForAppend(String markerFilePath) throws IOException {
+ Path baseFilePathForAppend = new Path(basePath, markerFilePath);
String fileId = FSUtils.getFileIdFromFilePath(baseFilePathForAppend);
String baseCommitTime = FSUtils.getCommitTime(baseFilePathForAppend.getName());
- String partitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), new Path(basePath, appendBaseFilePath).getParent());
- Map<FileStatus, Long> writtenLogFileSizeMap = getWrittenLogFileSizeMap(partitionPath, baseCommitTime, fileId);
- Map<String, Long> writtenLogFileStrSizeMap = new HashMap<>();
- for (Map.Entry<FileStatus, Long> entry : writtenLogFileSizeMap.entrySet()) {
- writtenLogFileStrSizeMap.put(entry.getKey().getPath().toString(), entry.getValue());
- }
- return new HoodieRollbackRequest(partitionPath, fileId, baseCommitTime, Collections.emptyList(), writtenLogFileStrSizeMap);
+ String relativePartitionPath = FSUtils.getRelativePartitionPath(new Path(basePath), baseFilePathForAppend.getParent());
+ Path partitionPath = FSUtils.getPartitionPath(config.getBasePath(), relativePartitionPath);
+
+ // NOTE: Since we're rolling back incomplete Delta Commit, it only could have appended its
+ // block to the latest log-file
+ // TODO(HUDI-1517) use provided marker-file's path instead
+ HoodieLogFile latestLogFile = FSUtils.getLatestLogFile(table.getMetaClient().getFs(), partitionPath, fileId,
+ HoodieFileFormat.HOODIE_LOG.getFileExtension(), baseCommitTime).get();
+
+ // NOTE: Marker's don't carry information about the cumulative size of the blocks that have been appended,
+ // therefore we simply stub this value.
+ Map<String, Long> logFilesWithBlocsToRollback =
Review comment:
but how do we reconcile this -1 while merging multiple metadata records?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024989828
If you don't mind, can we add tests for the fix in this patch. The tests should fail if not the fix and should pass w/ the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r797157620
##########
File path: hudi-common/src/main/avro/HoodieRollbackMetadata.avsc
##########
@@ -38,14 +38,6 @@
"type": "long",
"doc": "Size of this file in bytes"
}
- }], "default":null },
Review comment:
hmmm, interesting. Can you try this out explicitly. write an avro using master branch. and then try to read it using this branch. or you can just try it out using a stand alone java main class too, which ever works.
just wanted to ensure we are good here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1028323837
@nsivabalan here's the test:
```
@Test
void testAvroBWC() throws IOException {
Writer w = new Writer("foo", 0xDEAD);
byte[] buf = HoodieAvroUtils.avroToBytes(w);
GenericRecord r = HoodieAvroUtils.bytesToAvro(buf, Writer.SCHEMA$, Reader.SCHEMA$);
assertEquals(w.getKey(), r.get("key"));
}
```
For the following schema:
```
{
"namespace": "org.apache.hudi.avro.model",
"type": "record",
"name": "TestAvroRecord",
"fields": [
{
"name": "writer",
"type": ["null", {
"type": "record",
"name": "Writer",
"fields": [
{
"name": "key",
"type": ["null", "string"]
},
{
"name": "value",
"type": ["null", "int"]
}
]
}]
},
{
"name": "reader",
"type": ["null", {
"name": "Reader",
"type": "record",
"fields": [
{
"name": "key",
"type": ["null", "string"]
}
]
}]
}
]
}
```
Working as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#discussion_r796210413
##########
File path: hudi-common/src/main/avro/HoodieRollbackMetadata.avsc
##########
@@ -38,14 +38,6 @@
"type": "long",
"doc": "Size of this file in bytes"
}
- }], "default":null },
Review comment:
This is an excerpt from Avro's docs:
https://avro.apache.org/docs/1.7.7/spec.html#Schema+Resolution
<img width="1339" alt="Screen Shot 2022-01-31 at 5 42 02 PM" src="https://user-images.githubusercontent.com/428277/151900978-a62c84d6-3bb7-4723-bdfd-c8d937689c34.png">
##########
File path: hudi-common/src/main/avro/HoodieRollbackMetadata.avsc
##########
@@ -38,14 +38,6 @@
"type": "long",
"doc": "Size of this file in bytes"
}
- }], "default":null },
Review comment:
I've researched this a bit since my intuition was telling that fields removal is BWC change in Avro (since Avro also supports _forward compatibility_, when you essentially read data with old schema that was produced with a new one, which is essentially the same case)
And seems like deletion is BWC in Avro:
https://docs.confluent.io/platform/current/schema-registry/avro.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024799152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024799152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4716: [HUDI-3322][HUDI-3343] Fixing Metadata Table Records Duplication Issues
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4716:
URL: https://github.com/apache/hudi/pull/4716#issuecomment-1024799819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62b022ec6642cef208e7fb370f95ed0ed1c5ba83",
"triggerType" : "PUSH"
}, {
"hash" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589",
"triggerID" : "059ba5b6bc6ec995bf66538da8fa63bac494fd69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62b022ec6642cef208e7fb370f95ed0ed1c5ba83 UNKNOWN
* 059ba5b6bc6ec995bf66538da8fa63bac494fd69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5589)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org