You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by yi...@apache.org on 2024/02/22 23:34:42 UTC
(hudi) branch branch-0.x created (now 1ca8cc7dc55)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a change to branch branch-0.x
in repository https://gitbox.apache.org/repos/asf/hudi.git
at 1ca8cc7dc55 [HUDI-7207] Sequentially delete complete instant files in archival to prevent inconsistency during data reads (#10711)
This branch includes the following new commits:
new 1ca8cc7dc55 [HUDI-7207] Sequentially delete complete instant files in archival to prevent inconsistency during data reads (#10711)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(hudi) 01/01: [HUDI-7207] Sequentially delete complete instant files in archival to prevent inconsistency during data reads (#10711)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch branch-0.x
in repository https://gitbox.apache.org/repos/asf/hudi.git
commit 1ca8cc7dc55de0ae7fcaa3b30939abfca152cbb2
Author: majian <47...@users.noreply.github.com>
AuthorDate: Thu Feb 22 10:51:48 2024 +0800
[HUDI-7207] Sequentially delete complete instant files in archival to prevent inconsistency during data reads (#10711)
---
.../java/org/apache/hudi/client/HoodieTimelineArchiver.java | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java
index d4abfa82d59..718f8ad2c46 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java
@@ -594,11 +594,13 @@ public class HoodieTimelineArchiver<T extends HoodieAvroPayload, I, K, O> {
);
}
if (!completedInstants.isEmpty()) {
- context.foreach(
- completedInstants,
- instant -> activeTimeline.deleteInstantFileIfExists(instant),
- Math.min(completedInstants.size(), config.getArchiveDeleteParallelism())
- );
+ // Due to the concurrency between deleting completed instants and reading data,
+ // there may be hole in the timeline, which can lead to errors when reading data.
+ // Therefore, the concurrency of deleting completed instants is temporarily disabled,
+ // and instants are deleted in ascending order to prevent the occurrence of such holes.
+ // See HUDI-7207 and #10325.
+ completedInstants.stream()
+ .forEach(instant -> activeTimeline.deleteInstantFileIfExists(instant));
}
return true;