You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "suryaprasanna (via GitHub)" <gi...@apache.org> on 2023/06/17 22:46:43 UTC
[GitHub] [hudi] suryaprasanna opened a new pull request, #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
suryaprasanna opened a new pull request, #9007:
URL: https://github.com/apache/hudi/pull/9007
### Change Logs
This change fixes the transitions from requested to inflight, requested to complete, requested to rollback and inflight to rollback in IncrementalTimelineSyncFileSystemView
There are some cases, where incremental file sync breaks. Especially if there are rollbacks on replacecommits or logcompaction commits.
For example, if current writer saw pending replacecommit or logcompaction commit, and other writers rollbacked those commits, now if the current writer tries to incrementally sync those rollback commits it would cause failure.
### Impact
No public facing API but the impact is medium
### Risk level (write none, low medium or high below)
High
### Documentation Update
All the required documentation is provided within the code.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1613834275
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189)
* f154ee335eb307e2bcffd895cfd95bfb1f417a1e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264810398
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -170,7 +170,7 @@ public interface HoodieTimeline extends Serializable {
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants();
Review Comment:
We are not changing the definition, the instants cover two parts: completed commits and also all the rewrite instants. For rewrite instants, it could include a clustering commit also an insert overwrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1635861329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 78b7acc447a6cdadccf1b0ca57e1cc634233c879 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233)
* 26b3151e371774b3e99324bd9c305157fcde5789 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636586011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 578401a45b7c5ffbd9360de3bd3e18c362b4b2b5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592)
* ea6504e78fbb1c776687d3632c5875e74070cebd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264287885
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -170,7 +170,7 @@ public interface HoodieTimeline extends Serializable {
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants();
Review Comment:
we should name it as "filterCompletedRewriteInstants"
rewrite refers to commit, delta commits and replace commits.
completed refers to state.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -53,6 +53,7 @@ public static Comparator<String> getReverseCommitTimeComparator() {
/**
* Timeline, based on which all getter work.
+ * This should be a write timeline that contains either completed instants or pending compaction instants.
Review Comment:
can we rename the variable then.
completedWriteAndCompactionTimeline
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -146,17 +189,22 @@ public static class TimelineDiffResult {
private final List<HoodieInstant> newlySeenInstants;
private final List<HoodieInstant> finishedCompactionInstants;
- private final List<HoodieInstant> finishedOrRemovedLogCompactionInstants;
+ // Completed instants will have true as the value where as instants removed due to rollback will have false as value.
+ private final List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants;
+ // Completed instants will have true as the value where as instants removed due to rollback will have false as value.
+ private final List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants;
private final boolean canSyncIncrementally;
public static final TimelineDiffResult UNSAFE_SYNC_RESULT =
- new TimelineDiffResult(null, null, null, false);
+ new TimelineDiffResult(null, null, null, null, false);
public TimelineDiffResult(List<HoodieInstant> newlySeenInstants, List<HoodieInstant> finishedCompactionInstants,
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants, boolean canSyncIncrementally) {
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants,
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants, boolean canSyncIncrementally) {
this.newlySeenInstants = newlySeenInstants;
this.finishedCompactionInstants = finishedCompactionInstants;
this.finishedOrRemovedLogCompactionInstants = finishedOrRemovedLogCompactionInstants;
+ this.finishedOrRemovedReplaceCommitInstants = finishedOrRemovedReplaceCommitInstants;
Review Comment:
if your ans to my previous comment is, only clustering, should we rename all these variables accordingly. it might confuse down the line.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
Review Comment:
btw, we are not interested in any replace commit right. only the clustering ones? can you point me to the code where we filter only for clustering replace commits?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -134,14 +134,20 @@ protected void init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActi
* @param visibleActiveTimeline Visible Active Timeline
*/
protected void refreshTimeline(HoodieTimeline visibleActiveTimeline) {
- this.visibleCommitsAndCompactionTimeline = visibleActiveTimeline.getWriteTimeline();
+ this.visibleCommitsAndCompactionTimeline = visibleActiveTimeline
+ .getWriteTimeline()
+ .filterCompletedAndCompactionInstants();
}
/**
* Adds the provided statuses into the file system view, and also caches it inside this object.
*/
public List<HoodieFileGroup> addFilesToView(FileStatus[] statuses) {
- HoodieTimer timer = HoodieTimer.start();
+ HoodieTimer timer = new HoodieTimer().startTimer();
+ // Timeline determines whether a file slice is committed or not. Timeline object is attached to each HoodieFileGroup.
+ // Using this timeline object HoodieFileGroup determines whether a fileSlice is committed or not.
+ // visibleCommitsAndCompactionTimeline contains entire write timeline with pending commits, so filtering it
Review Comment:
this comment does not make sense.
in L 137, we already filter for completed and compaction timeline. So, why add this comment here. and infact we don't do any filtering in this code snippet (L146 to 150). So its confusing.
or did you meant to add this comment in L137 ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
Review Comment:
can we simplifly the map as below
```
.map(instantPair -> Pair.of(instantPair.getKey(), instantPair.getValue() != null))
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1246,6 +1301,19 @@ Stream<FileSlice> fetchAllFileSlices(String partitionPath) {
.flatMap(HoodieFileGroup::getAllFileSlices);
}
+ /**
+ * Default implementation for fetching all file-slices for a partition-path.
+ *
+ * @param partitionPath Partition path
+ * @return file-slice stream
+ */
+ Stream<FileSlice> fetchAllFileSlices(String partitionPath, boolean includePending) {
Review Comment:
same comment as above. can we try to re use code as much as possible.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java:
##########
@@ -137,7 +137,7 @@ public void close() {
*/
private static RocksDbBasedFileSystemView createRocksDBBasedFileSystemView(SerializableConfiguration conf,
FileSystemViewStorageConfig viewConf, HoodieTableMetaClient metaClient) {
- HoodieTimeline timeline = metaClient.getActiveTimeline().filterCompletedAndCompactionInstants();
+ HoodieTimeline timeline = metaClient.getActiveTimeline();
Review Comment:
can you tell me how did you chase and fix all these?
found the caller stack trace for AbstractHoodieFileSystemview.init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline) and fixed all of them ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -170,15 +176,25 @@ public List<HoodieFileGroup> addFilesToView(FileStatus[] statuses) {
/**
* Build FileGroups from passed in file-status.
+ * TODO: Note here timeline is completedAndCompactionTimeline.
Review Comment:
can you rename the "timeline" to "completedWriteAndCompactionTimeline" so that all the call stack knows what this timeline is about.
in fact, we can chase and fix all places.
for eg, we can rename in the constructor of HoodieFileGroup as well.
will avoid mistakes by someone in future who might make changes only in HoodieFileGroup for instance.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -787,6 +803,21 @@ public final Stream<FileSlice> getLatestFileSlices(String partitionStr) {
}
}
+ @Override
+ public final Stream<FileSlice> getLatestFileSlices(String partitionStr, boolean includePending) {
Review Comment:
why can't we make one of this call the other.
getLatestFileSlices(String partitionStr)
I see opportunity for code re-use here as well
```
/**
* Default implementation for fetching latest file-slices for a partition path.
*/
Stream<FileSlice> fetchLatestFileSlices(String partitionPath) {
return fetchAllStoredFileGroups(partitionPath).map(HoodieFileGroup::getLatestFileSlice).filter(Option::isPresent)
.map(Option::get);
}
/**
* Default implementation for fetching latest file-slices for a partition path.
*/
Stream<FileSlice> fetchLatestFileSlices(String partitionPath, boolean includePending) {
return fetchAllStoredFileGroups(partitionPath).map(fileGroup -> fileGroup.getLatestFileSlice(includePending)).filter(Option::isPresent)
.map(Option::get);
}
```
We can just call fileGroup.getLatestFileSlice(false) in other method.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -181,17 +204,24 @@ private void removePendingCompactionInstant(HoodieInstant instant) throws IOExce
}
/**
- * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit,
- * so you no longer need to track them as pending.
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
Review Comment:
minor.
`Remove pending log compaction instant`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/RocksDbBasedFileSystemView.java:
##########
@@ -544,10 +574,13 @@ protected Option<HoodieInstant> getReplaceInstant(final HoodieFileGroupId fileGr
}
private Stream<HoodieFileGroup> getFileGroups(Stream<FileSlice> sliceStream) {
+ // HoodieFileGroup using timeline to determine whether a file slice is committed or not so passing
+ // completedAndCompactionTimeline object while creating HoodieFileGroup.
+ HoodieTimeline completedAndCompactionTimeline = getVisibleCommitsAndCompactionTimeline();
Review Comment:
may be we shd name this
completedWriteAndCompactionTimeline
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -325,6 +325,15 @@ private void registerFileSlicesAPI() {
writeValueAsString(ctx, dtos);
}, true));
+ app.get(RemoteHoodieTableFileSystemView.LATEST_PARTITION_SLICES_INCLUDE_PENDING_URL, new ViewHandler(ctx -> {
+ metricsRegistry.add("LATEST_PARTITION_SLICES", 1);
Review Comment:
shouldn't we fix this to LATEST_PARTITION_SLICES_INCLUDE_PENDING
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -327,11 +357,43 @@ private void addRollbackInstant(HoodieTimeline timeline, HoodieInstant instant)
LOG.info("Done Syncing rollback instant (" + instant + ")");
}
+ /**
+ * Add pending replace file groups
+ * @param pendingClusteringInstant pending instant
+ * @throws IOException
+ */
+ private void addPendingClusteringInstant(HoodieInstant pendingClusteringInstant) throws IOException {
+ LOG.info("Syncing pending clustering instant (" + pendingClusteringInstant + ")");
+ Stream<Pair<HoodieFileGroupId, HoodieInstant>> fileGroupStreamToInstant =
+ ClusteringUtils.getFileGroupEntriesFromClusteringInstant(pendingClusteringInstant, metaClient);
+ addFileGroupsInPendingClustering(fileGroupStreamToInstant);
+ }
+
+ /**
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
Review Comment:
minor. fix the docs. it talks about compaction while the actual method is for clustering
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java:
##########
@@ -145,6 +150,17 @@ Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath, String max
*/
Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime);
+ /**
+ * Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the
Review Comment:
minor.
"before or on an ..."
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
+ HoodieTimeline newTimeline) {
+ Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
+
+ return oldTimeline.filterPendingReplaceTimeline().getInstantsAsStream().map(instant -> {
+ if (newTimelineInstants.contains(instant)) {
Review Comment:
guess this is what danny was proposing
```
private static List<Pair<HoodieInstant, HoodieInstant>> getPendingLogCompactionTransitions(HoodieTimeline oldTimeline,
Function<HoodieTimeline, HoodieTimeline> getPendingActionTimelineFromOld,
HoodieTimeline newTimeline,
String completedAction, String pendingAction) {
Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
return getPendingActionTimelineFromOld.apply(oldTimeline).getInstantsAsStream().map(instant -> {
if (newTimelineInstants.contains(instant)) {
return Pair.of(instant, instant);
} else {
HoodieInstant completedInstant =
new HoodieInstant(State.COMPLETED, completedAction, instant.getTimestamp());
if (newTimelineInstants.contains(completedInstant)) {
return Pair.of(instant, completedInstant);
}
HoodieInstant inflightInstant =
new HoodieInstant(State.INFLIGHT, pendingAction, instant.getTimestamp());
if (newTimelineInstants.contains(inflightInstant)) {
return Pair.of(instant, inflightInstant);
}
return Pair.<HoodieInstant, HoodieInstant>of(instant, null);
}
}).collect(Collectors.toList());
}
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
+ HoodieTimeline newTimeline) {
+ Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
+
+ return oldTimeline.filterPendingReplaceTimeline().getInstantsAsStream().map(instant -> {
+ if (newTimelineInstants.contains(instant)) {
Review Comment:
surya, next time when you push an update, you can add this.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -953,6 +984,18 @@ public final Stream<FileSlice> getAllFileSlices(String partitionStr) {
}
}
+ @Override
+ public final Stream<FileSlice> getAllFileSlices(String partitionStr, boolean includePending) {
Review Comment:
same comment as above
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -159,7 +177,12 @@ private void runIncrementalSync(HoodieTimeline timeline, TimelineDiffResult diff
} else if (instant.getAction().equals(HoodieTimeline.ROLLBACK_ACTION)) {
addRollbackInstant(timeline, instant);
} else if (instant.getAction().equals(HoodieTimeline.REPLACE_COMMIT_ACTION)) {
- addReplaceInstant(timeline, instant);
+ boolean isClusteringCommit = ClusteringUtils.isClusteringCommit(metaClient, instant);
+ if (!instant.isCompleted() && isClusteringCommit) {
+ addPendingClusteringInstant(instant);
+ } else {
+ addReplaceInstant(timeline, instant);
Review Comment:
in L163, we already filter only for clustering replace commits. So, wouldn't this mean, addCompletedClusteringInstant.
or am I mis-understanding something ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -181,17 +204,24 @@ private void removePendingCompactionInstant(HoodieInstant instant) throws IOExce
}
/**
- * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit,
- * so you no longer need to track them as pending.
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
+ * operation is failed, so it is no longer need to be tracked as pending.
*
* @param instant Log Compaction Instant to be removed
+ * @param isSuccessful true if log compaction operation is successful, false if the operation is failed and rollbacked.
*/
- private void removePendingLogCompactionInstant(HoodieInstant instant) throws IOException {
- LOG.info("Removing completed log compaction instant (" + instant + ")");
- HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
- removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
- .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
- CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ private void removePendingLogCompactionInstant(HoodieInstant instant, boolean isSuccessful) throws IOException {
+ if (isSuccessful) {
+ LOG.info("Removing completed log compaction instant (" + instant + ")");
+ HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
+ removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
+ .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
+ CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ } else {
+ LOG.info("Removing failed log compaction instant (" + instant + ")");
+ removePendingLogCompactionOperations(instant.getTimestamp());
Review Comment:
why I don't see this if else with removePendingCompactionInstant. can you help me understand please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264638709
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -787,6 +803,21 @@ public final Stream<FileSlice> getLatestFileSlices(String partitionStr) {
}
}
+ @Override
+ public final Stream<FileSlice> getLatestFileSlices(String partitionStr, boolean includePending) {
Review Comment:
As @danny0405 pointed out in one of the comments, some of these are test APIs. If I make one of them dependent on the other, people may start using them. Not sure what path to follow, so for now made these APIs as separate, let me know what your thoughts are on this.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java:
##########
@@ -137,7 +137,7 @@ public void close() {
*/
private static RocksDbBasedFileSystemView createRocksDBBasedFileSystemView(SerializableConfiguration conf,
FileSystemViewStorageConfig viewConf, HoodieTableMetaClient metaClient) {
- HoodieTimeline timeline = metaClient.getActiveTimeline().filterCompletedAndCompactionInstants();
+ HoodieTimeline timeline = metaClient.getActiveTimeline();
Review Comment:
With the current change I went though the caller's caller and fixed at all the places.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -53,6 +53,7 @@ public static Comparator<String> getReverseCommitTimeComparator() {
/**
* Timeline, based on which all getter work.
+ * This should be a write timeline that contains either completed instants or pending compaction instants.
Review Comment:
As agreed offline will be using fileSystemViewTimeline variable, since we are using getFileSystemViewTimeline API.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -448,14 +523,19 @@ protected void applyDeltaFileSlicesToPartitionView(String partition, List<Hoodie
throw new IllegalStateException("Unknown diff apply mode=" + mode);
}
- HoodieTimeline timeline = deltaFileGroups.stream().map(df -> df.getTimeline()).findAny().get();
+ // Here timeline is fetched from the existing file groups so it is automatically completedWriteAndCompactionTimeline
+ HoodieTimeline completedWriteAndCompactionTimeline = deltaFileGroups.stream().map(df -> df.getTimeline()).findAny().get();
List<HoodieFileGroup> fgs =
- buildFileGroups(viewDataFiles.values().stream(), viewLogFiles.values().stream(), timeline, true);
+ buildFileGroups(viewDataFiles.values().stream(), viewLogFiles.values().stream(), completedWriteAndCompactionTimeline, true);
storePartitionView(partition, fgs);
}
@Override
public HoodieTimeline getTimeline() {
return visibleActiveTimeline;
}
+
+ public boolean isLastIncrementalSyncSuccessful() {
Review Comment:
Good idea, marked it.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
I agree. Core writer flow does not use them, some are used by query engines. They are able to get it working in a weird way, like some times they pass active timeline to the file system view and sometimes they pass write timeline etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636979388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 7ffcac1db2479936bdee038951fab00071597d26 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264718695
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -144,44 +172,54 @@ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingCompactionTran
*/
public static class TimelineDiffResult {
- private final List<HoodieInstant> newlySeenInstants;
+ private final List<HoodieInstant> newlySeenCompletedAndRewriteInstants;
private final List<HoodieInstant> finishedCompactionInstants;
Review Comment:
Ideally compaction plans are not removed, since after creating a compaction plan even though they are in inflight state new data is written to new file slices. So, if we were to remove compaction, then we might need to remove latest files. We could add APIs to remove them, but it is bit risky.
Also compaction plans are immutable plans so writers cannot delete them, only developers can remove them manually by using restore operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264702031
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java:
##########
@@ -833,11 +832,11 @@ private Pair<List<HoodieRecord>, List<HoodieRecord>> twoUpsertCommitDataWithTwoP
SyncableFileSystemView fsView = getFileSystemViewWithUnCommittedSlices(table.getMetaClient());
List<HoodieFileGroup> firstPartitionCommit2FileGroups = fsView.getAllFileGroups(DEFAULT_FIRST_PARTITION_PATH).collect(Collectors.toList());
assertEquals(1, firstPartitionCommit2FileGroups.size());
- firstPartitionCommit2FileSlices.addAll(firstPartitionCommit2FileGroups.get(0).getAllFileSlices().collect(Collectors.toList()));
+ firstPartitionCommit2FileSlices.addAll(firstPartitionCommit2FileGroups.get(0).getAllFileSlicesIncludingInflight().collect(Collectors.toList()));
Review Comment:
why are we including inflight slices as well?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java:
##########
@@ -145,6 +150,17 @@ Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath, String max
*/
Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime);
+ /**
+ * Stream all "merged" file-slices before or on an instant time If a file-group has a pending compaction request, the
+ * file-slice before and after compaction request instant is merged and returned. This API can also include pending instants.
+ *
+ * @param partitionPath Partition Path
+ * @param maxInstantTime Max Instant Time
+ * @param includePending Ability to include pending instants as well
+ * @return
+ */
+ Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime, boolean includePending);
Review Comment:
please add tests for the new methods
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -181,17 +204,24 @@ private void removePendingCompactionInstant(HoodieInstant instant) throws IOExce
}
/**
- * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit,
- * so you no longer need to track them as pending.
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
+ * operation is failed, so it is no longer need to be tracked as pending.
*
* @param instant Log Compaction Instant to be removed
+ * @param isSuccessful true if log compaction operation is successful, false if the operation is failed and rollbacked.
*/
- private void removePendingLogCompactionInstant(HoodieInstant instant) throws IOException {
- LOG.info("Removing completed log compaction instant (" + instant + ")");
- HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
- removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
- .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
- CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ private void removePendingLogCompactionInstant(HoodieInstant instant, boolean isSuccessful) throws IOException {
+ if (isSuccessful) {
+ LOG.info("Removing completed log compaction instant (" + instant + ")");
+ HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
+ removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
+ .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
+ CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ } else {
+ LOG.info("Removing failed log compaction instant (" + instant + ")");
+ removePendingLogCompactionOperations(instant.getTimestamp());
Review Comment:
yes i think we should apply the same to clustering commit too.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -134,15 +137,16 @@ protected void init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActi
* @param visibleActiveTimeline Visible Active Timeline
*/
protected void refreshTimeline(HoodieTimeline visibleActiveTimeline) {
- this.visibleCommitsAndCompactionTimeline = visibleActiveTimeline.getWriteTimeline();
+ this.visibleFileSystemViewTimeline = visibleActiveTimeline
+ .getFileSystemViewTimeline();
}
/**
* Adds the provided statuses into the file system view, and also caches it inside this object.
*/
public List<HoodieFileGroup> addFilesToView(FileStatus[] statuses) {
- HoodieTimer timer = HoodieTimer.start();
- List<HoodieFileGroup> fileGroups = buildFileGroups(statuses, visibleCommitsAndCompactionTimeline, true);
+ HoodieTimer timer = new HoodieTimer().startTimer();
Review Comment:
is there a need to change this? it re-introduces deprecated code.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -37,27 +38,31 @@ public class TimelineDiffHelper {
private static final Logger LOG = LoggerFactory.getLogger(TimelineDiffHelper.class);
- public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline oldTimeline,
- HoodieTimeline newTimeline) {
-
- HoodieTimeline oldT = oldTimeline.filterCompletedAndCompactionInstants();
- HoodieTimeline newT = newTimeline.filterCompletedAndCompactionInstants();
+ /**
+ * It performs validation to see if the incremental sync is possible. Once it determines it is safe to do,
+ * It tracks both the instants that are newly added and also instants that have transitioned from pending
+ * state to completed or removed.
+ * @param oldCompletedAndRewriteTimeline old completed and rewrite timeline
+ * @param newCompletedAndRewriteTimeline new completed and rewrite timeline
+ * @return TimelineDiffResult object which is useful for incrementally syncing the timeline.
+ */
+ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline oldCompletedAndRewriteTimeline,
Review Comment:
Let's make sure that we add tests for the diff helper as it is used for MDT based fs view too.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java:
##########
@@ -299,6 +310,17 @@ void removeFileGroupsInPendingClustering(Stream<Pair<HoodieFileGroupId, HoodieIn
});
}
+ @Override
+ protected void removeFileGroupsInPendingClustering(String clusteringInstant) {
+ List<HoodieFileGroupId> fileGroupIdsToRemove = fgIdToPendingClustering.entrySet()
+ .stream().filter(fgIdInstantEntry -> clusteringInstant.equals(fgIdInstantEntry.getValue().getTimestamp()))
+ .map(Map.Entry::getKey).collect(Collectors.toList());
+ ValidationUtils.checkArgument(!fileGroupIdsToRemove.isEmpty(),
Review Comment:
Is this validation strictly necessary? If it's not found then also incremental sync can continue, isn't it?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -163,14 +163,19 @@ public interface HoodieTimeline extends Serializable {
*/
HoodieTimeline filterCompletedAndCompactionInstants();
- HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants();
+ /**
+ * File system view that is used for constructing file groups.
+ * This is the filtered timeline contains completed instants that are part of either
+ * write timeline or savepoint timeline. Along with this it also includes any pending compaction commits.
+ */
+ HoodieTimeline getFileSystemViewTimeline();
Review Comment:
Why should it not include rollback and restore timeline? Let's make it clear in the documentation.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
let's make sure we have tests covering all these methods.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -150,11 +151,25 @@ public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClu
public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+ if (clusteringPlan == null || clusteringPlan.getInputGroups() == null || clusteringPlan.getInputGroups().isEmpty()) {
+ return Stream.empty();
+ }
Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
return partitionToFileIdLists.map(e -> Pair.of(e, instant));
}
- private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+ public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesFromClusteringInstant(
+ HoodieInstant clusteringInstant, HoodieTableMetaClient metaClient) throws IOException {
+ Option<HoodieRequestedReplaceMetadata> requestedReplaceMetadata =
+ ClusteringUtils.getRequestedReplaceMetadata(metaClient, clusteringInstant);
Review Comment:
note that for `insert_overwrite` operation, requested instant does not contain the clustering plan, so at line 165 below it might throw NPE.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -478,8 +488,11 @@ protected FileStatus[] listPartition(Path partitionPath) throws IOException {
* @param statuses List of File-Status
*/
private Stream<HoodieBaseFile> convertFileStatusesToBaseFiles(FileStatus[] statuses) {
- Predicate<FileStatus> roFilePredicate = fileStatus -> fileStatus.getPath().getName()
- .contains(metaClient.getTableConfig().getBaseFileFormat().getFileExtension());
+ // Format of .hoodie_partition_metadata is changed to .hoodie_partition_metadata.parquet
Review Comment:
When did this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264721375
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -181,17 +204,24 @@ private void removePendingCompactionInstant(HoodieInstant instant) throws IOExce
}
/**
- * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit,
- * so you no longer need to track them as pending.
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
+ * operation is failed, so it is no longer need to be tracked as pending.
*
* @param instant Log Compaction Instant to be removed
+ * @param isSuccessful true if log compaction operation is successful, false if the operation is failed and rollbacked.
*/
- private void removePendingLogCompactionInstant(HoodieInstant instant) throws IOException {
- LOG.info("Removing completed log compaction instant (" + instant + ")");
- HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
- removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
- .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
- CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ private void removePendingLogCompactionInstant(HoodieInstant instant, boolean isSuccessful) throws IOException {
+ if (isSuccessful) {
+ LOG.info("Removing completed log compaction instant (" + instant + ")");
+ HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
+ removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
+ .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
+ CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ } else {
+ LOG.info("Removing failed log compaction instant (" + instant + ")");
+ removePendingLogCompactionOperations(instant.getTimestamp());
Review Comment:
The concern, is similar to the one raised in TimelineDiffHelper class, I explained the reasoning behind it there.
Also, one more point to note is Compaction plans are by their nature immutable, whereas replacecommits plans can be either mutable or immutable. It depends on what UpdateStrategy implementation users are using. That means traditional flow cannot rollback compaction plans except a developer manually doing a restore, and replacecommit can be rolled back.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -134,15 +137,16 @@ protected void init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActi
* @param visibleActiveTimeline Visible Active Timeline
*/
protected void refreshTimeline(HoodieTimeline visibleActiveTimeline) {
- this.visibleCommitsAndCompactionTimeline = visibleActiveTimeline.getWriteTimeline();
+ this.visibleFileSystemViewTimeline = visibleActiveTimeline
+ .getFileSystemViewTimeline();
}
/**
* Adds the provided statuses into the file system view, and also caches it inside this object.
*/
public List<HoodieFileGroup> addFilesToView(FileStatus[] statuses) {
- HoodieTimer timer = HoodieTimer.start();
- List<HoodieFileGroup> fileGroups = buildFileGroups(statuses, visibleCommitsAndCompactionTimeline, true);
+ HoodieTimer timer = new HoodieTimer().startTimer();
Review Comment:
Good catch, rebasing might have introduced it fixed it now.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -130,36 +141,50 @@ private void runIncrementalSync(HoodieTimeline timeline, TimelineDiffResult diff
});
// Now remove pending log compaction instants which were completed or removed
- diffResult.getFinishedOrRemovedLogCompactionInstants().stream().forEach(instant -> {
+ diffResult.getFinishedOrRemovedLogCompactionInstants().stream().forEach(instantPair -> {
+ try {
+ removePendingLogCompactionInstant(instantPair.getKey(), instantPair.getValue());
+ } catch (IOException e) {
+ throw new HoodieException(e);
+ }
+ });
+
+ // Now remove pending replace instants which were completed or removed
+ diffResult.getFinishedOrRemovedReplaceInstants().stream().forEach(instantPair -> {
try {
- removePendingLogCompactionInstant(instant);
+ removePendingFileGroupsInPendingClustering(instantPair.getKey(), instantPair.getValue());
} catch (IOException e) {
throw new HoodieException(e);
}
});
- // Add new completed instants found in the latest timeline, this also contains inflight instants.
- diffResult.getNewlySeenInstants().stream()
- .filter(instant -> instant.isCompleted()
- || instant.getAction().equals(HoodieTimeline.COMPACTION_ACTION)
- || instant.getAction().equals(HoodieTimeline.LOG_COMPACTION_ACTION))
+ HoodieTimeline newFileSystemViewTimeline =
+ newActiveTimeline.getFileSystemViewTimeline();
+ // Add new completed instants found in the latest timeline,
+ // this also contains inflight instants from rewrite timeline.
+ diffResult.getNewlySeenCompletedAndRewriteInstants()
.forEach(instant -> {
try {
if (instant.getAction().equals(HoodieTimeline.COMMIT_ACTION)
|| instant.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)) {
- addCommitInstant(timeline, instant);
+ addCommitInstant(newFileSystemViewTimeline, instant);
Review Comment:
It is, if I pass active timeline multiple times we will have to filter and recreate fileSystemViewTimeline. So, I created it once and passed it for methods that construct file groups. Also, all these method are private and are present in the same class so they could use a common view when building file groups.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -90,9 +91,11 @@ public abstract class AbstractTableFileSystemView implements SyncableFileSystemV
protected HoodieTableMetaClient metaClient;
- // This is the commits timeline that will be visible for all views extending this view
- // This is nothing but the write timeline, which contains both ingestion and compaction(major and minor) writers.
- private HoodieTimeline visibleCommitsAndCompactionTimeline;
+ // Timeline determines whether a file slice is committed or not. This timeline object is attached
Review Comment:
Yeah, fixed it.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -150,11 +151,25 @@ public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClu
public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+ if (clusteringPlan == null || clusteringPlan.getInputGroups() == null || clusteringPlan.getInputGroups().isEmpty()) {
+ return Stream.empty();
+ }
Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
return partitionToFileIdLists.map(e -> Pair.of(e, instant));
}
- private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+ public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesFromClusteringInstant(
+ HoodieInstant clusteringInstant, HoodieTableMetaClient metaClient) throws IOException {
+ Option<HoodieRequestedReplaceMetadata> requestedReplaceMetadata =
+ ClusteringUtils.getRequestedReplaceMetadata(metaClient, clusteringInstant);
Review Comment:
isCluteringInstant checks are already before calling this API.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -478,8 +488,11 @@ protected FileStatus[] listPartition(Path partitionPath) throws IOException {
* @param statuses List of File-Status
*/
private Stream<HoodieBaseFile> convertFileStatusesToBaseFiles(FileStatus[] statuses) {
- Predicate<FileStatus> roFilePredicate = fileStatus -> fileStatus.getPath().getName()
- .contains(metaClient.getTableConfig().getBaseFileFormat().getFileExtension());
+ // Format of .hoodie_partition_metadata is changed to .hoodie_partition_metadata.parquet
Review Comment:
Ref: https://github.com/apache/hudi/pull/5179/files#diff-c2a2767975107231dd51373bedd57cff81256256e0cb9ea2393b5af78f2c66dbR112
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java:
##########
@@ -299,6 +310,17 @@ void removeFileGroupsInPendingClustering(Stream<Pair<HoodieFileGroupId, HoodieIn
});
}
+ @Override
+ protected void removeFileGroupsInPendingClustering(String clusteringInstant) {
+ List<HoodieFileGroupId> fileGroupIdsToRemove = fgIdToPendingClustering.entrySet()
+ .stream().filter(fgIdInstantEntry -> clusteringInstant.equals(fgIdInstantEntry.getValue().getTimestamp()))
+ .map(Map.Entry::getKey).collect(Collectors.toList());
+ ValidationUtils.checkArgument(!fileGroupIdsToRemove.isEmpty(),
Review Comment:
If we have arrived at this point, that means old timeline has replacecommit in-inflight and new timeline either completed it or rolled back.
In that case, if we are not finding the filegroups to remove that means there maybe some other issues. So, it is could to fail incremental file sync and let full sync happen. If required at later point we could add monitoring for incremental sync to make sure everything is working as expected.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -144,44 +172,54 @@ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingCompactionTran
*/
public static class TimelineDiffResult {
- private final List<HoodieInstant> newlySeenInstants;
+ private final List<HoodieInstant> newlySeenCompletedAndRewriteInstants;
private final List<HoodieInstant> finishedCompactionInstants;
Review Comment:
Ideally compaction are not removed, since after creating a compaction plan even though they are in inflight state new data is written to new file slices. So, if we were to remove compaction, then we might need to remove latest files. We could add APIs to remove them, but it is bit risky.
Also compaction plans are immutable plans so writers cannot delete them, only developers can remove them manually by using restore operations.
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java:
##########
@@ -833,11 +832,11 @@ private Pair<List<HoodieRecord>, List<HoodieRecord>> twoUpsertCommitDataWithTwoP
SyncableFileSystemView fsView = getFileSystemViewWithUnCommittedSlices(table.getMetaClient());
List<HoodieFileGroup> firstPartitionCommit2FileGroups = fsView.getAllFileGroups(DEFAULT_FIRST_PARTITION_PATH).collect(Collectors.toList());
assertEquals(1, firstPartitionCommit2FileGroups.size());
- firstPartitionCommit2FileSlices.addAll(firstPartitionCommit2FileGroups.get(0).getAllFileSlices().collect(Collectors.toList()));
+ firstPartitionCommit2FileSlices.addAll(firstPartitionCommit2FileGroups.get(0).getAllFileSlicesIncludingInflight().collect(Collectors.toList()));
Review Comment:
Existing unit tests are written to read inflight file slices from the views, it is not evident from the APIs we have in File system view classes.
For example look at line 832 in this class., by passing active timeline they are expecting to read inflight file slices and by passing completedAndCompactionTimeline in FileSystemViewManager class they can read committed file slices. With this change everyone have to pass active timeline and use the appropriate APIs to read the respective slice streams.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
I have created these methods to fix broken test cases, existing unit tests are already use the logic present in these methods in a different way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264737094
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -165,12 +165,18 @@ public interface HoodieTimeline extends Serializable {
HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants();
+ /**
+ * This is the filtered write timeline that contains only completed and pending compaction instants.
+ * This is the timeline that is used in constructing file groups.
+ */
+ HoodieTimeline filterCompletedWriteAndCompactionInstants();
+
/**
* Timeline to just include completed commits or all rewrites like compaction, logcompaction and replace actions.
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants(HoodieTableMetaClient metaClient);
Review Comment:
ok, lets go w/ rewrite for now. but if we can come up w/ a good name, should try to use one.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -144,44 +172,54 @@ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingCompactionTran
*/
public static class TimelineDiffResult {
- private final List<HoodieInstant> newlySeenInstants;
+ private final List<HoodieInstant> newlySeenCompletedAndRewriteInstants;
private final List<HoodieInstant> finishedCompactionInstants;
Review Comment:
synced up f2f.
bcoz, log compaction may not result in new file slices.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -123,21 +126,24 @@ public HoodieTimeline filterCompletedAndCompactionInstants() {
}
@Override
- public HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants() {
- return new HoodieDefaultTimeline(getInstantsAsStream().filter(s -> s.isCompleted()
- || s.getAction().equals(HoodieTimeline.COMPACTION_ACTION) || s.getAction().equals(HoodieTimeline.LOG_COMPACTION_ACTION)), details);
+ public HoodieTimeline getFileSystemViewTimeline() {
+ return new HoodieDefaultTimeline(getInstantsAsStream()
+ .filter(x ->
+ (x.isCompleted() && (HoodieTimeline.SAVEPOINT_ACTION.equals(x.getAction()) || VALID_WRITE_ACTIONS.contains(x.getAction())))
+ || HoodieTimeline.COMPACTION_ACTION.equals(x.getAction())
+ ), details);
}
@Override
- public HoodieDefaultTimeline filterCompletedInstantsOrRewriteTimeline() {
- Set<String> validActions = CollectionUtils.createSet(COMPACTION_ACTION, LOG_COMPACTION_ACTION, REPLACE_COMMIT_ACTION);
- return new HoodieDefaultTimeline(getInstantsAsStream().filter(s -> s.isCompleted() || validActions.contains(s.getAction())), details);
+ public HoodieDefaultTimeline filterCompletedAndRewriteInstants() {
+ List<String> validActions = Arrays.asList(COMPACTION_ACTION, LOG_COMPACTION_ACTION, REPLACE_COMMIT_ACTION);
Review Comment:
can we name this rewriteActions
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -130,36 +141,50 @@ private void runIncrementalSync(HoodieTimeline timeline, TimelineDiffResult diff
});
// Now remove pending log compaction instants which were completed or removed
- diffResult.getFinishedOrRemovedLogCompactionInstants().stream().forEach(instant -> {
+ diffResult.getFinishedOrRemovedLogCompactionInstants().stream().forEach(instantPair -> {
+ try {
+ removePendingLogCompactionInstant(instantPair.getKey(), instantPair.getValue());
+ } catch (IOException e) {
+ throw new HoodieException(e);
+ }
+ });
+
+ // Now remove pending replace instants which were completed or removed
+ diffResult.getFinishedOrRemovedReplaceInstants().stream().forEach(instantPair -> {
try {
- removePendingLogCompactionInstant(instant);
+ removePendingFileGroupsInPendingClustering(instantPair.getKey(), instantPair.getValue());
} catch (IOException e) {
throw new HoodieException(e);
}
});
- // Add new completed instants found in the latest timeline, this also contains inflight instants.
- diffResult.getNewlySeenInstants().stream()
- .filter(instant -> instant.isCompleted()
- || instant.getAction().equals(HoodieTimeline.COMPACTION_ACTION)
- || instant.getAction().equals(HoodieTimeline.LOG_COMPACTION_ACTION))
+ HoodieTimeline newFileSystemViewTimeline =
+ newActiveTimeline.getFileSystemViewTimeline();
+ // Add new completed instants found in the latest timeline,
+ // this also contains inflight instants from rewrite timeline.
+ diffResult.getNewlySeenCompletedAndRewriteInstants()
.forEach(instant -> {
try {
if (instant.getAction().equals(HoodieTimeline.COMMIT_ACTION)
|| instant.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)) {
- addCommitInstant(timeline, instant);
+ addCommitInstant(newFileSystemViewTimeline, instant);
Review Comment:
sg
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -123,21 +126,24 @@ public HoodieTimeline filterCompletedAndCompactionInstants() {
}
@Override
- public HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants() {
- return new HoodieDefaultTimeline(getInstantsAsStream().filter(s -> s.isCompleted()
- || s.getAction().equals(HoodieTimeline.COMPACTION_ACTION) || s.getAction().equals(HoodieTimeline.LOG_COMPACTION_ACTION)), details);
+ public HoodieTimeline getFileSystemViewTimeline() {
+ return new HoodieDefaultTimeline(getInstantsAsStream()
+ .filter(x ->
+ (x.isCompleted() && (HoodieTimeline.SAVEPOINT_ACTION.equals(x.getAction()) || VALID_WRITE_ACTIONS.contains(x.getAction())))
+ || HoodieTimeline.COMPACTION_ACTION.equals(x.getAction())
Review Comment:
lets add documentation on why compaction needs special treatment.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1246,6 +1304,19 @@ Stream<FileSlice> fetchAllFileSlices(String partitionPath) {
.flatMap(HoodieFileGroup::getAllFileSlices);
}
+ /**
+ * Default implementation for fetching all file-slices for a partition-path.
+ *
+ * @param partitionPath Partition path
+ * @return file-slice stream
+ */
+ Stream<FileSlice> fetchAllFileSlices(String partitionPath, boolean includePending) {
Review Comment:
can you check out the methods used only in tests nad add annotations.
also, revisit access specifiers for the ones you are touching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1596699202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3360fa18333a0097fa762824f02eb9cd6c4bad5d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1614815163
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 78b7acc447a6cdadccf1b0ca57e1cc634233c879 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1612553846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3360fa18333a0097fa762824f02eb9cd6c4bad5d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958)
* 3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636629167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 578401a45b7c5ffbd9360de3bd3e18c362b4b2b5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592)
* ea6504e78fbb1c776687d3632c5875e74070cebd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604)
* de9fca47509129e13c9b3a422261e8c55978faa0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636545530
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 578401a45b7c5ffbd9360de3bd3e18c362b4b2b5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636928771
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 488e22d276930ac0836fd54d5c3cd5623faeb544 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610)
* 191423c89583c99c085dd527eab46ab72894d1df Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612)
* 7ffcac1db2479936bdee038951fab00071597d26 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637083432
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264807691
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java:
##########
@@ -833,11 +832,11 @@ private Pair<List<HoodieRecord>, List<HoodieRecord>> twoUpsertCommitDataWithTwoP
SyncableFileSystemView fsView = getFileSystemViewWithUnCommittedSlices(table.getMetaClient());
List<HoodieFileGroup> firstPartitionCommit2FileGroups = fsView.getAllFileGroups(DEFAULT_FIRST_PARTITION_PATH).collect(Collectors.toList());
assertEquals(1, firstPartitionCommit2FileGroups.size());
- firstPartitionCommit2FileSlices.addAll(firstPartitionCommit2FileGroups.get(0).getAllFileSlices().collect(Collectors.toList()));
+ firstPartitionCommit2FileSlices.addAll(firstPartitionCommit2FileGroups.get(0).getAllFileSlicesIncludingInflight().collect(Collectors.toList()));
Review Comment:
> With this change everyone have to pass active timeline and use the appropriate APIs to read the respective slice streams.
I'm kind of -1 for this direction, passing active timeline to fetch the pending file slices make more sense to me, at least it does not introduce core API on timeline for just testing purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264721653
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -150,11 +151,25 @@ public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClu
public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+ if (clusteringPlan == null || clusteringPlan.getInputGroups() == null || clusteringPlan.getInputGroups().isEmpty()) {
+ return Stream.empty();
+ }
Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
return partitionToFileIdLists.map(e -> Pair.of(e, instant));
}
- private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+ public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesFromClusteringInstant(
+ HoodieInstant clusteringInstant, HoodieTableMetaClient metaClient) throws IOException {
+ Option<HoodieRequestedReplaceMetadata> requestedReplaceMetadata =
+ ClusteringUtils.getRequestedReplaceMetadata(metaClient, clusteringInstant);
Review Comment:
isCluteringInstant checks are already added before calling this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264602796
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
In general, we should add core API only for tesing purpose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264602474
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -448,14 +523,19 @@ protected void applyDeltaFileSlicesToPartitionView(String partition, List<Hoodie
throw new IllegalStateException("Unknown diff apply mode=" + mode);
}
- HoodieTimeline timeline = deltaFileGroups.stream().map(df -> df.getTimeline()).findAny().get();
+ // Here timeline is fetched from the existing file groups so it is automatically completedWriteAndCompactionTimeline
+ HoodieTimeline completedWriteAndCompactionTimeline = deltaFileGroups.stream().map(df -> df.getTimeline()).findAny().get();
List<HoodieFileGroup> fgs =
- buildFileGroups(viewDataFiles.values().stream(), viewLogFiles.values().stream(), timeline, true);
+ buildFileGroups(viewDataFiles.values().stream(), viewLogFiles.values().stream(), completedWriteAndCompactionTimeline, true);
storePartitionView(partition, fgs);
}
@Override
public HoodieTimeline getTimeline() {
return visibleActiveTimeline;
}
+
+ public boolean isLastIncrementalSyncSuccessful() {
Review Comment:
Marked as `@VisibleForTesting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636084210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 26b3151e371774b3e99324bd9c305157fcde5789 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636680266
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 3e318aa173aaa30e984554380d41c7706b1e7061 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1612545219
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3360fa18333a0097fa762824f02eb9cd6c4bad5d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958)
* 3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1595884668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1596431522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939)
* 3360fa18333a0097fa762824f02eb9cd6c4bad5d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1596453840
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939)
* 3360fa18333a0097fa762824f02eb9cd6c4bad5d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1233519003
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -40,8 +40,8 @@ public class TimelineDiffHelper {
public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline oldTimeline,
HoodieTimeline newTimeline) {
- HoodieTimeline oldT = oldTimeline.filterCompletedAndCompactionInstants();
- HoodieTimeline newT = newTimeline.filterCompletedAndCompactionInstants();
+ HoodieTimeline oldT = oldTimeline.filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline newT = newTimeline.filterCompletedInstantsOrRewriteTimeline();
Review Comment:
The `filterCompletedInstantsOrRewriteTimeline` does not look like a good name, how about rename it to `filterCompletedAndRewriteInstants`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
+ HoodieTimeline newTimeline) {
+ Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
+
+ return oldTimeline.filterPendingReplaceTimeline().getInstantsAsStream().map(instant -> {
+ if (newTimelineInstants.contains(instant)) {
Review Comment:
The methods `getPendingReplaceCommitTransitions ` and `getPendingLogCompactionTransitions` look almost the same except the action type, can we abstract a little to merge them altogether?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -118,6 +118,8 @@ public class HoodieTableMetaClient implements Serializable {
private ConsistencyGuardConfig consistencyGuardConfig = ConsistencyGuardConfig.newBuilder().build();
private FileSystemRetryConfig fileSystemRetryConfig = FileSystemRetryConfig.newBuilder().build();
protected HoodieMetaserverConfig metaserverConfig;
+ // Tracks the status of the last incremental file sync
+ private boolean isLastIncrementalSyncSuccessful;
Review Comment:
Don't think the meta client should hold state like this. We should move the state into `IncrementalTimelineSyncFileSystemView`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
Review Comment:
The strategy handling of deleted pending instants for compaction and logcompaction/clustering are different, for compaction we return early, do you think we should unify the handling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1595923271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636728603
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 488e22d276930ac0836fd54d5c3cd5623faeb544 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264600927
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -170,7 +170,7 @@ public interface HoodieTimeline extends Serializable {
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants();
Review Comment:
+1 to `filterCompletedAndRewriteInstants `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636653219
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* de9fca47509129e13c9b3a422261e8c55978faa0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606)
* 3e318aa173aaa30e984554380d41c7706b1e7061 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1614141754
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f154ee335eb307e2bcffd895cfd95bfb1f417a1e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1247516881
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -118,6 +118,8 @@ public class HoodieTableMetaClient implements Serializable {
private ConsistencyGuardConfig consistencyGuardConfig = ConsistencyGuardConfig.newBuilder().build();
private FileSystemRetryConfig fileSystemRetryConfig = FileSystemRetryConfig.newBuilder().build();
protected HoodieMetaserverConfig metaserverConfig;
+ // Tracks the status of the last incremental file sync
+ private boolean isLastIncrementalSyncSuccessful;
Review Comment:
Initially I wanted to remove this state variable altogether, but it makes sense to move this to IncrementalTimelineSyncFileSystemView. So, refactored the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1247519679
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
Review Comment:
Updated the above comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1614245819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f154ee335eb307e2bcffd895cfd95bfb1f417a1e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218)
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1614305979
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f154ee335eb307e2bcffd895cfd95bfb1f417a1e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218)
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 78b7acc447a6cdadccf1b0ca57e1cc634233c879 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636611094
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 578401a45b7c5ffbd9360de3bd3e18c362b4b2b5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592)
* ea6504e78fbb1c776687d3632c5875e74070cebd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636222736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 26b3151e371774b3e99324bd9c305157fcde5789 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585)
* 07eb1aa79162259f3ac79e61bec621f68afb5551 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636322264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 07eb1aa79162259f3ac79e61bec621f68afb5551 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588)
* 578401a45b7c5ffbd9360de3bd3e18c362b4b2b5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264527544
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -53,6 +53,7 @@ public static Comparator<String> getReverseCommitTimeComparator() {
/**
* Timeline, based on which all getter work.
+ * This should be a write timeline that contains either completed instants or pending compaction instants.
Review Comment:
Good idea, I will use this name across all the classes.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -134,14 +134,20 @@ protected void init(HoodieTableMetaClient metaClient, HoodieTimeline visibleActi
* @param visibleActiveTimeline Visible Active Timeline
*/
protected void refreshTimeline(HoodieTimeline visibleActiveTimeline) {
- this.visibleCommitsAndCompactionTimeline = visibleActiveTimeline.getWriteTimeline();
+ this.visibleCommitsAndCompactionTimeline = visibleActiveTimeline
+ .getWriteTimeline()
+ .filterCompletedAndCompactionInstants();
}
/**
* Adds the provided statuses into the file system view, and also caches it inside this object.
*/
public List<HoodieFileGroup> addFilesToView(FileStatus[] statuses) {
- HoodieTimer timer = HoodieTimer.start();
+ HoodieTimer timer = new HoodieTimer().startTimer();
+ // Timeline determines whether a file slice is committed or not. Timeline object is attached to each HoodieFileGroup.
+ // Using this timeline object HoodieFileGroup determines whether a fileSlice is committed or not.
+ // visibleCommitsAndCompactionTimeline contains entire write timeline with pending commits, so filtering it
Review Comment:
I think this comment was leftover from original PR that I have raised. Removed it from here and added it at the appropriate place.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/TableFileSystemView.java:
##########
@@ -145,6 +150,17 @@ Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath, String max
*/
Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime);
+ /**
+ * Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the
Review Comment:
Fixed it.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -170,15 +176,25 @@ public List<HoodieFileGroup> addFilesToView(FileStatus[] statuses) {
/**
* Build FileGroups from passed in file-status.
+ * TODO: Note here timeline is completedAndCompactionTimeline.
Review Comment:
Yeah, makes sense. We can also avoid doing multiple times filtering on the activeTimeline to get this timeline. So, renamed in all the places so that developers dont even have to perform extra filtering.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -327,11 +357,43 @@ private void addRollbackInstant(HoodieTimeline timeline, HoodieInstant instant)
LOG.info("Done Syncing rollback instant (" + instant + ")");
}
+ /**
+ * Add pending replace file groups
+ * @param pendingClusteringInstant pending instant
+ * @throws IOException
+ */
+ private void addPendingClusteringInstant(HoodieInstant pendingClusteringInstant) throws IOException {
+ LOG.info("Syncing pending clustering instant (" + pendingClusteringInstant + ")");
+ Stream<Pair<HoodieFileGroupId, HoodieInstant>> fileGroupStreamToInstant =
+ ClusteringUtils.getFileGroupEntriesFromClusteringInstant(pendingClusteringInstant, metaClient);
+ addFileGroupsInPendingClustering(fileGroupStreamToInstant);
+ }
+
+ /**
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
Review Comment:
Fixed it.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -181,17 +204,24 @@ private void removePendingCompactionInstant(HoodieInstant instant) throws IOExce
}
/**
- * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit,
- * so you no longer need to track them as pending.
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
+ * operation is failed, so it is no longer need to be tracked as pending.
*
* @param instant Log Compaction Instant to be removed
+ * @param isSuccessful true if log compaction operation is successful, false if the operation is failed and rollbacked.
*/
- private void removePendingLogCompactionInstant(HoodieInstant instant) throws IOException {
- LOG.info("Removing completed log compaction instant (" + instant + ")");
- HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
- removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
- .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
- CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ private void removePendingLogCompactionInstant(HoodieInstant instant, boolean isSuccessful) throws IOException {
+ if (isSuccessful) {
+ LOG.info("Removing completed log compaction instant (" + instant + ")");
+ HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
+ removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
+ .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
+ CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ } else {
+ LOG.info("Removing failed log compaction instant (" + instant + ")");
+ removePendingLogCompactionOperations(instant.getTimestamp());
Review Comment:
If a compaction instant is removed then it is not safe to do incremental file sync, because there could be removal of instants and also removal of file slices, so we fail the check in TimelineDiffHelper and the job automatically fallback on doing full sync.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -181,17 +204,24 @@ private void removePendingCompactionInstant(HoodieInstant instant) throws IOExce
}
/**
- * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit,
- * so you no longer need to track them as pending.
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
Review Comment:
Fixed it.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -146,17 +189,22 @@ public static class TimelineDiffResult {
private final List<HoodieInstant> newlySeenInstants;
private final List<HoodieInstant> finishedCompactionInstants;
- private final List<HoodieInstant> finishedOrRemovedLogCompactionInstants;
+ // Completed instants will have true as the value where as instants removed due to rollback will have false as value.
+ private final List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants;
+ // Completed instants will have true as the value where as instants removed due to rollback will have false as value.
+ private final List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants;
private final boolean canSyncIncrementally;
public static final TimelineDiffResult UNSAFE_SYNC_RESULT =
- new TimelineDiffResult(null, null, null, false);
+ new TimelineDiffResult(null, null, null, null, false);
public TimelineDiffResult(List<HoodieInstant> newlySeenInstants, List<HoodieInstant> finishedCompactionInstants,
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants, boolean canSyncIncrementally) {
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants,
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants, boolean canSyncIncrementally) {
this.newlySeenInstants = newlySeenInstants;
this.finishedCompactionInstants = finishedCompactionInstants;
this.finishedOrRemovedLogCompactionInstants = finishedOrRemovedLogCompactionInstants;
+ this.finishedOrRemovedReplaceCommitInstants = finishedOrRemovedReplaceCommitInstants;
Review Comment:
Yes, renamed all these variables.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -170,7 +170,7 @@ public interface HoodieTimeline extends Serializable {
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants();
Review Comment:
Here we need to filter based on completed instants and rewrite instants. So, with rewrite you can get pending replacecommits, logcompactions and compactions with completed you will get all the commits that are completed.
This API is similar to filterCompletedAndCompactionInstants, but instead of just including compactions it also includes other rewrites instants.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -159,7 +177,12 @@ private void runIncrementalSync(HoodieTimeline timeline, TimelineDiffResult diff
} else if (instant.getAction().equals(HoodieTimeline.ROLLBACK_ACTION)) {
addRollbackInstant(timeline, instant);
} else if (instant.getAction().equals(HoodieTimeline.REPLACE_COMMIT_ACTION)) {
- addReplaceInstant(timeline, instant);
+ boolean isClusteringCommit = ClusteringUtils.isClusteringCommit(metaClient, instant);
+ if (!instant.isCompleted() && isClusteringCommit) {
+ addPendingClusteringInstant(instant);
+ } else {
+ addReplaceInstant(timeline, instant);
Review Comment:
Completed insert-overwrite commits also can come here.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
Review Comment:
Refactored it.
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -325,6 +325,15 @@ private void registerFileSlicesAPI() {
writeValueAsString(ctx, dtos);
}, true));
+ app.get(RemoteHoodieTableFileSystemView.LATEST_PARTITION_SLICES_INCLUDE_PENDING_URL, new ViewHandler(ctx -> {
+ metricsRegistry.add("LATEST_PARTITION_SLICES", 1);
Review Comment:
Fixed it.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
Review Comment:
Good catch, I think the filtering is missing in the HoodieTimeline's filterCompletedAndRewriteInstants method. I added it now. Also, renamed as clustering transitions instead of replace-commit transitions.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/RocksDbBasedFileSystemView.java:
##########
@@ -544,10 +574,13 @@ protected Option<HoodieInstant> getReplaceInstant(final HoodieFileGroupId fileGr
}
private Stream<HoodieFileGroup> getFileGroups(Stream<FileSlice> sliceStream) {
+ // HoodieFileGroup using timeline to determine whether a file slice is committed or not so passing
+ // completedAndCompactionTimeline object while creating HoodieFileGroup.
+ HoodieTimeline completedAndCompactionTimeline = getVisibleCommitsAndCompactionTimeline();
Review Comment:
Refactored it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636931047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 191423c89583c99c085dd527eab46ab72894d1df Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612)
* 7ffcac1db2479936bdee038951fab00071597d26 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1244870841
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -40,8 +40,8 @@ public class TimelineDiffHelper {
public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline oldTimeline,
HoodieTimeline newTimeline) {
- HoodieTimeline oldT = oldTimeline.filterCompletedAndCompactionInstants();
- HoodieTimeline newT = newTimeline.filterCompletedAndCompactionInstants();
+ HoodieTimeline oldT = oldTimeline.filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline newT = newTimeline.filterCompletedInstantsOrRewriteTimeline();
Review Comment:
Yeah makes sense, renaming the API.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -118,6 +118,8 @@ public class HoodieTableMetaClient implements Serializable {
private ConsistencyGuardConfig consistencyGuardConfig = ConsistencyGuardConfig.newBuilder().build();
private FileSystemRetryConfig fileSystemRetryConfig = FileSystemRetryConfig.newBuilder().build();
protected HoodieMetaserverConfig metaserverConfig;
+ // Tracks the status of the last incremental file sync
+ private boolean isLastIncrementalSyncSuccessful;
Review Comment:
I have created this for testing purposes, I will remove this after addressing all the unit test failures.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
+ HoodieTimeline newTimeline) {
+ Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
+
+ return oldTimeline.filterPendingReplaceTimeline().getInstantsAsStream().map(instant -> {
+ if (newTimelineInstants.contains(instant)) {
Review Comment:
Yeah, I tried to avoid refactoring in this PR because the logic in itself is complex. So, we can have a followup task to address the refactoring.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
Review Comment:
Compaction plans are immutable plans, so once they are created even though the execution of the plan is not completed, new log files are still written to new file slice. Whereas Log compaction and clustering can be immutable, they can be removed so the logic varies.
Compaction plans are removed only during a restore operation, so it is ok to do incremental file sync such cases, whereas for other 2 action types, incremental file sync should be able to handle the transitions from inflight to completed or inflight to rollback etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636308312
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 26b3151e371774b3e99324bd9c305157fcde5789 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585)
* 07eb1aa79162259f3ac79e61bec621f68afb5551 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588)
* 578401a45b7c5ffbd9360de3bd3e18c362b4b2b5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636639990
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* ea6504e78fbb1c776687d3632c5875e74070cebd Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604)
* de9fca47509129e13c9b3a422261e8c55978faa0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264602796
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
In general, we should not add core API only for tesing purpose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636698257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 3e318aa173aaa30e984554380d41c7706b1e7061 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608)
* 488e22d276930ac0836fd54d5c3cd5623faeb544 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264601895
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
We should be cautious to expose inflight file slices on the filesystem view, the MVCC completeness are somehow broken for all the readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637034941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 7ffcac1db2479936bdee038951fab00071597d26 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618)
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637158641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
}, {
"hash" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626)
* 0964e4bc4aeb82ba6230920736c3d432bfdb5d59 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264808494
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
> let's make sure we have tests covering all these methods.
We should not introduce these APIs in the very first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636630900
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* ea6504e78fbb1c776687d3632c5875e74070cebd Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604)
* de9fca47509129e13c9b3a422261e8c55978faa0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636641347
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* de9fca47509129e13c9b3a422261e8c55978faa0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606)
* 3e318aa173aaa30e984554380d41c7706b1e7061 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636912447
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 488e22d276930ac0836fd54d5c3cd5623faeb544 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610)
* 191423c89583c99c085dd527eab46ab72894d1df UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637046766
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625)
* 34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637172084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
}, {
"hash" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18629",
"triggerID" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"triggerType" : "PUSH"
}, {
"hash" : "d8073e7e042d879324f45f26f00d9d059c1e5d0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8073e7e042d879324f45f26f00d9d059c1e5d0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626)
* 0964e4bc4aeb82ba6230920736c3d432bfdb5d59 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18629)
* d8073e7e042d879324f45f26f00d9d059c1e5d0a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637160169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
}, {
"hash" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18629",
"triggerID" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626)
* 0964e4bc4aeb82ba6230920736c3d432bfdb5d59 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18629)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264717414
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
I have created these methods to fix broken test cases, existing unit tests are already using the logic present in these methods in a different way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264808139
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -53,6 +53,7 @@ public static Comparator<String> getReverseCommitTimeComparator() {
/**
* Timeline, based on which all getter work.
+ * This should be a write timeline that contains either completed instants or pending compaction instants.
Review Comment:
Can we stop renaming all kinds of variables to make the review faster? We can address that in another PR I think, the variable naming is very subjective, everyone has a preference, let's not get stucked for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1595883259
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1244878470
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
Review Comment:
Compaction plans are immutable plans, so once they are created even though the execution of the plan is not completed, new log files are still written to new file slice. Whereas Log compaction and clustering can be mutable, they can be removed so the logic varies for them.
Compaction plans are removed only during a restore operation, so if a compaction commit is removed and another tries to do incremental sync in such cases incremental file sync fails it we are ok in doing full sync. Whereas for other 2 action types, incremental file sync should be able to handle the transitions from inflight to completed or inflight to rollback etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636231430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 26b3151e371774b3e99324bd9c305157fcde5789 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585)
* 07eb1aa79162259f3ac79e61bec621f68afb5551 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636695073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 3e318aa173aaa30e984554380d41c7706b1e7061 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608)
* 488e22d276930ac0836fd54d5c3cd5623faeb544 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264579177
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java:
##########
@@ -137,7 +137,7 @@ public void close() {
*/
private static RocksDbBasedFileSystemView createRocksDBBasedFileSystemView(SerializableConfiguration conf,
FileSystemViewStorageConfig viewConf, HoodieTableMetaClient metaClient) {
- HoodieTimeline timeline = metaClient.getActiveTimeline().filterCompletedAndCompactionInstants();
+ HoodieTimeline timeline = metaClient.getActiveTimeline();
Review Comment:
Yeah, I have only checked and fixed one level of calls, i.e. mainly on the view manager. Need to check all the other levels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636913850
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 488e22d276930ac0836fd54d5c3cd5623faeb544 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610)
* 191423c89583c99c085dd527eab46ab72894d1df Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637021393
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 7ffcac1db2479936bdee038951fab00071597d26 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618)
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637033324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 7ffcac1db2479936bdee038951fab00071597d26 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618)
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637045262
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 7ffcac1db2479936bdee038951fab00071597d26 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618)
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625)
* 34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264703653
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
+ HoodieTimeline newTimeline) {
+ Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
+
+ return oldTimeline.filterPendingReplaceTimeline().getInstantsAsStream().map(instant -> {
+ if (newTimelineInstants.contains(instant)) {
Review Comment:
Refactored the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264603260
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1246,6 +1304,19 @@ Stream<FileSlice> fetchAllFileSlices(String partitionPath) {
.flatMap(HoodieFileGroup::getAllFileSlices);
}
+ /**
+ * Default implementation for fetching all file-slices for a partition-path.
+ *
+ * @param partitionPath Partition path
+ * @return file-slice stream
+ */
+ Stream<FileSlice> fetchAllFileSlices(String partitionPath, boolean includePending) {
Review Comment:
do we need access specifier here? private
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java:
##########
@@ -137,7 +137,7 @@ public void close() {
*/
private static RocksDbBasedFileSystemView createRocksDBBasedFileSystemView(SerializableConfiguration conf,
FileSystemViewStorageConfig viewConf, HoodieTableMetaClient metaClient) {
- HoodieTimeline timeline = metaClient.getActiveTimeline().filterCompletedAndCompactionInstants();
+ HoodieTimeline timeline = metaClient.getActiveTimeline();
Review Comment:
ok
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -129,9 +131,15 @@ public HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants() {
}
@Override
- public HoodieDefaultTimeline filterCompletedInstantsOrRewriteTimeline() {
- Set<String> validActions = CollectionUtils.createSet(COMPACTION_ACTION, LOG_COMPACTION_ACTION, REPLACE_COMMIT_ACTION);
- return new HoodieDefaultTimeline(getInstantsAsStream().filter(s -> s.isCompleted() || validActions.contains(s.getAction())), details);
+ public HoodieTimeline filterCompletedWriteAndCompactionInstants() {
+ return getWriteTimeline().filterCompletedAndCompactionInstants();
+ }
+
+ @Override
+ public HoodieDefaultTimeline filterCompletedAndRewriteInstants(HoodieTableMetaClient metaClient) {
Review Comment:
filterCompletedWriteAndRewriteInstants
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -165,12 +165,18 @@ public interface HoodieTimeline extends Serializable {
HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants();
+ /**
+ * This is the filtered write timeline that contains only completed and pending compaction instants.
+ * This is the timeline that is used in constructing file groups.
+ */
+ HoodieTimeline filterCompletedWriteAndCompactionInstants();
+
/**
* Timeline to just include completed commits or all rewrites like compaction, logcompaction and replace actions.
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants(HoodieTableMetaClient metaClient);
Review Comment:
whats the expected behavior of this api.
completed commits and all compaction and all replace actions
OR
only completed (writes, compaction, replace commits)
?
can you fix the naming and java docs accordingly. its kind of mis-leading
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -1298,6 +1369,14 @@ Stream<FileSlice> fetchLatestFileSlices(String partitionPath) {
.map(Option::get);
}
+ /**
+ * Default implementation for fetching latest file-slices for a partition path.
+ */
+ Stream<FileSlice> fetchLatestFileSlices(String partitionPath, boolean includePending) {
Review Comment:
access specifier?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -144,44 +172,54 @@ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingCompactionTran
*/
public static class TimelineDiffResult {
- private final List<HoodieInstant> newlySeenInstants;
+ private final List<HoodieInstant> newlySeenCompletedAndRewriteInstants;
private final List<HoodieInstant> finishedCompactionInstants;
Review Comment:
so we are not not interested in removed Compaction instants is it? curious to know why this is diff compared to log compactions.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -72,27 +82,75 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
&& instantPair.getValue().isCompleted())
.map(Pair::getKey).collect(Collectors.toList());
- newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Collect new instants that are not present in oldCompletedAndRewriteTimeline
+ // but are present in new active timeline.
+ List<HoodieInstant> newCompletedAndRewriteInstants = new ArrayList<>();
+ newCompletedAndRewriteTimeline.getInstantsAsStream()
+ .filter(instant -> !oldTimelineInstants.contains(instant))
+ .forEach(newCompletedAndRewriteInstants::add);
- List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ // Check for log compaction commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants =
+ getPendingLogCompactionTransitions(oldTimeline, newCompletedAndRewriteTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> Pair.of(instantPair.getKey(), instantPair.getValue() != null))
+ .collect(Collectors.toList());
+
+ // Check if replacecommits are completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants =
+ getPendingReplaceCommitTransitions(oldTimeline, newCompletedAndRewriteTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> Pair.of(instantPair.getKey(), instantPair.getValue() != null))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, log-compaction, clustering)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newCompletedAndRewriteInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(
+ HoodieTimeline oldTimeline, HoodieTimeline newCompletedAndRewriteTimeline) {
+ Set<HoodieInstant> newTimelineInstants = newCompletedAndRewriteTimeline.getInstantsAsStream().collect(Collectors.toSet());
+
+ return oldTimeline.filterPendingReplaceTimeline().getInstantsAsStream().map(instant -> {
+ if (newTimelineInstants.contains(instant)) {
+ return Pair.of(instant, instant);
+ } else {
+ HoodieInstant completedReplaceCommit =
+ new HoodieInstant(State.COMPLETED, HoodieTimeline.REPLACE_COMMIT_ACTION, instant.getTimestamp());
+ if (newTimelineInstants.contains(completedReplaceCommit)) {
+ return Pair.of(instant, completedReplaceCommit);
+ }
+ HoodieInstant inflightRepaceCommit =
+ new HoodieInstant(State.INFLIGHT, HoodieTimeline.REPLACE_COMMIT_ACTION, instant.getTimestamp());
+ if (newTimelineInstants.contains(inflightRepaceCommit)) {
+ return Pair.of(instant, inflightRepaceCommit);
+ }
+ return Pair.<HoodieInstant, HoodieInstant>of(instant, null);
+ }
+ }).collect(Collectors.toList());
+ }
+
/**
* Getting pending log compaction transitions.
*/
- private static List<Pair<HoodieInstant, HoodieInstant>> getPendingLogCompactionTransitions(HoodieTimeline oldTimeline,
- HoodieTimeline newTimeline) {
- Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingLogCompactionTransitions(
+ HoodieTimeline oldTimeline, HoodieTimeline newCompletedAndRewriteTimeline) {
Review Comment:
what does newCompletedAndRewriteTimeline refer to?
completed writes and compaction commits and all rewrite instants (replace commit)?
or does it include all compaction instants too. then, I feel we shd fix the naming. it is mis-leading
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -90,9 +91,11 @@ public abstract class AbstractTableFileSystemView implements SyncableFileSystemV
protected HoodieTableMetaClient metaClient;
- // This is the commits timeline that will be visible for all views extending this view
- // This is nothing but the write timeline, which contains both ingestion and compaction(major and minor) writers.
- private HoodieTimeline visibleCommitsAndCompactionTimeline;
+ // Timeline determines whether a file slice is committed or not. This timeline object is attached
Review Comment:
can we fix the java docs.
```
// This is the commits timeline that will be visible for all views extending this view. This timeline contains completed write instants, all compaction instants(including pending).
This timeline object is attached to each HoodieFileGroup. Using this timeline object, HoodieFileGroup determines whether a fileSlice is committed or not.
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -170,7 +170,7 @@ public interface HoodieTimeline extends Serializable {
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants();
Review Comment:
generally, we have used rewrite to denote only replace commits. So, are we changing the definition ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -165,12 +165,18 @@ public interface HoodieTimeline extends Serializable {
HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants();
+ /**
+ * This is the filtered write timeline that contains only completed and pending compaction instants.
+ * This is the timeline that is used in constructing file groups.
+ */
+ HoodieTimeline filterCompletedWriteAndCompactionInstants();
+
/**
* Timeline to just include completed commits or all rewrites like compaction, logcompaction and replace actions.
*
* @return
*/
- HoodieTimeline filterCompletedInstantsOrRewriteTimeline();
+ HoodieTimeline filterCompletedAndRewriteInstants(HoodieTableMetaClient metaClient);
Review Comment:
are you introducing a new terminology called "rewrite" with this patch. or is there precedence already?
my understanding is, rewrite refers to only replace commits. but some apis in this patch, we are using rewrite to refer to compaction as well? can we ensure its uniform across the board.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -130,36 +141,50 @@ private void runIncrementalSync(HoodieTimeline timeline, TimelineDiffResult diff
});
// Now remove pending log compaction instants which were completed or removed
- diffResult.getFinishedOrRemovedLogCompactionInstants().stream().forEach(instant -> {
+ diffResult.getFinishedOrRemovedLogCompactionInstants().stream().forEach(instantPair -> {
+ try {
+ removePendingLogCompactionInstant(instantPair.getKey(), instantPair.getValue());
+ } catch (IOException e) {
+ throw new HoodieException(e);
+ }
+ });
+
+ // Now remove pending replace instants which were completed or removed
+ diffResult.getFinishedOrRemovedReplaceInstants().stream().forEach(instantPair -> {
try {
- removePendingLogCompactionInstant(instant);
+ removePendingFileGroupsInPendingClustering(instantPair.getKey(), instantPair.getValue());
} catch (IOException e) {
throw new HoodieException(e);
}
});
- // Add new completed instants found in the latest timeline, this also contains inflight instants.
- diffResult.getNewlySeenInstants().stream()
- .filter(instant -> instant.isCompleted()
- || instant.getAction().equals(HoodieTimeline.COMPACTION_ACTION)
- || instant.getAction().equals(HoodieTimeline.LOG_COMPACTION_ACTION))
+ HoodieTimeline newFileSystemViewTimeline =
+ newActiveTimeline.getFileSystemViewTimeline();
+ // Add new completed instants found in the latest timeline,
+ // this also contains inflight instants from rewrite timeline.
+ diffResult.getNewlySeenCompletedAndRewriteInstants()
.forEach(instant -> {
try {
if (instant.getAction().equals(HoodieTimeline.COMMIT_ACTION)
|| instant.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)) {
- addCommitInstant(timeline, instant);
+ addCommitInstant(newFileSystemViewTimeline, instant);
Review Comment:
wouldn't newActiveTimeline a super set of newFileSystemViewTimeline? can we use newActiveTimeline for all from 170L to 187 ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:
##########
@@ -181,17 +204,24 @@ private void removePendingCompactionInstant(HoodieInstant instant) throws IOExce
}
/**
- * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit,
- * so you no longer need to track them as pending.
+ * Remove Pending compaction instant. This is called when logcompaction is converted to delta commit or the log compaction
+ * operation is failed, so it is no longer need to be tracked as pending.
*
* @param instant Log Compaction Instant to be removed
+ * @param isSuccessful true if log compaction operation is successful, false if the operation is failed and rollbacked.
*/
- private void removePendingLogCompactionInstant(HoodieInstant instant) throws IOException {
- LOG.info("Removing completed log compaction instant (" + instant + ")");
- HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
- removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
- .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
- CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ private void removePendingLogCompactionInstant(HoodieInstant instant, boolean isSuccessful) throws IOException {
+ if (isSuccessful) {
+ LOG.info("Removing completed log compaction instant (" + instant + ")");
+ HoodieCompactionPlan plan = CompactionUtils.getLogCompactionPlan(metaClient, instant.getTimestamp());
+ removePendingLogCompactionOperations(CompactionUtils.getPendingCompactionOperations(instant, plan)
+ .map(instantPair -> Pair.of(instantPair.getValue().getKey(),
+ CompactionOperation.convertFromAvroRecordInstance(instantPair.getValue().getValue()))));
+ } else {
+ LOG.info("Removing failed log compaction instant (" + instant + ")");
+ removePendingLogCompactionOperations(instant.getTimestamp());
Review Comment:
I see why we are not interested in completed compaction instants now.
but wouldn't the logic applicable to clustering commits as well then ? lets sync up f2f.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1613826553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189)
* f154ee335eb307e2bcffd895cfd95bfb1f417a1e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264806760
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:
##########
@@ -178,7 +203,15 @@ public Option<HoodieBaseFile> getLatestDataFile() {
* Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
*/
public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime) {
- return Option.fromJavaOptional(getAllFileSlices().filter(slice -> compareTimestamps(slice.getBaseInstantTime(), LESSER_THAN_OR_EQUALS, maxInstantTime)).findFirst());
+ return getLatestFileSliceBeforeOrOn(maxInstantTime, false);
+ }
+
+ /**
+ * Obtain the latest file slice, upto a instantTime i.e <= maxInstantTime.
+ */
+ public Option<FileSlice> getLatestFileSliceBeforeOrOn(String maxInstantTime, boolean includePending) {
Review Comment:
Yeah, the timeline of the filesystem view is a chaos, can we avoid introducing these testing APIs on the filesystem view and just refactors the tests as correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1612915077
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1244878470
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
Review Comment:
Compaction plans are immutable plans, so once they are created even though the execution of the plan is not completed, new log files are still written to new file slice. Whereas Log compaction and clustering can be mutable, they can be removed so the logic varies for them.
Compaction plans are removed only during a restore operation, so it is ok to do incremental file sync such cases, whereas for other 2 action types, incremental file sync should be able to handle the transitions from inflight to completed or inflight to rollback etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1244878470
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
Review Comment:
Compaction plans are immutable plans, so once they are created even though the execution of the plan is not completed, new log files are still written to new file slice. Whereas Log compaction and clustering can be mutable, they can be removed so the logic varies.
Compaction plans are removed only during a restore operation, so it is ok to do incremental file sync such cases, whereas for other 2 action types, incremental file sync should be able to handle the transitions from inflight to completed or inflight to rollback etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264009862
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:
##########
@@ -74,19 +74,62 @@ public static TimelineDiffResult getNewInstantsForIncrementalSync(HoodieTimeline
newTimeline.getInstantsAsStream().filter(instant -> !oldTimelineInstants.contains(instant)).forEach(newInstants::add);
+ // Check for log compaction commits completed or removed.
List<Pair<HoodieInstant, HoodieInstant>> logCompactionInstants = getPendingLogCompactionTransitions(oldTimeline, newTimeline);
- List<HoodieInstant> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedLogCompactionInstants = logCompactionInstants.stream()
.filter(instantPair -> !instantPair.getKey().isCompleted()
&& (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
- .map(Pair::getKey).collect(Collectors.toList());
- return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants, true);
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // Check for replace commits completed or removed.
+ List<Pair<HoodieInstant, HoodieInstant>> replaceCommitInstants = getPendingReplaceCommitTransitions(oldTimeline, newTimeline);
+ List<Pair<HoodieInstant, Boolean>> finishedOrRemovedReplaceCommitInstants = replaceCommitInstants.stream()
+ .filter(instantPair -> !instantPair.getKey().isCompleted()
+ && (instantPair.getValue() == null || instantPair.getValue().isCompleted()))
+ .map(instantPair -> (instantPair.getValue() == null)
+ ? Pair.of(instantPair.getKey(), false) : Pair.of(instantPair.getKey(), true))
+ .collect(Collectors.toList());
+
+ // New instants will contains instants that are newly completed commits or newly created pending rewrite commits
+ // (i.e. compaction, logcompaciton, replacecommit)
+ // Finished or removed rewrite commits are handled independently.
+ return new TimelineDiffResult(newInstants, finishedCompactionInstants, finishedOrRemovedLogCompactionInstants,
+ finishedOrRemovedReplaceCommitInstants, true);
} else {
// One or more timelines is empty
LOG.warn("One or more timelines is empty");
return TimelineDiffResult.UNSAFE_SYNC_RESULT;
}
}
+ /**
+ * Get pending replacecommit transitions.
+ */
+ private static List<Pair<HoodieInstant, HoodieInstant>> getPendingReplaceCommitTransitions(HoodieTimeline oldTimeline,
+ HoodieTimeline newTimeline) {
+ Set<HoodieInstant> newTimelineInstants = newTimeline.getInstantsAsStream().collect(Collectors.toSet());
+
+ return oldTimeline.filterPendingReplaceTimeline().getInstantsAsStream().map(instant -> {
+ if (newTimelineInstants.contains(instant)) {
Review Comment:
Created [HUDI-6538](https://issues.apache.org/jira/browse/HUDI-6538) as followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636939942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 191423c89583c99c085dd527eab46ab72894d1df Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612)
* 7ffcac1db2479936bdee038951fab00071597d26 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637173575
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
}, {
"hash" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18629",
"triggerID" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"triggerType" : "PUSH"
}, {
"hash" : "d8073e7e042d879324f45f26f00d9d059c1e5d0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18630",
"triggerID" : "d8073e7e042d879324f45f26f00d9d059c1e5d0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* 0964e4bc4aeb82ba6230920736c3d432bfdb5d59 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18629)
* d8073e7e042d879324f45f26f00d9d059c1e5d0a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18630)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1637203502
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
}, {
"hash" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18588",
"triggerID" : "07eb1aa79162259f3ac79e61bec621f68afb5551",
"triggerType" : "PUSH"
}, {
"hash" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18592",
"triggerID" : "578401a45b7c5ffbd9360de3bd3e18c362b4b2b5",
"triggerType" : "PUSH"
}, {
"hash" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18604",
"triggerID" : "ea6504e78fbb1c776687d3632c5875e74070cebd",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"triggerType" : "PUSH"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18607",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "de9fca47509129e13c9b3a422261e8c55978faa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18606",
"triggerID" : "1636636242",
"triggerType" : "MANUAL"
}, {
"hash" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18608",
"triggerID" : "3e318aa173aaa30e984554380d41c7706b1e7061",
"triggerType" : "PUSH"
}, {
"hash" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18610",
"triggerID" : "488e22d276930ac0836fd54d5c3cd5623faeb544",
"triggerType" : "PUSH"
}, {
"hash" : "191423c89583c99c085dd527eab46ab72894d1df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18612",
"triggerID" : "191423c89583c99c085dd527eab46ab72894d1df",
"triggerType" : "PUSH"
}, {
"hash" : "7ffcac1db2479936bdee038951fab00071597d26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18618",
"triggerID" : "7ffcac1db2479936bdee038951fab00071597d26",
"triggerType" : "PUSH"
}, {
"hash" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fe9c258bac6a569434d5322e4b74254b6cbb80c",
"triggerType" : "PUSH"
}, {
"hash" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18625",
"triggerID" : "3d459ad5a5b87ce1d86cbbc5c9dfe10bf65db1ca",
"triggerType" : "PUSH"
}, {
"hash" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18626",
"triggerID" : "34eafed1c209350b1bb019bf6b5d6f0f9f1f9d48",
"triggerType" : "PUSH"
}, {
"hash" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18629",
"triggerID" : "0964e4bc4aeb82ba6230920736c3d432bfdb5d59",
"triggerType" : "PUSH"
}, {
"hash" : "d8073e7e042d879324f45f26f00d9d059c1e5d0a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18630",
"triggerID" : "d8073e7e042d879324f45f26f00d9d059c1e5d0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 4fe9c258bac6a569434d5322e4b74254b6cbb80c UNKNOWN
* d8073e7e042d879324f45f26f00d9d059c1e5d0a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18630)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1614254342
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f154ee335eb307e2bcffd895cfd95bfb1f417a1e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218)
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 78b7acc447a6cdadccf1b0ca57e1cc634233c879 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9007:
URL: https://github.com/apache/hudi/pull/9007#discussion_r1264809549
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -163,14 +163,19 @@ public interface HoodieTimeline extends Serializable {
*/
HoodieTimeline filterCompletedAndCompactionInstants();
- HoodieTimeline filterCompletedOrMajorOrMinorCompactionInstants();
+ /**
+ * File system view that is used for constructing file groups.
+ * This is the filtered timeline contains completed instants that are part of either
+ * write timeline or savepoint timeline. Along with this it also includes any pending compaction commits.
+ */
+ HoodieTimeline getFileSystemViewTimeline();
Review Comment:
I'm confused by the notion, what is even a file system view timeline? File system view depends on the timeline for all kinds of file view APIs, that is a clear concept to developers, but what is a file system view timeline, it seems you are bringing in new notions?
Maybe you have good reason for these not including 'rollback and restore' restrictions, but it is very subjective and also decided by specific use case and it is hard to form a standard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1635928412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17939",
"triggerID" : "add83e6b63f7ba3d2f791fbd4a761ddbb83d61c3",
"triggerType" : "PUSH"
}, {
"hash" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17958",
"triggerID" : "3360fa18333a0097fa762824f02eb9cd6c4bad5d",
"triggerType" : "PUSH"
}, {
"hash" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18189",
"triggerID" : "3b6d13a83efdae5e46eebe9ae168ba7e0d8e9f34",
"triggerType" : "PUSH"
}, {
"hash" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18218",
"triggerID" : "f154ee335eb307e2bcffd895cfd95bfb1f417a1e",
"triggerType" : "PUSH"
}, {
"hash" : "c221efd733a444258780949b698830c2cef47931",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c221efd733a444258780949b698830c2cef47931",
"triggerType" : "PUSH"
}, {
"hash" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233",
"triggerID" : "78b7acc447a6cdadccf1b0ca57e1cc634233c879",
"triggerType" : "PUSH"
}, {
"hash" : "26b3151e371774b3e99324bd9c305157fcde5789",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585",
"triggerID" : "26b3151e371774b3e99324bd9c305157fcde5789",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c221efd733a444258780949b698830c2cef47931 UNKNOWN
* 78b7acc447a6cdadccf1b0ca57e1cc634233c879 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18233)
* 26b3151e371774b3e99324bd9c305157fcde5789 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] suryaprasanna commented on pull request #9007: [HUDI-6405] Fix incremental file sync for clustering and logcompaction
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on PR #9007:
URL: https://github.com/apache/hudi/pull/9007#issuecomment-1636636242
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org