You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2023/10/16 16:27:10 UTC
[PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
codope opened a new pull request, #9871:
URL: https://github.com/apache/hudi/pull/9871
### Change Logs
Metadata table (MDT) has special handling for compaction. This PR ensures MDT compaction is handled in completion time based filesystem view. Previously, out-of-rder commit tests were disabled. This PR now enables that test to show the file slicing is happens correctly.
NOTE: This PR does not remove the limitation of single writer concurrency mode from MDT. That will be tackled separately.
### Impact
Support out of order commits due to compaction in MDT with completion time view.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1361415820
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -126,14 +127,18 @@ public Option<String> getCompletionTime(String baseInstant, String instantTime)
if (completionTimeOpt.isPresent()) {
String completionTime = completionTimeOpt.get();
if (completionTime.length() != baseInstant.length()) {
+ // MDT uses compaction instant time as [delta_instant] + "001".
+ // For example, "20210101000000_001" is the compaction instant time for "20210101000000".
+ // However, in reality, MDT compaction completes in MDT timeline before the deltacommit in data timeline.
+ if (baseInstant.length() - instantTime.length() == 3 && hasCompactionOperationSuffix(baseInstant)) {
+ return completionTimeOpt;
Review Comment:
Do you think we should use directly its own real instant time for MDT compaction instead of suffixing the delta commit ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1764887056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4087ad6ee9f83d314874de0703eb2232abfcd7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1785754612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0a1258092388ff7d2ac67b8de7180be25a2137e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402)
* 975e7bcd435002af8902a17bf3500accb437d959 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1378302314
##########
hudi-client/hudi-java-client/src/test/java/org/apache/hudi/client/TestJavaHoodieBackedMetadata.java:
##########
@@ -525,7 +523,6 @@ public void testVirtualKeysInBaseFiles() throws Exception {
HoodieTableMetadata tableMetadata = metadata(writeConfig, context);
assertTrue(tableMetadata.getLatestCompactionTime().isPresent());
- assertEquals(tableMetadata.getLatestCompactionTime().get(), "0000003001");
Review Comment:
Since instant time is being generated at runtime (in HoodieBackedTableMetadataWriter), we are not able to assert it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1785820498
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0a1258092388ff7d2ac67b8de7180be25a2137e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402)
* 975e7bcd435002af8902a17bf3500accb437d959 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1364766424
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -107,46 +106,13 @@ public boolean isCompletedBefore(String baseInstant, String instantTime) {
* Returns whether the give instant time {@code instantTime} is sliced after or on the base instant {@code baseInstant}.
*/
public boolean isSlicedAfterOrOn(String baseInstant, String instantTime) {
- Option<String> completionTimeOpt = getCompletionTime(baseInstant, instantTime);
+ Option<String> completionTimeOpt = getCompletionTime(instantTime);
if (completionTimeOpt.isPresent()) {
return HoodieTimeline.compareTimestamps(completionTimeOpt.get(), GREATER_THAN_OR_EQUALS, baseInstant);
}
return true;
}
- /**
- * Get completion time with a base instant time as a reference to fix the compatibility.
- *
- * @param baseInstant The base instant
- * @param instantTime The instant time to query the completion time with
- *
- * @return Probability fixed completion time.
- */
- public Option<String> getCompletionTime(String baseInstant, String instantTime) {
- Option<String> completionTimeOpt = getCompletionTime(instantTime);
- if (completionTimeOpt.isPresent()) {
Review Comment:
We can not remove this unless you fix all the test cases that use Integer string as the compaction instant time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1792464812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
}, {
"hash" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626",
"triggerID" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"triggerType" : "PUSH"
}, {
"hash" : "65517de0a82e58ff668b414b9c9c7fc7abd08520",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20670",
"triggerID" : "65517de0a82e58ff668b414b9c9c7fc7abd08520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a6b85794428dcd7a7b45f28430bfb5f6c42fc910 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626)
* 65517de0a82e58ff668b414b9c9c7fc7abd08520 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20670)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1382548171
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -228,9 +228,10 @@ public HoodieDefaultTimeline findInstantsInRangeByCompletionTime(String startTs,
@Override
public HoodieDefaultTimeline findInstantsModifiedAfterByCompletionTime(String instantTime) {
return new HoodieDefaultTimeline(instants.stream()
- .filter(s -> s.getCompletionTime() != null
- && HoodieTimeline.compareTimestamps(s.getCompletionTime(), GREATER_THAN, instantTime)
- && !s.getTimestamp().equals(instantTime)), details);
+ // either pending or completionTime greater than instantTime
+ .filter(s -> (s.getCompletionTime() == null && compareTimestamps(s.getTimestamp(), GREATER_THAN, instantTime))
+ || (compareTimestamps(s.getCompletionTime(), GREATER_THAN, instantTime) && !s.getTimestamp().equals(instantTime))),
Review Comment:
my bad.. i meant to keep it.. added now. thanks for pointing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1769198316
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61761397a6d981c465c2b05a020666b3382647c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1789919679
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
}, {
"hash" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea5ea6e38a4eb27a0a061031804dabc42824945c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609)
* a6b85794428dcd7a7b45f28430bfb5f6c42fc910 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1770932984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61761397a6d981c465c2b05a020666b3382647c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384)
* 6c99a1f414f82b85331fd1b256325641b0417f2e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1771360969
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6c99a1f414f82b85331fd1b256325641b0417f2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397)
* f0a1258092388ff7d2ac67b8de7180be25a2137e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1771375858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6c99a1f414f82b85331fd1b256325641b0417f2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397)
* f0a1258092388ff7d2ac67b8de7180be25a2137e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1771010540
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6c99a1f414f82b85331fd1b256325641b0417f2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1769109683
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4087ad6ee9f83d314874de0703eb2232abfcd7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352)
* 61761397a6d981c465c2b05a020666b3382647c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1382533555
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -228,9 +228,10 @@ public HoodieDefaultTimeline findInstantsInRangeByCompletionTime(String startTs,
@Override
public HoodieDefaultTimeline findInstantsModifiedAfterByCompletionTime(String instantTime) {
return new HoodieDefaultTimeline(instants.stream()
- .filter(s -> s.getCompletionTime() != null
- && HoodieTimeline.compareTimestamps(s.getCompletionTime(), GREATER_THAN, instantTime)
- && !s.getTimestamp().equals(instantTime)), details);
+ // either pending or completionTime greater than instantTime
+ .filter(s -> (s.getCompletionTime() == null && compareTimestamps(s.getTimestamp(), GREATER_THAN, instantTime))
+ || (compareTimestamps(s.getCompletionTime(), GREATER_THAN, instantTime) && !s.getTimestamp().equals(instantTime))),
Review Comment:
The `s.getCompletionTime() != null` check is missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1378742958
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -270,13 +270,41 @@ private List<HoodieInstant> getCommitInstantsToArchive() throws IOException {
// stop at first savepoint commit
return !firstSavepoint.isPresent() || compareTimestamps(s.getTimestamp(), LESSER_THAN, firstSavepoint.get().getTimestamp());
}
- }).filter(s -> earliestInstantToRetain
- .map(instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp()))
+ })
+ .filter(s -> earliestInstantToRetain.map(
+ instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp())
+ // for the compaction c2 in metadata table triggered by commit c1 in data table,
+ // c2.getTimestamp() > c1.getTimestamp() because: c2 happens before c1 completes,
+ // and we are generating new instant time for c2 after c1 has started. Effectively,
Review Comment:
Shouldn’t we fix line 208 ~ line 222 instead?
And this issue also exists for the master code, the compaction instant is delta_commit + “001” which happens before delta_commit with a greater instant time, but we still only compares the instant directly by using the compaction instant time.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -270,13 +270,41 @@ private List<HoodieInstant> getCommitInstantsToArchive() throws IOException {
// stop at first savepoint commit
return !firstSavepoint.isPresent() || compareTimestamps(s.getTimestamp(), LESSER_THAN, firstSavepoint.get().getTimestamp());
}
- }).filter(s -> earliestInstantToRetain
- .map(instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp()))
+ })
+ .filter(s -> earliestInstantToRetain.map(
+ instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp())
+ // for the compaction c2 in metadata table triggered by commit c1 in data table,
+ // c2.getTimestamp() > c1.getTimestamp() because: c2 happens before c1 completes,
+ // and we are generating new instant time for c2 after c1 has started. Effectively,
Review Comment:
Shouldn’t we fix line 208 ~ line 222 instead?
And this issue also exists for the master code, the compaction instant is delta_commit + “001” which happens before delta_commit with a greater instant time, but we still only compares the instant directly by using the compaction instant time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1764872147
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4087ad6ee9f83d314874de0703eb2232abfcd7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1771511294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0a1258092388ff7d2ac67b8de7180be25a2137e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #9871:
URL: https://github.com/apache/hudi/pull/9871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1788589049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 975e7bcd435002af8902a17bf3500accb437d959 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569)
* ea5ea6e38a4eb27a0a061031804dabc42824945c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1789967114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
}, {
"hash" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626",
"triggerID" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a6b85794428dcd7a7b45f28430bfb5f6c42fc910 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1378336573
##########
hudi-client/hudi-java-client/src/test/java/org/apache/hudi/client/TestJavaHoodieBackedMetadata.java:
##########
@@ -525,7 +523,6 @@ public void testVirtualKeysInBaseFiles() throws Exception {
HoodieTableMetadata tableMetadata = metadata(writeConfig, context);
assertTrue(tableMetadata.getLatestCompactionTime().isPresent());
- assertEquals(tableMetadata.getLatestCompactionTime().get(), "0000003001");
Review Comment:
Yeah, I think it does not make much sense to assert the compaction instant string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1379469734
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -270,13 +270,41 @@ private List<HoodieInstant> getCommitInstantsToArchive() throws IOException {
// stop at first savepoint commit
return !firstSavepoint.isPresent() || compareTimestamps(s.getTimestamp(), LESSER_THAN, firstSavepoint.get().getTimestamp());
}
- }).filter(s -> earliestInstantToRetain
- .map(instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp()))
+ })
+ .filter(s -> earliestInstantToRetain.map(
+ instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp())
+ // for the compaction c2 in metadata table triggered by commit c1 in data table,
+ // c2.getTimestamp() > c1.getTimestamp() because: c2 happens before c1 completes,
+ // and we are generating new instant time for c2 after c1 has started. Effectively,
Review Comment:
> Lines 208-222 is fine as it is trying to get the earliest candidate instant to retain. The main filtering happens after all such candidate instants have been collected. I think the fix should be where the filtering happens.
Still think line 208 ~ 222 is the right place to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1789926631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
}, {
"hash" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626",
"triggerID" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea5ea6e38a4eb27a0a061031804dabc42824945c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609)
* a6b85794428dcd7a7b45f28430bfb5f6c42fc910 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1381672056
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/io/TestHoodieTimelineArchiver.java:
##########
@@ -1221,35 +1222,6 @@ public void testArchiveTableWithMetadataTableCompaction() throws Exception {
commitsAfterArchival);
}
- @Test
- public void testArchiveCommitsWithCompactionCommitInMetadataTableTimeline() throws Exception {
Review Comment:
this test is redundant and uses dummy instants. we're already checking archival with compaction in metadata table in several other tests such as `testArchiveTableWithMetadataTableCompaction` and `testArchivalAndCompactionInMetadataTable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1765058691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4087ad6ee9f83d314874de0703eb2232abfcd7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1365448346
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -107,46 +106,13 @@ public boolean isCompletedBefore(String baseInstant, String instantTime) {
* Returns whether the give instant time {@code instantTime} is sliced after or on the base instant {@code baseInstant}.
*/
public boolean isSlicedAfterOrOn(String baseInstant, String instantTime) {
- Option<String> completionTimeOpt = getCompletionTime(baseInstant, instantTime);
+ Option<String> completionTimeOpt = getCompletionTime(instantTime);
if (completionTimeOpt.isPresent()) {
return HoodieTimeline.compareTimestamps(completionTimeOpt.get(), GREATER_THAN_OR_EQUALS, baseInstant);
}
return true;
}
- /**
- * Get completion time with a base instant time as a reference to fix the compatibility.
- *
- * @param baseInstant The base instant
- * @param instantTime The instant time to query the completion time with
- *
- * @return Probability fixed completion time.
- */
- public Option<String> getCompletionTime(String baseInstant, String instantTime) {
- Option<String> completionTimeOpt = getCompletionTime(instantTime);
- if (completionTimeOpt.isPresent()) {
Review Comment:
yeah, there are too many tests doing that. i've kept this condition just for tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1365447251
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -126,14 +127,18 @@ public Option<String> getCompletionTime(String baseInstant, String instantTime)
if (completionTimeOpt.isPresent()) {
String completionTime = completionTimeOpt.get();
if (completionTime.length() != baseInstant.length()) {
+ // MDT uses compaction instant time as [delta_instant] + "001".
+ // For example, "20210101000000_001" is the compaction instant time for "20210101000000".
+ // However, in reality, MDT compaction completes in MDT timeline before the deltacommit in data timeline.
+ if (baseInstant.length() - instantTime.length() == 3 && hasCompactionOperationSuffix(baseInstant)) {
+ return completionTimeOpt;
Review Comment:
Done. Updated in `HoodieBackedTableMetadataWriter#compactIfNecessary`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1792447778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
}, {
"hash" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626",
"triggerID" : "a6b85794428dcd7a7b45f28430bfb5f6c42fc910",
"triggerType" : "PUSH"
}, {
"hash" : "65517de0a82e58ff668b414b9c9c7fc7abd08520",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "65517de0a82e58ff668b414b9c9c7fc7abd08520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a6b85794428dcd7a7b45f28430bfb5f6c42fc910 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20626)
* 65517de0a82e58ff668b414b9c9c7fc7abd08520 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1789151084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea5ea6e38a4eb27a0a061031804dabc42824945c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1793343188
Thanks for the contribution @codope , I have reviewed and created a patch:
[2461.patch.zip](https://github.com/apache/hudi/files/13255858/2461.patch.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1376941626
##########
hudi-client/hudi-java-client/src/test/java/org/apache/hudi/client/TestJavaHoodieBackedMetadata.java:
##########
@@ -525,7 +523,6 @@ public void testVirtualKeysInBaseFiles() throws Exception {
HoodieTableMetadata tableMetadata = metadata(writeConfig, context);
assertTrue(tableMetadata.getLatestCompactionTime().isPresent());
- assertEquals(tableMetadata.getLatestCompactionTime().get(), "0000003001");
Review Comment:
Should we remove the instant time equals check or change the assert val?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1788299517
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 975e7bcd435002af8902a17bf3500accb437d959 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1788599391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20397",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
}, {
"hash" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20402",
"triggerID" : "f0a1258092388ff7d2ac67b8de7180be25a2137e",
"triggerType" : "PUSH"
}, {
"hash" : "975e7bcd435002af8902a17bf3500accb437d959",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569",
"triggerID" : "975e7bcd435002af8902a17bf3500accb437d959",
"triggerType" : "PUSH"
}, {
"hash" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609",
"triggerID" : "ea5ea6e38a4eb27a0a061031804dabc42824945c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 975e7bcd435002af8902a17bf3500accb437d959 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20569)
* ea5ea6e38a4eb27a0a061031804dabc42824945c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20609)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1378785387
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -270,13 +270,41 @@ private List<HoodieInstant> getCommitInstantsToArchive() throws IOException {
// stop at first savepoint commit
return !firstSavepoint.isPresent() || compareTimestamps(s.getTimestamp(), LESSER_THAN, firstSavepoint.get().getTimestamp());
}
- }).filter(s -> earliestInstantToRetain
- .map(instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp()))
+ })
+ .filter(s -> earliestInstantToRetain.map(
+ instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp())
+ // for the compaction c2 in metadata table triggered by commit c1 in data table,
+ // c2.getTimestamp() > c1.getTimestamp() because: c2 happens before c1 completes,
+ // and we are generating new instant time for c2 after c1 has started. Effectively,
Review Comment:
Lines 208-222 is fine as it is trying to get the earliest candidate instant to retain. The main filtering happens after all such candidate instants have been collected. I think the fix should be where the filtering happens.
> And this issue also exists for the master code, the compaction instant is delta_commit + “001” which happens before delta_commit with a greater instant time
That is not the case. Let's say the data table had a commit 00000006 and now 00000007 happened which triggered the compaction in MDT. As per master code, first the compaction of MDT will be done, which will take the instant time 00000006001 and not 00000007001, and 00000006001 < 00000007. In this PR, we have changed the compaction instant time to be the new instant time at that point. So, it's always greater than the start time of the commit that triggered it.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -270,13 +270,41 @@ private List<HoodieInstant> getCommitInstantsToArchive() throws IOException {
// stop at first savepoint commit
return !firstSavepoint.isPresent() || compareTimestamps(s.getTimestamp(), LESSER_THAN, firstSavepoint.get().getTimestamp());
}
- }).filter(s -> earliestInstantToRetain
- .map(instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp()))
+ })
+ .filter(s -> earliestInstantToRetain.map(
+ instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp())
+ // for the compaction c2 in metadata table triggered by commit c1 in data table,
+ // c2.getTimestamp() > c1.getTimestamp() because: c2 happens before c1 completes,
+ // and we are generating new instant time for c2 after c1 has started. Effectively,
Review Comment:
Lines 208-222 is fine as it is trying to get the earliest candidate instant to retain. The main filtering happens after all such candidate instants have been collected. I think the fix should be where the filtering happens.
> And this issue also exists for the master code, the compaction instant is delta_commit + “001” which happens before delta_commit with a greater instant time
That is not the case. Let's say the data table had a commit 00000006 and now 00000007 happened which triggered the compaction in MDT. As per master code, first the compaction of MDT will be done, which will take the instant time 00000006001 and not 00000007001, and 00000006001 < 00000007. In this PR, we have changed the compaction instant time to be the new instant time at that point. So, it's always greater than the start time of the commit that triggered it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1379469734
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -270,13 +270,41 @@ private List<HoodieInstant> getCommitInstantsToArchive() throws IOException {
// stop at first savepoint commit
return !firstSavepoint.isPresent() || compareTimestamps(s.getTimestamp(), LESSER_THAN, firstSavepoint.get().getTimestamp());
}
- }).filter(s -> earliestInstantToRetain
- .map(instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp()))
+ })
+ .filter(s -> earliestInstantToRetain.map(
+ instant -> compareTimestamps(s.getTimestamp(), LESSER_THAN, instant.getTimestamp())
+ // for the compaction c2 in metadata table triggered by commit c1 in data table,
+ // c2.getTimestamp() > c1.getTimestamp() because: c2 happens before c1 completes,
+ // and we are generating new instant time for c2 after c1 has started. Effectively,
Review Comment:
> Lines 208-222 is fine as it is trying to get the earliest candidate instant to retain. The main filtering happens after all such candidate instants have been collected. I think the fix should be where the filtering happens.
Still think line 208 ~ 222 is the right place to fix. It actually does the similar thing before by filtering the instants with MDT compaction instant, why not just fix it by using the completion time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1769048421
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4087ad6ee9f83d314874de0703eb2232abfcd7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352)
* 61761397a6d981c465c2b05a020666b3382647c8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9871:
URL: https://github.com/apache/hudi/pull/9871#issuecomment-1770917183
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20352",
"triggerID" : "f4087ad6ee9f83d314874de0703eb2232abfcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "61761397a6d981c465c2b05a020666b3382647c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384",
"triggerID" : "61761397a6d981c465c2b05a020666b3382647c8",
"triggerType" : "PUSH"
}, {
"hash" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6c99a1f414f82b85331fd1b256325641b0417f2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61761397a6d981c465c2b05a020666b3382647c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20384)
* 6c99a1f414f82b85331fd1b256325641b0417f2e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-2461] Support out of order commits in MDT with completion time view [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9871:
URL: https://github.com/apache/hudi/pull/9871#discussion_r1365814203
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1038,7 +1038,7 @@ protected void validateRollback(
// are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
final String compactionInstantTime = compactionInstant.getTimestamp();
- if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitToRollbackInstantTime, compactionInstantTime)) {
+ if (commitToRollbackInstantTime.length() == compactionInstantTime.length() && HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitToRollbackInstantTime, compactionInstantTime)) {
Review Comment:
now that the PR changed the MDT compaction time to a regular instant, we have to do this for tests because in tests we have rollback instants like "000001".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org