You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/11 20:24:22 UTC
[GitHub] [hudi] pratyakshsharma opened a new pull request, #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
pratyakshsharma opened a new pull request, #6926:
URL: https://github.com/apache/hudi/pull/6926
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275305454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7a1e1f6b57cd4d72e9126a4a82698159d785b07 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140)
* 711ed80f8c66f3c66e1c925a47b396011c4c4e98 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275833459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 711ed80f8c66f3c66e1c925a47b396011c4c4e98 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275244379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7a1e1f6b57cd4d72e9126a4a82698159d785b07 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3676] Enhance tests for trigger clean every Nth commit [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-2027979187
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
}, {
"hash" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081",
"triggerID" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"triggerType" : "PUSH"
}, {
"hash" : "36c0fdaa872c9ccc14729b09169b647f3d88efa4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23057",
"triggerID" : "36c0fdaa872c9ccc14729b09169b647f3d88efa4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36c0fdaa872c9ccc14729b09169b647f3d88efa4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275309910
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7a1e1f6b57cd4d72e9126a4a82698159d785b07 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140)
* 711ed80f8c66f3c66e1c925a47b396011c4c4e98 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a diff in pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #6926:
URL: https://github.com/apache/hudi/pull/6926#discussion_r1134846459
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
+ public void testTriggerCleanEveryNthCommit(int minCommitsToTriggerClean) throws Exception {
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder().withPath(basePath)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(false).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.EAGER)
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+ .retainCommits(1)
+ .withMaxCommitsBeforeCleaning(minCommitsToTriggerClean)
+ .build()).build();
+
+ HoodieTestTable testTable = HoodieTestTable.of(metaClient);
Review Comment:
@pratyakshsharma : Can you pass in HoodieTestTable instance to the testcase. That way, you can test for both metadata enabled and disabled (since HoodieMetadataTestTable extends HoodieTestTable)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1295207937
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76bc62cf70c15699c11989eb9169bf41c3483974 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618)
* 620cd686b307a74af7f0b8b35279661c25fad24d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275657576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 711ed80f8c66f3c66e1c925a47b396011c4c4e98 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1605472412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
}, {
"hash" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081",
"triggerID" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275238740
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7a1e1f6b57cd4d72e9126a4a82698159d785b07 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a diff in pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on code in PR #6926:
URL: https://github.com/apache/hudi/pull/6926#discussion_r1007197479
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
Review Comment:
so this method is intended to supply the values for the config `hoodie.clean.max.commits` with the documentation `Number of commits after the last clean operation, before scheduling of a new clean is attempted.` Hence I kept this name. @yihua
I am open to hearing your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1295220612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76bc62cf70c15699c11989eb9169bf41c3483974 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618)
* 620cd686b307a74af7f0b8b35279661c25fad24d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1605356249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
}, {
"hash" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081",
"triggerID" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 620cd686b307a74af7f0b8b35279661c25fad24d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654)
* 5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1605348129
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
}, {
"hash" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 620cd686b307a74af7f0b8b35279661c25fad24d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654)
* 5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1294052235
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76bc62cf70c15699c11989eb9169bf41c3483974 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a diff in pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on code in PR #6926:
URL: https://github.com/apache/hudi/pull/6926#discussion_r1007200947
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
+ public void testTriggerCleanEveryNthCommit(int minCommitsToTriggerClean) throws Exception {
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder().withPath(basePath)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(false).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.EAGER)
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+ .retainCommits(1)
+ .withMaxCommitsBeforeCleaning(minCommitsToTriggerClean)
+ .build()).build();
+
+ HoodieTestTable testTable = HoodieTestTable.of(metaClient);
+ String p0 = "2020/01/01";
+ String p1 = "2020/01/02";
+
+ // make 1 commit, with 1 file per partition
+ String file1P0C0 = UUID.randomUUID().toString();
+ String file1P1C0 = UUID.randomUUID().toString();
+ testTable.addInflightCommit("00000000000001").withBaseFilesInPartition(p0, file1P0C0).withBaseFilesInPartition(p1, file1P1C0);
+
+ HoodieCommitMetadata commitMetadata = generateCommitMetadata("00000000000001",
+ Collections.unmodifiableMap(new HashMap<String, List<String>>() {
+ {
+ put(p0, CollectionUtils.createImmutableList(file1P0C0));
+ put(p1, CollectionUtils.createImmutableList(file1P1C0));
+ }
+ })
+ );
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000001"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ metaClient = HoodieTableMetaClient.reload(metaClient);
+ List<HoodieCleanStat> hoodieCleanStatsOne =
+ runCleaner(config, false, false, 2, true);
+ assertEquals(0, hoodieCleanStatsOne.size(), "Must not scan any partitions and clean any files");
+ assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertTrue(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+
+ // make next commit, with 1 insert & 1 update per partition
+ Map<String, String> partitionAndFileId002 = testTable.addInflightCommit("00000000000003").getFileIdsWithBaseFilesInPartitions(p0, p1);
+ String file2P0C1 = partitionAndFileId002.get(p0);
+ String file2P1C1 = partitionAndFileId002.get(p1);
+ testTable.forCommit("00000000000003").withBaseFilesInPartition(p0, file1P0C0).withBaseFilesInPartition(p1, file1P1C0);
+ commitMetadata = generateCommitMetadata("00000000000003", new HashMap<String, List<String>>() {
+ {
+ put(p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1));
+ put(p1, CollectionUtils.createImmutableList(file1P1C0, file2P1C1));
+ }
+ });
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000003"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+ List<HoodieCleanStat> hoodieCleanStatsTwo =
+ runCleaner(config, false, false, 4, true);
+ assertEquals(0, hoodieCleanStatsTwo.size(), "Must not clean any file. We have to keep 1 version before the latest commit time");
+ assertTrue(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertTrue(testTable.baseFileExists(p1, "00000000000003", file2P1C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertTrue(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+
+ // make next commit, with 2 updates to existing files, and 1 insert
+ String file3P0C2 = testTable.addInflightCommit("00000000000005")
+ .withBaseFilesInPartition(p0, file1P0C0)
+ .withBaseFilesInPartition(p0, file2P0C1)
+ .getFileIdsWithBaseFilesInPartitions(p0).get(p0);
+ commitMetadata = generateCommitMetadata("00000000000005",
+ CollectionUtils.createImmutableMap(
+ p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1, file3P0C2)));
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000005"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ List<HoodieCleanStat> hoodieCleanStatsThree =
+ runCleaner(config, false, false, 6, true);
+
+ switch (minCommitsToTriggerClean) {
+ case 1:
+ case 2:
+ case 3:
+ assertEquals(2, hoodieCleanStatsThree.size(), "Must clean at least 1 files");
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+ break;
+ case 4:
+ assertEquals(0, hoodieCleanStatsThree.size(),
+ "Must not clean any file. Only 3 commits have occurred, need at least 4 for cleaning to trigger");
+ assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ break;
+ default:
+ }
+
+ // make next commit, with 2 updates to existing files, and 1 insert
+ String file4P0C3 = testTable.addInflightCommit("00000000000007")
+ .withBaseFilesInPartition(p0, file1P0C0)
+ .withBaseFilesInPartition(p0, file2P0C1)
+ .getFileIdsWithBaseFilesInPartitions(p0).get(p0);
+ commitMetadata = generateCommitMetadata("00000000000007",
+ CollectionUtils.createImmutableMap(
+ p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1, file4P0C3)));
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000007"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ List<HoodieCleanStat> hoodieCleanStatsFour =
+ runCleaner(config, false, false, 8, true);
+ HoodieCleanStat partitionZeroCleanStat = getCleanStat(hoodieCleanStatsFour, p0);
+ HoodieCleanStat partitionOneCleanStat = getCleanStat(hoodieCleanStatsFour, p1);
+
+ switch (minCommitsToTriggerClean) {
+ case 1:
+ assertEquals(2, partitionZeroCleanStat.getSuccessDeleteFiles().size(), "Must clean at least one old file");
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ break;
+ case 2:
+ case 3:
+ assertEquals(0, hoodieCleanStatsFour.size(),
+ "Must not clean any file. Only one commit has happened since the last clean!");
+ break;
+ case 4:
+ assertEquals(3, partitionZeroCleanStat.getSuccessDeleteFiles().size(), "Must clean at least one old file");
+ assertEquals(1, partitionOneCleanStat.getSuccessDeleteFiles().size());
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ break;
+ default:
+ }
+
+ metaClient = HoodieTableMetaClient.reload(metaClient);
+
+ String file5P0C4 = testTable.addInflightCommit("00000000000009")
+ .withBaseFilesInPartition(p0, file1P0C0)
+ .withBaseFilesInPartition(p0, file2P0C1)
+ .getFileIdsWithBaseFilesInPartitions(p0).get(p0);
+ commitMetadata = generateCommitMetadata("00000000000009", CollectionUtils.createImmutableMap(
+ p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1, file5P0C4)));
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000009"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ List<HoodieCleanStat> hoodieCleanStatsFive =
+ runCleaner(config, false, false, 10, true);
+
+ switch (minCommitsToTriggerClean) {
+ case 1:
+ assertEquals(1, hoodieCleanStatsFive.size(), "Must clean at least one old file");
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file3P0C2));
+ assertTrue(testTable.baseFileExists(p0, "00000000000007", file1P0C0));
+ break;
+ case 2:
+ assertEquals(1, hoodieCleanStatsFive.size(), "Must clean files written in second and third commits");
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file3P0C2));
+ assertTrue(testTable.baseFileExists(p0, "00000000000007", file1P0C0));
+ break;
+ case 3:
+ assertEquals(0, hoodieCleanStatsFive.size(),
+ "Must not clean any file. Only two commits have happened since the last clean!");
+ break;
+ case 4:
+ assertEquals(0, hoodieCleanStatsFive.size(),
+ "Must not clean any file. Only one commit has happened since the last clean!");
+ break;
+ default:
+ }
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
+ public void testTriggerCleanEveryNthCommitWithMetadataTableEnabled(int minCommitsToTriggerClean) throws Exception {
+ //TODO: this has some bug and needs to be checked once.
+ }
Review Comment:
As per the above comment, I intend to implement this test case in this PR itself. Will do the needful soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a diff in pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on code in PR #6926:
URL: https://github.com/apache/hudi/pull/6926#discussion_r1007200255
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
+ public void testTriggerCleanEveryNthCommit(int minCommitsToTriggerClean) throws Exception {
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder().withPath(basePath)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(false).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.EAGER)
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+ .retainCommits(1)
+ .withMaxCommitsBeforeCleaning(minCommitsToTriggerClean)
+ .build()).build();
+
+ HoodieTestTable testTable = HoodieTestTable.of(metaClient);
Review Comment:
I see. I believe this is the missing piece I was looking for because the test case was failing in its current form with metadata table enabled. I am thinking of adding the other test case in this PR itself, so that we have a single PR for enhancing cleaner test cases. Open to hearing your thoughts though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3676] Enhance tests for trigger clean every Nth commit [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-2027956043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
}, {
"hash" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081",
"triggerID" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"triggerType" : "PUSH"
}, {
"hash" : "36c0fdaa872c9ccc14729b09169b647f3d88efa4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36c0fdaa872c9ccc14729b09169b647f3d88efa4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081)
* 36c0fdaa872c9ccc14729b09169b647f3d88efa4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275625059
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6926:
URL: https://github.com/apache/hudi/pull/6926#discussion_r1002301042
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
+ public void testTriggerCleanEveryNthCommit(int minCommitsToTriggerClean) throws Exception {
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder().withPath(basePath)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(false).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.EAGER)
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+ .retainCommits(1)
+ .withMaxCommitsBeforeCleaning(minCommitsToTriggerClean)
+ .build()).build();
+
+ HoodieTestTable testTable = HoodieTestTable.of(metaClient);
Review Comment:
To run this test with metadata table enabled, I believe you need to use `HoodieMetadataTestTable`. You can tackle this in a separate PR.
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
Review Comment:
nit: make the name consistent?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
+ public void testTriggerCleanEveryNthCommit(int minCommitsToTriggerClean) throws Exception {
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder().withPath(basePath)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(false).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.EAGER)
+ .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+ .retainCommits(1)
+ .withMaxCommitsBeforeCleaning(minCommitsToTriggerClean)
+ .build()).build();
+
+ HoodieTestTable testTable = HoodieTestTable.of(metaClient);
+ String p0 = "2020/01/01";
+ String p1 = "2020/01/02";
+
+ // make 1 commit, with 1 file per partition
+ String file1P0C0 = UUID.randomUUID().toString();
+ String file1P1C0 = UUID.randomUUID().toString();
+ testTable.addInflightCommit("00000000000001").withBaseFilesInPartition(p0, file1P0C0).withBaseFilesInPartition(p1, file1P1C0);
+
+ HoodieCommitMetadata commitMetadata = generateCommitMetadata("00000000000001",
+ Collections.unmodifiableMap(new HashMap<String, List<String>>() {
+ {
+ put(p0, CollectionUtils.createImmutableList(file1P0C0));
+ put(p1, CollectionUtils.createImmutableList(file1P1C0));
+ }
+ })
+ );
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000001"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ metaClient = HoodieTableMetaClient.reload(metaClient);
+ List<HoodieCleanStat> hoodieCleanStatsOne =
+ runCleaner(config, false, false, 2, true);
+ assertEquals(0, hoodieCleanStatsOne.size(), "Must not scan any partitions and clean any files");
+ assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertTrue(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+
+ // make next commit, with 1 insert & 1 update per partition
+ Map<String, String> partitionAndFileId002 = testTable.addInflightCommit("00000000000003").getFileIdsWithBaseFilesInPartitions(p0, p1);
+ String file2P0C1 = partitionAndFileId002.get(p0);
+ String file2P1C1 = partitionAndFileId002.get(p1);
+ testTable.forCommit("00000000000003").withBaseFilesInPartition(p0, file1P0C0).withBaseFilesInPartition(p1, file1P1C0);
+ commitMetadata = generateCommitMetadata("00000000000003", new HashMap<String, List<String>>() {
+ {
+ put(p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1));
+ put(p1, CollectionUtils.createImmutableList(file1P1C0, file2P1C1));
+ }
+ });
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000003"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+ List<HoodieCleanStat> hoodieCleanStatsTwo =
+ runCleaner(config, false, false, 4, true);
+ assertEquals(0, hoodieCleanStatsTwo.size(), "Must not clean any file. We have to keep 1 version before the latest commit time");
+ assertTrue(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertTrue(testTable.baseFileExists(p1, "00000000000003", file2P1C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertTrue(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+
+ // make next commit, with 2 updates to existing files, and 1 insert
+ String file3P0C2 = testTable.addInflightCommit("00000000000005")
+ .withBaseFilesInPartition(p0, file1P0C0)
+ .withBaseFilesInPartition(p0, file2P0C1)
+ .getFileIdsWithBaseFilesInPartitions(p0).get(p0);
+ commitMetadata = generateCommitMetadata("00000000000005",
+ CollectionUtils.createImmutableMap(
+ p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1, file3P0C2)));
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000005"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ List<HoodieCleanStat> hoodieCleanStatsThree =
+ runCleaner(config, false, false, 6, true);
+
+ switch (minCommitsToTriggerClean) {
+ case 1:
+ case 2:
+ case 3:
+ assertEquals(2, hoodieCleanStatsThree.size(), "Must clean at least 1 files");
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+ break;
+ case 4:
+ assertEquals(0, hoodieCleanStatsThree.size(),
+ "Must not clean any file. Only 3 commits have occurred, need at least 4 for cleaning to trigger");
+ assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ break;
+ default:
+ }
+
+ // make next commit, with 2 updates to existing files, and 1 insert
+ String file4P0C3 = testTable.addInflightCommit("00000000000007")
+ .withBaseFilesInPartition(p0, file1P0C0)
+ .withBaseFilesInPartition(p0, file2P0C1)
+ .getFileIdsWithBaseFilesInPartitions(p0).get(p0);
+ commitMetadata = generateCommitMetadata("00000000000007",
+ CollectionUtils.createImmutableMap(
+ p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1, file4P0C3)));
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000007"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ List<HoodieCleanStat> hoodieCleanStatsFour =
+ runCleaner(config, false, false, 8, true);
+ HoodieCleanStat partitionZeroCleanStat = getCleanStat(hoodieCleanStatsFour, p0);
+ HoodieCleanStat partitionOneCleanStat = getCleanStat(hoodieCleanStatsFour, p1);
+
+ switch (minCommitsToTriggerClean) {
+ case 1:
+ assertEquals(2, partitionZeroCleanStat.getSuccessDeleteFiles().size(), "Must clean at least one old file");
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ break;
+ case 2:
+ case 3:
+ assertEquals(0, hoodieCleanStatsFour.size(),
+ "Must not clean any file. Only one commit has happened since the last clean!");
+ break;
+ case 4:
+ assertEquals(3, partitionZeroCleanStat.getSuccessDeleteFiles().size(), "Must clean at least one old file");
+ assertEquals(1, partitionOneCleanStat.getSuccessDeleteFiles().size());
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ break;
+ default:
+ }
+
+ metaClient = HoodieTableMetaClient.reload(metaClient);
+
+ String file5P0C4 = testTable.addInflightCommit("00000000000009")
+ .withBaseFilesInPartition(p0, file1P0C0)
+ .withBaseFilesInPartition(p0, file2P0C1)
+ .getFileIdsWithBaseFilesInPartitions(p0).get(p0);
+ commitMetadata = generateCommitMetadata("00000000000009", CollectionUtils.createImmutableMap(
+ p0, CollectionUtils.createImmutableList(file1P0C0, file2P0C1, file5P0C4)));
+ metaClient.getActiveTimeline().saveAsComplete(
+ new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMMIT_ACTION, "00000000000009"),
+ Option.of(commitMetadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
+
+ List<HoodieCleanStat> hoodieCleanStatsFive =
+ runCleaner(config, false, false, 10, true);
+
+ switch (minCommitsToTriggerClean) {
+ case 1:
+ assertEquals(1, hoodieCleanStatsFive.size(), "Must clean at least one old file");
+ assertFalse(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file3P0C2));
+ assertTrue(testTable.baseFileExists(p0, "00000000000007", file1P0C0));
+ break;
+ case 2:
+ assertEquals(1, hoodieCleanStatsFive.size(), "Must clean files written in second and third commits");
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000003", file2P0C1));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file1P0C0));
+ assertFalse(testTable.baseFileExists(p0, "00000000000005", file2P0C1));
+ assertTrue(testTable.baseFileExists(p0, "00000000000005", file3P0C2));
+ assertTrue(testTable.baseFileExists(p0, "00000000000007", file1P0C0));
+ break;
+ case 3:
+ assertEquals(0, hoodieCleanStatsFive.size(),
+ "Must not clean any file. Only two commits have happened since the last clean!");
+ break;
+ case 4:
+ assertEquals(0, hoodieCleanStatsFive.size(),
+ "Must not clean any file. Only one commit has happened since the last clean!");
+ break;
+ default:
+ }
+ }
+
+ @ParameterizedTest
+ @MethodSource("argumentsForTestTriggerCleanEveryNCommits")
+ public void testTriggerCleanEveryNthCommitWithMetadataTableEnabled(int minCommitsToTriggerClean) throws Exception {
+ //TODO: this has some bug and needs to be checked once.
+ }
Review Comment:
Could you remove this if not used now?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestCleanPlanExecutor.java:
##########
@@ -99,6 +99,206 @@ private static Stream<Arguments> argumentsForTestKeepLatestCommits() {
);
}
+ private static Stream<Arguments> argumentsForTestTriggerCleanEveryNCommits() {
+ return Stream.of(
+ Arguments.of(1),
+ Arguments.of(2),
+ Arguments.of(3),
+ Arguments.of(4)
Review Comment:
nit: indentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1295495489
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 620cd686b307a74af7f0b8b35279661c25fad24d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3676] Enhance tests for trigger clean every Nth commit [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-2027957658
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
}, {
"hash" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12654",
"triggerID" : "620cd686b307a74af7f0b8b35279661c25fad24d",
"triggerType" : "PUSH"
}, {
"hash" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081",
"triggerID" : "5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e",
"triggerType" : "PUSH"
}, {
"hash" : "36c0fdaa872c9ccc14729b09169b647f3d88efa4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23057",
"triggerID" : "36c0fdaa872c9ccc14729b09169b647f3d88efa4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ebe5720f5f5b7ceb3210da2a7e259c25ac1169e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18081)
* 36c0fdaa872c9ccc14729b09169b647f3d88efa4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1474992604
@pratyakshsharma : Also please rebase this code against latest master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1293914873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 711ed80f8c66f3c66e1c925a47b396011c4c4e98 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150)
* 76bc62cf70c15699c11989eb9169bf41c3483974 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1293908332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150",
"triggerID" : "1275625059",
"triggerType" : "MANUAL"
}, {
"hash" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "76bc62cf70c15699c11989eb9169bf41c3483974",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 711ed80f8c66f3c66e1c925a47b396011c4c4e98 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12150)
* 76bc62cf70c15699c11989eb9169bf41c3483974 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6926: [HUDI-3676]: Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1275421255
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12140",
"triggerID" : "f7a1e1f6b57cd4d72e9126a4a82698159d785b07",
"triggerType" : "PUSH"
}, {
"hash" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141",
"triggerID" : "711ed80f8c66f3c66e1c925a47b396011c4c4e98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 711ed80f8c66f3c66e1c925a47b396011c4c4e98 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12141)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #6926: [HUDI-3676] Enhance tests for trigger clean every Nth commit
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on PR #6926:
URL: https://github.com/apache/hudi/pull/6926#issuecomment-1376839387
this is good for another pass @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org