You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/12/09 18:15:57 UTC
[GitHub] [hudi] yihua opened a new pull request, #7420: [HUDI-5358] Fix flaky tests in TestCleanerInsertAndCleanByCommits
yihua opened a new pull request, #7420:
URL: https://github.com/apache/hudi/pull/7420
### Change Logs
This PR fixes the validation logic inside cleaner tests in `TestCleanerInsertAndCleanByCommits`.
In the tests, the `KEEP_LATEST_COMMITS` cleaner policy is used. This policy first figures out the earliest commit to retain based on the config of the number of retained commits (`hoodie.cleaner.commits.retained`). Then, for each file group, one more version before the earliest commit to retain is also kept from cleaning. The commit for the version can be different among file groups. For example, given the following commits, with `x` denoting a base file written for the corresponding commit and file group,
```
c1 c2 c3 c4 c5
fg1: x x x x
fg2: x x x x
```
with `hoodie.cleaner.commits.retained=3`, no files are cleaned.
However, the current validation logic only statically picks the one commit before the earliest commit to retain in the Hudi timeline, i.e., c2 in the above example, for all file groups, marking the base file written by c1 for fg1 to be expected for cleaning, which does not match the `KEEP_LATEST_COMMITS` cleaner policy. The validation logic is fixed according to the expected cleaner behavior.
This fix is required for CI to pass for #7396.
### Impact
Only fixing flaky tests. The validation in cleaner tests is now more robust.
### Risk level (write none, low medium or high below)
none
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7420: [HUDI-5358] Fix flaky tests in TestCleanerInsertAndCleanByCommits
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7420:
URL: https://github.com/apache/hudi/pull/7420#issuecomment-1344649866
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc94e32978b70756d4d7dc586b43df80779df75f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13588",
"triggerID" : "bc94e32978b70756d4d7dc586b43df80779df75f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc94e32978b70756d4d7dc586b43df80779df75f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13588)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7420: [HUDI-5358] Fix flaky tests in TestCleanerInsertAndCleanByCommits
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #7420:
URL: https://github.com/apache/hudi/pull/7420#issuecomment-1344622420
Thanks for cleaning this up @yihua!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #7420: [HUDI-5358] Fix flaky tests in TestCleanerInsertAndCleanByCommits
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #7420:
URL: https://github.com/apache/hudi/pull/7420#issuecomment-1344821958
CI is green:
<img width="1630" alt="Screenshot 2022-12-09 at 2 03 04 PM" src="https://user-images.githubusercontent.com/428277/206802989-932820b2-62f7-42f1-a2df-da28b5a86312.png">
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=13588&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin merged pull request #7420: [HUDI-5358] Fix flaky tests in TestCleanerInsertAndCleanByCommits
Posted by GitBox <gi...@apache.org>.
alexeykudinkin merged PR #7420:
URL: https://github.com/apache/hudi/pull/7420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7420: [HUDI-5358] Fix flaky tests in TestCleanerInsertAndCleanByCommits
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7420:
URL: https://github.com/apache/hudi/pull/7420#issuecomment-1344644122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc94e32978b70756d4d7dc586b43df80779df75f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc94e32978b70756d4d7dc586b43df80779df75f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc94e32978b70756d4d7dc586b43df80779df75f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org