You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/20 10:28:40 UTC
[GitHub] [hudi] xicm opened a new pull request, #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
xicm opened a new pull request, #6149:
URL: https://github.com/apache/hudi/pull/6149
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Described in https://issues.apache.org/jira/browse/HUDI-4426,
The clean behavior is not accurate with CLEANER_COMMITS_RETAINED, it's not a big problem, but I think precision would be better.
## Brief change log
Fix logic error in getLatestVersionBeforeCommit
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1192667043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
}, {
"hash" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129",
"triggerID" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"triggerType" : "PUSH"
}, {
"hash" : "03e75725576d5e69a703d489ecc4808dafc30778",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "03e75725576d5e69a703d489ecc4808dafc30778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab62ffd77af6bfa92dd111072efd9f684a99bd94 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129)
* 03e75725576d5e69a703d489ecc4808dafc30778 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1190116065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d48f30beb18585f6cf33e634069192582379877 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm closed pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
xicm closed pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
URL: https://github.com/apache/hudi/pull/6149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6149:
URL: https://github.com/apache/hudi/pull/6149#discussion_r928058973
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -401,12 +401,12 @@ private List<CleanFileInfo> getReplacedFilesEligibleToClean(List<String> savepoi
}
/**
- * Gets the latest version < instantTime. This version file could still be used by queries.
+ * Gets the latest version <= instantTime. This version file could still be used by queries.
*/
private String getLatestVersionBeforeCommit(List<FileSlice> fileSliceList, HoodieInstant instantTime) {
for (FileSlice file : fileSliceList) {
String fileCommitTime = file.getBaseInstantTime();
Review Comment:
The method name indicates `before` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
xicm commented on code in PR #6149:
URL: https://github.com/apache/hudi/pull/6149#discussion_r928133052
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -401,12 +401,12 @@ private List<CleanFileInfo> getReplacedFilesEligibleToClean(List<String> savepoi
}
/**
- * Gets the latest version < instantTime. This version file could still be used by queries.
+ * Gets the latest version <= instantTime. This version file could still be used by queries.
*/
private String getLatestVersionBeforeCommit(List<FileSlice> fileSliceList, HoodieInstant instantTime) {
for (FileSlice file : fileSliceList) {
String fileCommitTime = file.getBaseInstantTime();
Review Comment:
Year, it's before.
Double check with you, if we set CLEANER_COMMITS_RETAINED=n, do we actualy retain n+1 commits by design? For example, if we have a fileGroup file_001.parquet,file_002.parquet,file_003.parquet,file_004.parquet, we set CLEANER_COMMITS_RETAINED=2, file_002.parquet,file_003.parquet,file_004.parquet will stay after a clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
xicm commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1194930042
Not a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1191104009
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
}, {
"hash" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129",
"triggerID" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d48f30beb18585f6cf33e634069192582379877 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096)
* ab62ffd77af6bfa92dd111072efd9f684a99bd94 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1192671509
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
}, {
"hash" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129",
"triggerID" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"triggerType" : "PUSH"
}, {
"hash" : "03e75725576d5e69a703d489ecc4808dafc30778",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10207",
"triggerID" : "03e75725576d5e69a703d489ecc4808dafc30778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab62ffd77af6bfa92dd111072efd9f684a99bd94 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129)
* 03e75725576d5e69a703d489ecc4808dafc30778 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10207)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1191286018
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
}, {
"hash" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129",
"triggerID" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab62ffd77af6bfa92dd111072efd9f684a99bd94 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1190111709
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d48f30beb18585f6cf33e634069192582379877 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1192738516
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
}, {
"hash" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10129",
"triggerID" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"triggerType" : "PUSH"
}, {
"hash" : "03e75725576d5e69a703d489ecc4808dafc30778",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10207",
"triggerID" : "03e75725576d5e69a703d489ecc4808dafc30778",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 03e75725576d5e69a703d489ecc4808dafc30778 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10207)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1190470958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d48f30beb18585f6cf33e634069192582379877 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
xicm commented on code in PR #6149:
URL: https://github.com/apache/hudi/pull/6149#discussion_r926292377
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestCleaner.java:
##########
@@ -1453,8 +1454,7 @@ private void testPendingCompactions(HoodieWriteConfig config, int expNumFilesDel
.withLogFile(partition, "fileId5", 1, 2)
.addCommit("011")
.withBaseFilesInPartition(partition, "fileId7")
- .withLogFile(partition, "fileId7", 1, 2)
- .addCommit("013");
Review Comment:
As commented, the earliest instance is 009, delete commit 013 here.
https://github.com/apache/hudi/blob/ab62ffd77af6bfa92dd111072efd9f684a99bd94/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestCleaner.java#L1171-L1188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6149: [HUDI-4426] The implementation of Clean is inconsistent with CLEANER_COMMITS_RETAINED
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6149:
URL: https://github.com/apache/hudi/pull/6149#issuecomment-1191100251
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6d48f30beb18585f6cf33e634069192582379877",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096",
"triggerID" : "6d48f30beb18585f6cf33e634069192582379877",
"triggerType" : "PUSH"
}, {
"hash" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab62ffd77af6bfa92dd111072efd9f684a99bd94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d48f30beb18585f6cf33e634069192582379877 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10096)
* ab62ffd77af6bfa92dd111072efd9f684a99bd94 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org