You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/30 23:25:08 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
nsivabalan opened a new pull request, #6548:
URL: https://github.com/apache/hudi/pull/6548
### Change Logs
Looks like we are doing file system based full listing when partitionsForFullCleaning is being invoked. When the patch was landed, DELETE_PARTITION was synchronous and hence the fix. but later we fixed DELETE_PARTITION to be lazy. and so, we can use metadata table if enabled.
### Impact
Will improve perf for clean policy based on FILE_VERSIONS when metadata is enabled.
**Risk level: low **
This will result in perf improvement for large tables.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1239834779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
}, {
"hash" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195",
"triggerID" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"triggerType" : "PUSH"
}, {
"hash" : "78d0f8bb6487e55b91443dcade5285e4a2412e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11229",
"triggerID" : "78d0f8bb6487e55b91443dcade5285e4a2412e3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195)
* 78d0f8bb6487e55b91443dcade5285e4a2412e3b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11229)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1232449966
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50a7437016b4fc6c876e11c4194a1a9944779f86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063)
* 2f188b98f0b2ab91ea07976696de2a162e327dd5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6548:
URL: https://github.com/apache/hudi/pull/6548#discussion_r964231352
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -206,15 +206,7 @@ private List<String> getPartitionPathsForIncrementalCleaning(HoodieCleanMetadata
*/
private List<String> getPartitionPathsForFullCleaning() {
// Go to brute force mode of scanning all partitions
- try {
- // Because the partition of BaseTableMetadata has been deleted,
- // all partition information can only be obtained from FileSystemBackedTableMetadata.
Review Comment:
One option is.
during DELETE_PARTITION, we will not delete it from metadata table.
When cleaner comes around and detects there are not valid file groups, it will physically delete it and will relay it to metadata table. at this juncture, metadata table will remove the partition of interest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6548:
URL: https://github.com/apache/hudi/pull/6548#discussion_r960987424
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -206,15 +206,7 @@ private List<String> getPartitionPathsForIncrementalCleaning(HoodieCleanMetadata
*/
private List<String> getPartitionPathsForFullCleaning() {
// Go to brute force mode of scanning all partitions
- try {
- // Because the partition of BaseTableMetadata has been deleted,
- // all partition information can only be obtained from FileSystemBackedTableMetadata.
Review Comment:
yeah. when Sagar wrote that code, here is how DELETE_PARTITION was implemented.
it wasn't well thought out design/impl. The way it was designed was, partition was synchronously deleted in data table and replace commit was relayed to MDT where in all file groups will be marked as invalid. but the partition in MDT remained.
if you remember, I chimed in and rewrote the entire impl, where in we made DELETE_PARTITION lazy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1232291045
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50a7437016b4fc6c876e11c4194a1a9944779f86 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063)
* 2f188b98f0b2ab91ea07976696de2a162e327dd5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4749][WIP] Fixing full cleaning to leverage metadata table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-2344558514
Looks like we fixed this already in some other patch.
https://github.com/apache/hudi/blob/ac79b148eaf03f4e8ca3fc4ad115d10d4a319553/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java#L264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1232458455
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f188b98f0b2ab91ea07976696de2a162e327dd5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1233157502
@nsivabalan is this fixing any GH issues? you may want to link it from the development field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1232329150
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50a7437016b4fc6c876e11c4194a1a9944779f86 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063)
* 2f188b98f0b2ab91ea07976696de2a162e327dd5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1238718932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
}, {
"hash" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195",
"triggerID" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f188b98f0b2ab91ea07976696de2a162e327dd5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064)
* 67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6548:
URL: https://github.com/apache/hudi/pull/6548#discussion_r959904836
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -206,15 +206,7 @@ private List<String> getPartitionPathsForIncrementalCleaning(HoodieCleanMetadata
*/
private List<String> getPartitionPathsForFullCleaning() {
// Go to brute force mode of scanning all partitions
- try {
- // Because the partition of BaseTableMetadata has been deleted,
- // all partition information can only be obtained from FileSystemBackedTableMetadata.
Review Comment:
trying to understand this previous comment, as questioned here https://github.com/apache/hudi/pull/5169/files#r957849431
under what scenario has the metadata table's partition being delete? even deleting synchronously, we should make use of metadata table first then delete metadata and data altogether?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6548:
URL: https://github.com/apache/hudi/pull/6548#discussion_r964230632
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -206,15 +206,7 @@ private List<String> getPartitionPathsForIncrementalCleaning(HoodieCleanMetadata
*/
private List<String> getPartitionPathsForFullCleaning() {
// Go to brute force mode of scanning all partitions
- try {
- // Because the partition of BaseTableMetadata has been deleted,
- // all partition information can only be obtained from FileSystemBackedTableMetadata.
Review Comment:
guess we can't go w/ this change. essentially what we are doing in cleaning is
1. fetch list of partitions to clean
2. Fetch list of files to clean
3. for those partitions where there is no valid file groups, we mark the partition to be deleted and physically delete the directory (lazy execution of DELETE_PARTITION)
So, in step1, we have to go w/ FS based listing. if we revert to metadata table based listing, we may not get the deleted partition only. and so the physical directory will never get deleted.
Need to think about an optimized way to go about this. for now, will punt on the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4749][WIP] Fixing full cleaning to leverage metadata table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #6548: [HUDI-4749][WIP] Fixing full cleaning to leverage metadata table
URL: https://github.com/apache/hudi/pull/6548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1239829803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
}, {
"hash" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195",
"triggerID" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"triggerType" : "PUSH"
}, {
"hash" : "78d0f8bb6487e55b91443dcade5285e4a2412e3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "78d0f8bb6487e55b91443dcade5285e4a2412e3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195)
* 78d0f8bb6487e55b91443dcade5285e4a2412e3b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1238683369
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
}, {
"hash" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f188b98f0b2ab91ea07976696de2a162e327dd5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064)
* 67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1306449082
Updates from our offline discussion: we may need some storage format changes and might need more thoughts around this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1240110458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
}, {
"hash" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195",
"triggerID" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"triggerType" : "PUSH"
}, {
"hash" : "78d0f8bb6487e55b91443dcade5285e4a2412e3b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11229",
"triggerID" : "78d0f8bb6487e55b91443dcade5285e4a2412e3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78d0f8bb6487e55b91443dcade5285e4a2412e3b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11229)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #6548:
URL: https://github.com/apache/hudi/pull/6548#discussion_r972613010
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -206,15 +206,7 @@ private List<String> getPartitionPathsForIncrementalCleaning(HoodieCleanMetadata
*/
private List<String> getPartitionPathsForFullCleaning() {
// Go to brute force mode of scanning all partitions
- try {
- // Because the partition of BaseTableMetadata has been deleted,
- // all partition information can only be obtained from FileSystemBackedTableMetadata.
Review Comment:
@nsivabalan I think your suggestion makes sense given that we now do lazy cleaning. If `files` partition of metadata table exists then, most likely, the record is there with `isDeleted` set to true. The metadata based fs view should understand the [isDeleted](https://github.com/apache/hudi/blob/master/hudi-common/src/main/avro/HoodieMetadata.avsc#L50) flag of `HoodieMetadataFileInfo`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1233268462
@xushiyan : done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1232288004
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50a7437016b4fc6c876e11c4194a1a9944779f86 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6548:
URL: https://github.com/apache/hudi/pull/6548#issuecomment-1238757083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11063",
"triggerID" : "50a7437016b4fc6c876e11c4194a1a9944779f86",
"triggerType" : "PUSH"
}, {
"hash" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11064",
"triggerID" : "2f188b98f0b2ab91ea07976696de2a162e327dd5",
"triggerType" : "PUSH"
}, {
"hash" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195",
"triggerID" : "67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 67cd8a64f22c3ed2b57b2bd97723874d5ea3ae54 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11195)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org