You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "suryaprasanna (via GitHub)" <gi...@apache.org> on 2026/01/19 04:53:00 UTC
[PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
suryaprasanna opened a new pull request, #17942:
URL: https://github.com/apache/hudi/pull/17942
### Describe the issue this Pull Request addresses
Datasets with large record_index partitions cause OOM errors during clean planning when listing files. The current implementation uses the distributed engine context, which means increasing executor memory would affect all executors unnecessarily.
### Summary and Changelog
This PR addresses OOM errors during clean planning by using local engine context (driver-only) for metadata tables and non-partitioned datasets. This allows scaling only driver memory instead of all executor memory.
Changes:
- Added new config `hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables` (default: true)
- Modified `CleanPlanActionExecutor` to use `HoodieLocalEngineContext` for metadata tables and non-partitioned datasets
- Added corresponding getter method in `HoodieWriteConfig`
### Impact
New config property that changes the execution context for clean planning on specific table types. Users can now handle OOM errors by tuning only driver memory instead of all executor memory.
### Risk Level
Low - The change is isolated to clean planning and only affects metadata tables and non-partitioned datasets. The new config defaults to true, which is the optimized behavior.
### Documentation Update
Config description added in `HoodieCleanConfig.java` documenting the new property and its usage.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3829650332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c932105193ef3a7a23fc3e9f389c54ccaa797719 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3807766824
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3882570985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
}, {
"hash" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11838",
"triggerID" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"triggerType" : "PUSH"
}, {
"hash" : "202003f3b16d08c6613609d9318f9e2a3996de7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11927",
"triggerID" : "202003f3b16d08c6613609d9318f9e2a3996de7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 202003f3b16d08c6613609d9318f9e2a3996de7e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11927)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3766600453
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efe8adbae75a58d1f93467071c7ee43efc5fdb5c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197)
* b624af0fd5f349d23b8552cfba9d782ecd3c2a7a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2750378830
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
@danny0405 thank you for reviewing the PR. If data table is non-partitioned, then also we can use it. Don't want to increase the scope beyond that for main table. As it can backfire, if we are doing entire clean planning on the driver and people dont may not even consider something like this happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3766592991
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efe8adbae75a58d1f93467071c7ee43efc5fdb5c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197)
* b624af0fd5f349d23b8552cfba9d782ecd3c2a7a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2755586632
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
@danny0405 scope is for both data and metadata tables. But we do not want to enable local engine for all data tables, so when we detect that the data table is non-partitioned which basically means containing single parittion. We would like to use local engine context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2757212576
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
okay, sg, just the option name is so long, does it makes sense we just always do the optimization for such case and remove this option, if it is by default true. do we got some regression for corner cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2734777472
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
what about the data table clean planning, wondering why the MDT is specially handled here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3766400737
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efe8adbae75a58d1f93467071c7ee43efc5fdb5c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2765848416
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metrics/prometheus/TestPrometheusReporter.java:
##########
@@ -74,9 +74,13 @@ void setUp() {
@AfterEach
void shutdownMetrics() {
if (metrics != null) {
- metrics.shutdown();
+ try {
+ metrics.shutdown();
+ } catch (Exception e) {
+ log.debug("Exception during metrics shutdown: {}", e.getMessage());
Review Comment:
why not log.error?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
Review Comment:
I don't see a necessity for anyone to do this for partitioned data tables.
So, can we name this
`hoodie.clean.optimize.using.local.engine.context"
and make default as true.
programmatically, we can employ this only for non partitioned and metadata table.
if for some reason, users do not want that, they can disable via this config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3880950484
can we fix the PR desc based on latest code. for eg, config key is changed based on feedback right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3830116661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c932105193ef3a7a23fc3e9f389c54ccaa797719 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585)
* 23c61cf7372f00419d47ffb32f6b084d959417a4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3882088496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
}, {
"hash" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11838",
"triggerID" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"triggerType" : "PUSH"
}, {
"hash" : "202003f3b16d08c6613609d9318f9e2a3996de7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11927",
"triggerID" : "202003f3b16d08c6613609d9318f9e2a3996de7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9d29fcdf4f431f8fc41f3e194f684bf33e14105 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11838)
* 202003f3b16d08c6613609d9318f9e2a3996de7e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11927)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2784546648
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
Yeah, I agree. Although to begin with we can provide a config flag. Which we can remove it in the subsequent versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2784546648
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
Yeah, I agree. Although to begin with we can provide a config flag. Which we can remove it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2750378830
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
@danny0405 thank you reviewing the PR. If data table is non-partitioned, then also we can use it. Don't want to increase the scope beyond that for main table. As it can backfire, if we are doing entire clean planning on the driver and people dont may not even consider something like this happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3766521616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efe8adbae75a58d1f93467071c7ee43efc5fdb5c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #17942:
URL: https://github.com/apache/hudi/pull/17942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3882081837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
}, {
"hash" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11838",
"triggerID" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"triggerType" : "PUSH"
}, {
"hash" : "202003f3b16d08c6613609d9318f9e2a3996de7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "202003f3b16d08c6613609d9318f9e2a3996de7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9d29fcdf4f431f8fc41f3e194f684bf33e14105 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11838)
* 202003f3b16d08c6613609d9318f9e2a3996de7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2784556573
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metrics/prometheus/TestPrometheusReporter.java:
##########
@@ -74,9 +74,13 @@ void setUp() {
@AfterEach
void shutdownMetrics() {
if (metrics != null) {
- metrics.shutdown();
+ try {
+ metrics.shutdown();
+ } catch (Exception e) {
+ log.debug("Exception during metrics shutdown: {}", e.getMessage());
Review Comment:
Yes, added log at error level.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
Responded as part of a comment, please check.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
Review Comment:
Yes, made the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3830120990
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c932105193ef3a7a23fc3e9f389c54ccaa797719 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585)
* 23c61cf7372f00419d47ffb32f6b084d959417a4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3807648010
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b624af0fd5f349d23b8552cfba9d782ecd3c2a7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200)
* 698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3881145791
> can we fix the PR desc based on latest code. for eg, config key is changed based on feedback right
Good catch, updated it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3923556023
hey @suryaprasanna : may be we should rethink the config naming
`hoodie.clean.optimize.using.local.engine.context` -> `hoodie.clean.optimize.planner.using.local.engine.context`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3766721555
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b624af0fd5f349d23b8552cfba9d782ecd3c2a7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3766408021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efe8adbae75a58d1f93467071c7ee43efc5fdb5c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3881152974
Creating unit tests for verifying spark context used maybe harder to achieve. Since HoodieLocalEngineContext testing is already handled as part of the hudi-java-client unit tests, so I left it there. Let me know, if you have a way for testing these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3829571873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412)
* c932105193ef3a7a23fc3e9f389c54ccaa797719 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3807634759
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b624af0fd5f349d23b8552cfba9d782ecd3c2a7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200)
* 698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3829578571
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412)
* c932105193ef3a7a23fc3e9f389c54ccaa797719 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3830314530
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23c61cf7372f00419d47ffb32f6b084d959417a4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2752516348
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
if the scope is just MDT, the option can be moved into `HoodieMetadataConfig`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17942:
URL: https://github.com/apache/hudi/pull/17942#discussion_r2757212576
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -201,6 +201,16 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "table receives updates/deletes. Another reason to turn this on, would be to ensure data residing in bootstrap "
+ "base files are also physically deleted, to comply with data privacy enforcement processes.");
+ public static final ConfigProperty<Boolean> USE_LOCAL_ENGINE_FOR_METADATA_NON_PARTITIONED_DATASETS = ConfigProperty
+ .key("hoodie.clean.planner.use.local.engine.on.metadata.and.non-partitioned.tables")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("Some datasets have huge record_index partition, listing this partition is causing OOM errors on clean planner. "
+ + "So, if we increase executor memory it will increase for all the executors. So, by passing local engine context to clean "
+ + "planner, listing is done on the driver. In that case, if OOM error were come we can increase driver memory alone to "
Review Comment:
okay, sg, just the option name is so long, does it make sense we just always do the optimization for such case and remove this option, if it is by default true. do we got some regression for corner cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3873936930
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
}, {
"hash" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23c61cf7372f00419d47ffb32f6b084d959417a4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587)
* c9d29fcdf4f431f8fc41f3e194f684bf33e14105 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Use local engine context for clean planning on metadata and non-partitioned tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17942:
URL: https://github.com/apache/hudi/pull/17942#issuecomment-3874689731
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11197",
"triggerID" : "efe8adbae75a58d1f93467071c7ee43efc5fdb5c",
"triggerType" : "PUSH"
}, {
"hash" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11200",
"triggerID" : "b624af0fd5f349d23b8552cfba9d782ecd3c2a7a",
"triggerType" : "PUSH"
}, {
"hash" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11412",
"triggerID" : "698d7ef96ace8fbe0f8ff12aeb3a0044421a1db0",
"triggerType" : "PUSH"
}, {
"hash" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11585",
"triggerID" : "c932105193ef3a7a23fc3e9f389c54ccaa797719",
"triggerType" : "PUSH"
}, {
"hash" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11587",
"triggerID" : "23c61cf7372f00419d47ffb32f6b084d959417a4",
"triggerType" : "PUSH"
}, {
"hash" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11838",
"triggerID" : "c9d29fcdf4f431f8fc41f3e194f684bf33e14105",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9d29fcdf4f431f8fc41f3e194f684bf33e14105 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11838)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org