You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "HuangZhenQiu (via GitHub)" <gi...@apache.org> on 2026/09/12 04:47:01 UTC
[PR] feat(flink): support isolate slot group for compact plan generation [hudi]
HuangZhenQiu opened a new pull request, #19920:
URL: https://github.com/apache/hudi/pull/19920
### Describe the issue this Pull Request addresses
Support isolate slot group for compact plan generation
### Summary and Changelog
1. create isolated slot group for compaction plan generator operator with flink conf
2. create test cases to cover the changes
### Impact
none
### Risk Level
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19920:
URL: https://github.com/apache/hudi/pull/19920#discussion_r3995723051
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -721,11 +724,29 @@ private static DataStream<HoodieFlinkInternalRow> createBucketAssignStream(
* @return the compaction pipeline
*/
public static DataStreamSink<CompactionCommitEvent> compact(Configuration conf, DataStream<RowData> dataStream) {
- DataStreamSink<CompactionCommitEvent> compactionCommitEventDataStream = dataStream.transform("compact_plan_generate",
+ SingleOutputStreamOperator<CompactionPlanEvent> compactionPlanStream = dataStream.transform("compact_plan_generate",
TypeInformation.of(CompactionPlanEvent.class),
new CompactionPlanOperator(conf))
.setParallelism(1) // plan generate must be singleton
- .setMaxParallelism(1)
+ .setMaxParallelism(1);
+ if (conf.getOptional(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP).isPresent()) {
+ String slotSharingGroup = conf.get(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP);
+ SlotSharingGroup.Builder slotSharingGroupBuilder = SlotSharingGroup.newBuilder(slotSharingGroup);
+ // Flink requires cpu cores and task heap memory to be configured together when specifying
+ // resources for a slot sharing group, reuse the TaskManager-level configuration for both.
+ Option<Double> cpuCores = Option.fromJavaOptional(conf.getOptional(TaskManagerOptions.CPU_CORES));
+ Option<MemorySize> taskHeapMemory = Option.fromJavaOptional(conf.getOptional(TaskManagerOptions.TASK_HEAP_MEMORY));
+ if (!taskHeapMemory.isPresent()) {
+ taskHeapMemory = deriveTaskHeapMemory(conf);
+ }
+ if (cpuCores.isPresent() && taskHeapMemory.isPresent()) {
+ slotSharingGroupBuilder.setCpuCores(cpuCores.get()).setTaskHeapMemory(taskHeapMemory.get());
Review Comment:
🤖 Reusing the TaskManager totals (`taskmanager.cpu.cores`, full task heap) as the slot group's resource spec means this parallelism-1 operator requests an entire TM's CPU and task heap for its slot. The RM only fulfills a request when a TM's total/available profile is `allFieldsNoLessThan` it, so on a standalone/session cluster with `numberOfTaskSlots > 1` (or any TM already hosting default-group slots) this can never be matched and the job fails with `NoResourceAvailableException`; in fine-grained mode it monopolizes a whole TM. Would a fraction (or dedicated `compaction.plan_generate.slot_sharing_group.{cpu,memory}` options) be a safer definition of the slot's resources?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -721,11 +724,29 @@ private static DataStream<HoodieFlinkInternalRow> createBucketAssignStream(
* @return the compaction pipeline
*/
public static DataStreamSink<CompactionCommitEvent> compact(Configuration conf, DataStream<RowData> dataStream) {
- DataStreamSink<CompactionCommitEvent> compactionCommitEventDataStream = dataStream.transform("compact_plan_generate",
+ SingleOutputStreamOperator<CompactionPlanEvent> compactionPlanStream = dataStream.transform("compact_plan_generate",
TypeInformation.of(CompactionPlanEvent.class),
new CompactionPlanOperator(conf))
.setParallelism(1) // plan generate must be singleton
- .setMaxParallelism(1)
+ .setMaxParallelism(1);
+ if (conf.getOptional(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP).isPresent()) {
+ String slotSharingGroup = conf.get(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP);
Review Comment:
🤖 nit: this inline block (resolving cpu/heap, building and registering the slot sharing group) adds a good chunk of unrelated logic into `compact()`. Might be worth pulling it into a small helper like `configureSlotSharingGroup(conf, compactionPlanStream)` so `compact()` stays focused on wiring the pipeline stages together.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -721,11 +725,29 @@ private static DataStream<HoodieFlinkInternalRow> createBucketAssignStream(
* @return the compaction pipeline
*/
public static DataStreamSink<CompactionCommitEvent> compact(Configuration conf, DataStream<RowData> dataStream) {
- DataStreamSink<CompactionCommitEvent> compactionCommitEventDataStream = dataStream.transform("compact_plan_generate",
+ SingleOutputStreamOperator<CompactionPlanEvent> compactionPlanStream = dataStream.transform("compact_plan_generate",
TypeInformation.of(CompactionPlanEvent.class),
new CompactionPlanOperator(conf))
.setParallelism(1) // plan generate must be singleton
- .setMaxParallelism(1)
+ .setMaxParallelism(1);
+ if (conf.getOptional(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP).isPresent()) {
+ String slotSharingGroup = conf.get(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP);
+ SlotSharingGroup.Builder slotSharingGroupBuilder = SlotSharingGroup.newBuilder(slotSharingGroup);
+ // Flink requires cpu cores and task heap memory to be configured together when specifying
+ // resources for a slot sharing group, reuse the TaskManager-level configuration for both.
+ Optional<Double> cpuCores = conf.getOptional(TaskManagerOptions.CPU_CORES);
+ Optional<MemorySize> taskHeapMemory = conf.getOptional(TaskManagerOptions.TASK_HEAP_MEMORY);
+ if (!taskHeapMemory.isPresent()) {
+ taskHeapMemory = deriveTaskHeapMemory(conf);
+ }
+ if (cpuCores.isPresent() && taskHeapMemory.isPresent()) {
+ slotSharingGroupBuilder.setCpuCores(cpuCores.get()).setTaskHeapMemory(taskHeapMemory.get());
+ }
+ // the slot sharing group must be registered with the environment before it can be referenced by name
+ dataStream.getExecutionEnvironment().registerSlotSharingGroup(slotSharingGroupBuilder.build());
+ compactionPlanStream = compactionPlanStream.slotSharingGroup(slotSharingGroup);
Review Comment:
🤖 Confirmed against Flink 1.20.1: `StreamGraph.getSlotSharingGroup(id)` resolves the `partitionCustom` virtual node back to `compact_plan_generate`, so `compact_task` and `compact_commit` both inherit the new group. Since the group also carries a resource profile of the full TM cpu + task heap, that means each of the `compaction.tasks` subtasks would demand an exclusive slot with that profile — on a single-TM setup the job likely can't be scheduled at all. Pinning the two downstream operators to `"default"` (or the write pipeline's group) should address both the isolation and the scheduling side of it.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -721,11 +725,29 @@ private static DataStream<HoodieFlinkInternalRow> createBucketAssignStream(
* @return the compaction pipeline
*/
public static DataStreamSink<CompactionCommitEvent> compact(Configuration conf, DataStream<RowData> dataStream) {
- DataStreamSink<CompactionCommitEvent> compactionCommitEventDataStream = dataStream.transform("compact_plan_generate",
+ SingleOutputStreamOperator<CompactionPlanEvent> compactionPlanStream = dataStream.transform("compact_plan_generate",
TypeInformation.of(CompactionPlanEvent.class),
new CompactionPlanOperator(conf))
.setParallelism(1) // plan generate must be singleton
- .setMaxParallelism(1)
+ .setMaxParallelism(1);
+ if (conf.getOptional(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP).isPresent()) {
+ String slotSharingGroup = conf.get(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP);
+ SlotSharingGroup.Builder slotSharingGroupBuilder = SlotSharingGroup.newBuilder(slotSharingGroup);
+ // Flink requires cpu cores and task heap memory to be configured together when specifying
+ // resources for a slot sharing group, reuse the TaskManager-level configuration for both.
+ Optional<Double> cpuCores = conf.getOptional(TaskManagerOptions.CPU_CORES);
+ Optional<MemorySize> taskHeapMemory = conf.getOptional(TaskManagerOptions.TASK_HEAP_MEMORY);
Review Comment:
🤖 There's an existing precedent for this in `HoodieTableSink#getSinkRuntimeProvider`, which reads cluster-level settings via `dataStream.getExecutionEnvironment().getConfiguration()` (`OptionsInference.setupRuntimeConfigs`). `StreamContextEnvironment` merges the cluster flink-conf into that configuration, so falling back to it for `TaskManagerOptions.*` would pick up `taskmanager.cpu.cores` / memory settings without users having to copy them into the table options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19920:
URL: https://github.com/apache/hudi/pull/19920#issuecomment-5643745753
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"triggerType" : "PUSH"
}, {
"hash" : "b4c8a77d71c51b9023519c79e57dd00f7d92f7c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17035",
"triggerID" : "b4c8a77d71c51b9023519c79e57dd00f7d92f7c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28d13e287157ad5358bfc88b86dc79bd9abd808 UNKNOWN
* b4c8a77d71c51b9023519c79e57dd00f7d92f7c8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19920:
URL: https://github.com/apache/hudi/pull/19920#discussion_r3995151312
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -721,11 +725,29 @@ private static DataStream<HoodieFlinkInternalRow> createBucketAssignStream(
* @return the compaction pipeline
*/
public static DataStreamSink<CompactionCommitEvent> compact(Configuration conf, DataStream<RowData> dataStream) {
- DataStreamSink<CompactionCommitEvent> compactionCommitEventDataStream = dataStream.transform("compact_plan_generate",
+ SingleOutputStreamOperator<CompactionPlanEvent> compactionPlanStream = dataStream.transform("compact_plan_generate",
TypeInformation.of(CompactionPlanEvent.class),
new CompactionPlanOperator(conf))
.setParallelism(1) // plan generate must be singleton
- .setMaxParallelism(1)
+ .setMaxParallelism(1);
+ if (conf.getOptional(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP).isPresent()) {
+ String slotSharingGroup = conf.get(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP);
+ SlotSharingGroup.Builder slotSharingGroupBuilder = SlotSharingGroup.newBuilder(slotSharingGroup);
+ // Flink requires cpu cores and task heap memory to be configured together when specifying
+ // resources for a slot sharing group, reuse the TaskManager-level configuration for both.
+ Optional<Double> cpuCores = conf.getOptional(TaskManagerOptions.CPU_CORES);
+ Optional<MemorySize> taskHeapMemory = conf.getOptional(TaskManagerOptions.TASK_HEAP_MEMORY);
+ if (!taskHeapMemory.isPresent()) {
+ taskHeapMemory = deriveTaskHeapMemory(conf);
+ }
+ if (cpuCores.isPresent() && taskHeapMemory.isPresent()) {
+ slotSharingGroupBuilder.setCpuCores(cpuCores.get()).setTaskHeapMemory(taskHeapMemory.get());
+ }
+ // the slot sharing group must be registered with the environment before it can be referenced by name
+ dataStream.getExecutionEnvironment().registerSlotSharingGroup(slotSharingGroupBuilder.build());
+ compactionPlanStream = compactionPlanStream.slotSharingGroup(slotSharingGroup);
Review Comment:
🤖 I think this ends up pulling `compact_task` and `compact_commit` into the same group rather than isolating the planner: Flink's `StreamGraphGenerator.determineSlotSharingGroup` makes an operator with no explicit group inherit its input's group, and neither downstream operator sets one. Could you explicitly pin them to `"default"` (`.slotSharingGroup("default")`) and assert on the StreamGraph-level group (`getStreamGraph().getStreamNode(..).getSlotSharingGroup()`) in the test, since `Transformation.getSlotSharingGroup()` only reflects the explicitly-set value?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -721,11 +725,29 @@ private static DataStream<HoodieFlinkInternalRow> createBucketAssignStream(
* @return the compaction pipeline
*/
public static DataStreamSink<CompactionCommitEvent> compact(Configuration conf, DataStream<RowData> dataStream) {
- DataStreamSink<CompactionCommitEvent> compactionCommitEventDataStream = dataStream.transform("compact_plan_generate",
+ SingleOutputStreamOperator<CompactionPlanEvent> compactionPlanStream = dataStream.transform("compact_plan_generate",
TypeInformation.of(CompactionPlanEvent.class),
new CompactionPlanOperator(conf))
.setParallelism(1) // plan generate must be singleton
- .setMaxParallelism(1)
+ .setMaxParallelism(1);
+ if (conf.getOptional(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP).isPresent()) {
+ String slotSharingGroup = conf.get(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP);
+ SlotSharingGroup.Builder slotSharingGroupBuilder = SlotSharingGroup.newBuilder(slotSharingGroup);
+ // Flink requires cpu cores and task heap memory to be configured together when specifying
+ // resources for a slot sharing group, reuse the TaskManager-level configuration for both.
+ Optional<Double> cpuCores = conf.getOptional(TaskManagerOptions.CPU_CORES);
+ Optional<MemorySize> taskHeapMemory = conf.getOptional(TaskManagerOptions.TASK_HEAP_MEMORY);
Review Comment:
🤖 The `conf` here is the Hudi table/compaction-job options rather than the cluster's flink-conf, so `taskmanager.cpu.cores` / memory keys will only be present if a user copies them into the table properties — is that the intended way to enable the resource profile? Also, requesting the full TM cpu + task heap means nothing else can be scheduled on that TM; if that's the intent, it might be worth spelling out in the option description.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -1025,6 +1025,16 @@ public class FlinkOptions extends HoodieConfig {
.noDefaultValue()
.withDescription("Parallelism of tasks that do actual compaction, default same as the write task parallelism");
+ @AdvancedConfig
+ public static final ConfigOption<String> COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP = ConfigOptions
Review Comment:
🤖 nit: the key mixes underscores and dots (`plan_generate`, `slot_sharing_group`) whereas most other keys in this file use dot-separated segments directly (e.g. `compaction.tasks`, `compaction.delta_commits` already has underscores too, so double check what the dominant style is here) — consider `compaction.plan-generate.slot-sharing-group` or matching the surrounding keys' exact word separation so it doesn't read as a one-off.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -721,11 +725,29 @@ private static DataStream<HoodieFlinkInternalRow> createBucketAssignStream(
* @return the compaction pipeline
*/
public static DataStreamSink<CompactionCommitEvent> compact(Configuration conf, DataStream<RowData> dataStream) {
- DataStreamSink<CompactionCommitEvent> compactionCommitEventDataStream = dataStream.transform("compact_plan_generate",
+ SingleOutputStreamOperator<CompactionPlanEvent> compactionPlanStream = dataStream.transform("compact_plan_generate",
TypeInformation.of(CompactionPlanEvent.class),
new CompactionPlanOperator(conf))
.setParallelism(1) // plan generate must be singleton
- .setMaxParallelism(1)
+ .setMaxParallelism(1);
+ if (conf.getOptional(FlinkOptions.COMPACTION_PLAN_GENERATE_SLOT_SHARING_GROUP).isPresent()) {
Review Comment:
🤖 nit: the slot-sharing-group setup (registering the group, deriving CPU/heap memory) is about 15 lines inlined into `compact()`, which already does a lot; consider extracting it into a small private helper like `configureSlotSharingGroup(DataStream<RowData> dataStream, SingleOutputStreamOperator<CompactionPlanEvent> stream, Configuration conf)` so `compact()` stays focused on wiring the pipeline stages.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #19920:
URL: https://github.com/apache/hudi/pull/19920#issuecomment-5658845705
Could you share the concrete bottleneck this addresses and some before/after measurements? I'd prefer to avoid adding this complexity unless the benefit is clear, especially the logic that derives a slot group's resources from TaskManager CPU and memory settings. That introduces additional scheduling behavior and duplicates part of Flink's memory configuration logic that we would need to maintain.
If the requirement is simply to place the compaction plan operator in a separate slot-sharing group, could we keep this to a configurable group name and explicitly preserve the downstream operators' intended group? Flink supports named groups without an explicit resource profile. We could consider resource sizing separately once there is evidence that the simpler approach is insufficient.
Checkpoint latency, compaction delay, and resource usage before and after the change, with the cluster resource budget held constant, would help establish whether the added complexity is worthwhile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19920:
URL: https://github.com/apache/hudi/pull/19920#issuecomment-5643689389
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28d13e287157ad5358bfc88b86dc79bd9abd808 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19920:
URL: https://github.com/apache/hudi/pull/19920#issuecomment-5644414895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"triggerType" : "PUSH"
}, {
"hash" : "b4c8a77d71c51b9023519c79e57dd00f7d92f7c8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17035",
"triggerID" : "b4c8a77d71c51b9023519c79e57dd00f7d92f7c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28d13e287157ad5358bfc88b86dc79bd9abd808 UNKNOWN
* b4c8a77d71c51b9023519c79e57dd00f7d92f7c8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19920:
URL: https://github.com/apache/hudi/pull/19920#issuecomment-5643711471
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19920?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 79.96%. Comparing base ([`1d3a0af`](https://app.codecov.io/gh/apache/hudi/commit/1d3a0affad873868e74df7207013ae8eeb4dbb8c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b4c8a77`](https://app.codecov.io/gh/apache/hudi/commit/b4c8a77d71c51b9023519c79e57dd00f7d92f7c8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19920 +/- ##
============================================
- Coverage 80.21% 79.96% -0.25%
- Complexity 34674 34796 +122
============================================
Files 2546 2546
Lines 142463 143454 +991
Branches 17333 17526 +193
============================================
+ Hits 114271 114718 +447
- Misses 20283 20789 +506
- Partials 7909 7947 +38
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.63% <ø> (-0.23%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.46% <ø> (-0.83%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.56% <ø> (-0.01%)` | :arrow_down: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.28% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.14% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.99% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.63% <ø> (-0.19%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.02% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.61% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.34% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.99% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19920/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19920/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19920/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.90% <ø> (-0.06%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19920?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...va/org/apache/hudi/configuration/FlinkOptions.java](https://app.codecov.io/gh/apache/hudi/pull/19920?src=pr&el=tree&filepath=hudi-flink-datasource%2Fhudi-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fconfiguration%2FFlinkOptions.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1mbGluay1kYXRhc291cmNlL2h1ZGktZmxpbmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29uZmlndXJhdGlvbi9GbGlua09wdGlvbnMuamF2YQ==) | `99.79% <ø> (ø)` | |
| [...ain/java/org/apache/hudi/sink/utils/Pipelines.java](https://app.codecov.io/gh/apache/hudi/pull/19920?src=pr&el=tree&filepath=hudi-flink-datasource%2Fhudi-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fsink%2Futils%2FPipelines.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1mbGluay1kYXRhc291cmNlL2h1ZGktZmxpbmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvc2luay91dGlscy9QaXBlbGluZXMuamF2YQ==) | `94.11% <ø> (-0.05%)` | :arrow_down: |
... and [13 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19920/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19920:
URL: https://github.com/apache/hudi/pull/19920#issuecomment-5643720404
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d28d13e287157ad5358bfc88b86dc79bd9abd808",
"triggerType" : "PUSH"
}, {
"hash" : "b4c8a77d71c51b9023519c79e57dd00f7d92f7c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4c8a77d71c51b9023519c79e57dd00f7d92f7c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28d13e287157ad5358bfc88b86dc79bd9abd808 UNKNOWN
* b4c8a77d71c51b9023519c79e57dd00f7d92f7c8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #19920:
URL: https://github.com/apache/hudi/pull/19920#issuecomment-6701428163
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): support isolate slot group for compact plan generation [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #19920: feat(flink): support isolate slot group for compact plan generation
URL: https://github.com/apache/hudi/pull/19920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org