You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/09/02 07:11:35 UTC
[I] [BUG] Flink mapGroupsByKey executes lazy stream outside its dedicated ForkJoinPool [hudi]
cshuo opened a new issue, #19818:
URL: https://github.com/apache/hudi/issues/19818
### Describe the problem
HoodieFlinkEngineContext.mapGroupsByKey submits only a lazy Stream.map transformation to executeParallelStream. The transform returns a Stream before any terminal operation runs. executeParallelStream then shuts down its dedicated ForkJoinPool, and the caller performs flatMap and collect afterward.
As a result, processFunc and consumption of its result iterator do not run in the newly created ForkJoinPool. Because the source stream remains parallel, execution normally falls back to ForkJoinPool.commonPool when invoked from a regular Flink thread. The requested dedicated parallelism is therefore not honored and the work can contend with unrelated common-pool work.
### Current execution order
1. Create groupedValues.parallelStream.
2. Submit stream.map to the dedicated pool.
3. Return the unevaluated Stream from the submitted task.
4. Shut down the dedicated pool.
5. Execute flatMap and collect outside the pool, which finally invokes processFunc.
Exceptions thrown by processFunc also occur outside executeParallelStream and bypass its HoodieException wrapping.
There is an additional empty-input issue: groupedValues.size is zero, causing new ForkJoinPool(0) to throw IllegalArgumentException.
### Expected behavior
The complete parallel stream pipeline, including the terminal collect operation, should execute inside the dedicated ForkJoinPool. Empty input should return empty HoodieData without constructing a zero-parallelism pool.
### Suggested fix
Move flatMap and collect into the transform passed to executeParallelStream, handle empty groupedValues explicitly, and add tests that verify processFunc runs in a non-common ForkJoinPool and that empty input succeeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] Flink mapGroupsByKey executes lazy stream outside its dedicated ForkJoinPool [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #19818: [BUG] Flink mapGroupsByKey executes lazy stream outside its dedicated ForkJoinPool
URL: https://github.com/apache/hudi/issues/19818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org