You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Below0 (via GitHub)" <gi...@apache.org> on 2026/03/30 08:42:57 UTC
[I] Flink DynamicIcebergSink: DynamicRecordProcessor does not inherit upstream parallelism [iceberg]
Below0 opened a new issue, #15827:
URL: https://github.com/apache/iceberg/issues/15827
### Apache Iceberg version
main (unreleased)
### Query engine
Flink
### Please describe the bug 🐞
In `DynamicIcebergSink.Builder.append()`, the `DynamicRecordProcessor` (generator) operator does not have its parallelism explicitly set. It always falls back to the environment's default parallelism, ignoring any parallelism configured on the upstream operator.
```
Upstream source (parallelism=16)
↓
DynamicRecordProcessor (parallelism=env default) ← upstream ignored
↓
Sink Writer (parallelism=writeParallelism)
```
The existing `writeParallelism` option only applies to the sink writer operator, not the generator.
**Current Workaround**
`env.setParallelism(N)` can be used, but it applies globally to all operators (generator, updater, sink writer), making fine-grained control impossible.
**Proposal**
Inherit upstream parallelism automatically, consistent with typical Flink operator chaining behavior:
```java
converted.setParallelism(input.getParallelism());
```
When upstream has explicit parallelism, the generator follows it. When not set (`-1`), it falls back to the env default — same as current behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Flink DynamicIcebergSink: DynamicRecordProcessor does not inherit upstream parallelism [iceberg]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on issue #15827:
URL: https://github.com/apache/iceberg/issues/15827#issuecomment-4169274768
I'd like to work on this. Will put up a PR shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Flink DynamicIcebergSink: DynamicRecordProcessor does not inherit upstream parallelism [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary closed issue #15827: Flink DynamicIcebergSink: DynamicRecordProcessor does not inherit upstream parallelism
URL: https://github.com/apache/iceberg/issues/15827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org