You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sachinnn99 (via GitHub)" <gi...@apache.org> on 2026/04/01 13:08:32 UTC
[PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
sachinnn99 opened a new pull request, #15849:
URL: https://github.com/apache/iceberg/pull/15849
## Summary
- Set `DynamicRecordProcessor` (generator) operator parallelism to match upstream input parallelism in `DynamicIcebergSink.Builder.append()`
- Without this fix, the generator falls back to the environment default parallelism, ignoring any parallelism configured on the upstream operator
- Matches the established pattern used in `IcebergSink` and `FlinkSink`
Closes #15827
## Test plan
- [x] Added `testGeneratorDefaultParallelism` in `TestDynamicIcebergSink` (all 3 Flink versions)
- [x] Verified existing `testOperatorUidsFormat` still passes
- [x] spotlessCheck passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on PR #15849:
URL: https://github.com/apache/iceberg/pull/15849#issuecomment-4275200688
@manuzhang Thanks for catching this and submitting #16026 to fix it. Apologies for the regression — I should have rebased on #15433 before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #15849:
URL: https://github.com/apache/iceberg/pull/15849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #15849:
URL: https://github.com/apache/iceberg/pull/15849#issuecomment-4274157125
It looks this PR did not rebase on https://github.com/apache/iceberg/pull/15433 and brought in a test regression.
```
> Task :iceberg-flink:iceberg-flink-2.1:test
TestDynamicIcebergSink > testNoShuffleTopology() FAILED
org.opentest4j.AssertionFailedError:
Expecting value to be true but was false
at app//org.apache.iceberg.flink.sink.dynamic.TestDynamicIcebergSink.testNoShuffleTopology(TestDynamicIcebergSink.java:330)
```
#16026 has been submitted to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on PR #15849:
URL: https://github.com/apache/iceberg/pull/15849#issuecomment-4275200669
Thanks @mxm for the review and the CC, and @pvary for the approval and merge!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on PR #15849:
URL: https://github.com/apache/iceberg/pull/15849#issuecomment-4190828765
Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #15849:
URL: https://github.com/apache/iceberg/pull/15849#issuecomment-4267683972
CC @pvary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Set generator parallelism to match input in DynamicIcebergSink [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15849:
URL: https://github.com/apache/iceberg/pull/15849#issuecomment-4270182396
Merged to main.
Thanks @sachinnn99 for the PR and @mxm for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org