You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "uvashishtha (via GitHub)" <gi...@apache.org> on 2025/12/18 20:50:59 UTC
[I] Potential bug: The committer and the stream writer are chained into the same operator [iceberg]
uvashishtha opened a new issue, #14887:
URL: https://github.com/apache/iceberg/issues/14887
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
With an upgrade to using Flink 2.0 and iceberg 1.10, we noticed that the Iceberg Files committer stream and the writer stream were chained together in the same operator (vertex node). This was not the case with versions <= Flink 1.20 where the committer (with max parallelism 1) and writer streams were different chains (occupied different flink vertices). Is this expected?
If it is, then functionality (like autoscaling) that was dependent on per-vertex metric consumption breaks.
If there is a mistake in my understanding, please correct it.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] The committer and the stream writer are chained into the same operator [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14887:
URL: https://github.com/apache/iceberg/issues/14887#issuecomment-4828073140
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] The committer and the stream writer are chained into the same operator [iceberg]
Posted by "uvashishtha (via GitHub)" <gi...@apache.org>.
uvashishtha closed issue #14887: The committer and the stream writer are chained into the same operator
URL: https://github.com/apache/iceberg/issues/14887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] The committer and the stream writer are chained into the same operator [iceberg]
Posted by "owen6314 (via GitHub)" <gi...@apache.org>.
owen6314 commented on issue #14887:
URL: https://github.com/apache/iceberg/issues/14887#issuecomment-3699321765
Have you tried setting the writer parallelism > 1?
I think if both the writer and committer are running with parallelism = 1, then Flink may chain them. But the scaling behavior of the writer is still governed by its own parallelism setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org