You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "talatuyarer (via GitHub)" <gi...@apache.org> on 2023/12/16 20:02:46 UTC
[PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
talatuyarer opened a new pull request, #29792:
URL: https://github.com/apache/beam/pull/29792
When we have multiple KafkaIO on same job. Beam's Default max parallesim apply for all even though Kafka does not have that much partition on the topic. To prevent that We need to set max Parallesim on UnBoundedSource based on split count.
This is fix for that issue.
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-1959899288
@je-ik `maxParallelism` beyond the number of partitions in Kafka, you're basically giving workers extra splits that they don't need for that Kafka topic. This means each worker ends up using more threads than necessary. And when you've got a bunch of KafkaIO pipelines doing this, it eats up more memory and makes the checkpoints bigger. Bigger checkpoints can slow down how often you can do them, which isn't great for keeping things running smoothly.
Is there any benefit to set bigger than partition count ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-2088410206
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-1958163408
@je-ik Could you review this ? Do you see any issue with this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-1859363436
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-1948317594
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-2072196633
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-1958911168
Are there any issues you see with the empty splits? In general, these should simply emit MAX_WATERMARK and then should not cause any overhead. If this is not happening, then there might be some other issue we might want to uncover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-1959987633
I think there is no particular benefit, at least given the current implementation. On the other hand, before we place a hard limit, we should know it is really needed, that it is not just a manifestation of another issue.
From my understanding - if reader gets empty partitions, the particular split should not connect to Kafka, or contribute to checkpoint size, it should actually only emit MAX_WATERMARK (which means that there will be no more data from this split) and sleep forever (or terminate after `shutdownSourcesAfterIdleMs` milliseconds). Is this not working in your case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Default max Parallelism should not apply all UnBoundedSource. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #29792: Default max Parallelism should not apply all UnBoundedSource.
URL: https://github.com/apache/beam/pull/29792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org