You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "arunpandianp (via GitHub)" <gi...@apache.org> on 2026/08/12 15:07:51 UTC
[PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
arunpandianp opened a new pull request, #39737:
URL: https://github.com/apache/beam/pull/39737
Replaces LinkedBlockingQueue with a ConcurrentLinkedQueue and a Semaphore. The harness threads don't block on the queue anymore.
Improves the internal reshuffle benchmark run time by ~10%.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5275196300
Is delayed visibility acceptable here? With concurrent producers, one can pause after `queue.add()` but before `availableItems.release()`. Another producer can enqueue and release, allowing consumer to dequeue first item using second permit and leaving second item unavailable until first producer resumes. Does commit queue require each completed `put()` to become immediately consumable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-6267782304
@arunpandianp did you get a chance to run tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5339429136
@arunpandianp did you get a chance to test the performance of LinkedTransferQueue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5277684378
LinkedTransferQueue does seem like it would be simpler, so I'd prefer that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5268664684
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5276705042
@bvolpato Thanks, that is a good observation. Such interleaving are possible and i think it is fine, with how the queue is used today in StreamingEngineWorkCommitter and StreamingApplianceCommitter.
It is true that a consumer can consume items using permits released after a different item. Generally the logic to stop consumers (or draining queue) should know and happen after all producer puts are complete. So that edge i think resolves the visibility problems here.
Having said that, changing to LinkedTransferQueue might be a simpler option with similar benefits. Testing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5268649445
R: @scwhittle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-6267626718
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5353841633
That might be https://bugs.openjdk.org/browse/JDK-8267502
Have you ever tried https://github.com/JCTools/JCTools? Something else we could look into as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Reduce contention on CommitQueue [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on PR #39737:
URL: https://github.com/apache/beam/pull/39737#issuecomment-5344875094
I ran a few tests with LinkedTransferQueue. LinkedTransferQueue spins inside which seems to use more cpu than LinkedBoundedQueue. The tests are not conclusive and i'll look into it more. I'll convert the PR to draft, to prevent automation from nagging, while I work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org