[GitHub] [beam] damccorm opened a new issue, #17835: Support for limiting parallelism of a step

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #17835:
URL: https://github.com/apache/beam/issues/17835

   Users may want to limit the parallelism of a step. Two classic uses cases are:
   
   - User wants to produce at most k files, so sets TextIO.Write.withNumShards(k).
   - External API only supports k QPS, so user sets a limit of k/(expected QPS/step) on the ParDo that makes the API call.
   
   Unfortunately, there is no way to do this effectively within the Beam model. A GroupByKey with exactly k keys will guarantee that only k elements are produced, but runners are free to break fusion in ways that each element may be processed in parallel later.
   
   To implement this functionaltiy, I believe we need to add this support to the Beam Model.
   
   Imported from Jira [BEAM-68](https://issues.apache.org/jira/browse/BEAM-68). Original Jira may contain additional context.
   Reported by: dhalperi.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm closed issue #17835: Support for limiting parallelism of a step

[GitBox <gi...@apache.org>]

damccorm closed issue #17835: Support for limiting parallelism of a step
URL: https://github.com/apache/beam/issues/17835


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org