You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Daniel Halperin (JIRA)" <ji...@apache.org> on 2016/02/25 15:29:18 UTC

[jira] [Created] (BEAM-47) UnboundedSource should support dynamic work rebalancing

Daniel Halperin created BEAM-47:
-----------------------------------

             Summary: UnboundedSource should support dynamic work rebalancing
                 Key: BEAM-47
                 URL: https://issues.apache.org/jira/browse/BEAM-47
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-extensions
            Reporter: Daniel Halperin


BoundedSource has splitAtFraction and related code to enable large work items to be split dynamically and moved to other workers.

UnboundedSource does not have this support -- instead, runners are expected to allocate many splits initially and rebalance those between workers.

We should add dynamic work rebalancing to UnboundedSource. This will almost certainly be done by checkpointing the source, and then the dynamically produced splits will both inherit the checkpoint (or a split-adapted version of it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)