You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Daniel Halperin (JIRA)" <ji...@apache.org> on 2016/02/25 17:12:18 UTC

[jira] [Created] (BEAM-63) DatastoreIO.Read: dynamic work rebalancing

Daniel Halperin created BEAM-63:
-----------------------------------

             Summary: DatastoreIO.Read: dynamic work rebalancing
                 Key: BEAM-63
                 URL: https://issues.apache.org/jira/browse/BEAM-63
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-gcp
            Reporter: Daniel Halperin
            Priority: Minor


1. Change the API to DatastoreIO.Read so that it exposes a `PTransform<PBegin,Entity>` rather than a `BoundedSource<Entity>`

2. Under the hood, rewrite the Source to implement dynamic work rebalancing. Datastore team's suggestion is to simply request a large number of splits initially and then use `Create.<Query>of(splits)` for dynamic assigning of Query splits to different workers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)