[jira] [Created] (BEAM-347) Progress updates inaccurate for non-uniform keys in Bigtable

["Ian Zhou (JIRA)" <ji...@apache.org>]

Ian Zhou created BEAM-347:
-----------------------------

             Summary: Progress updates inaccurate for non-uniform keys in Bigtable
                 Key: BEAM-347
                 URL: https://issues.apache.org/jira/browse/BEAM-347
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-gcp
            Reporter: Ian Zhou
            Assignee: Daniel Halperin
            Priority: Minor


When reading from a Bigtable source with clustered keys, fraction consumed progress updates are inaccurate. For example, for a range spanning ['a', 'z'], a cluster of keys starting with the letter 'm' (e.g. 'me100,' ..., 'me999') will be recorded as ~50% complete upon reading the first key, and will remain at this percentage until the final key has been read. Instead, the start of the range should be changed to the first key read (e.g. new range ['me100', 'z']). The end of the range can be changed over time through dynamic work rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)