You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Sam Whittle (Jira)" <ji...@apache.org> on 2021/03/02 12:33:00 UTC

[jira] [Created] (BEAM-11910) Increase subsequent page size for bags after the first

Sam Whittle created BEAM-11910:
----------------------------------

             Summary: Increase subsequent page size for bags after the first
                 Key: BEAM-11910
                 URL: https://issues.apache.org/jira/browse/BEAM-11910
             Project: Beam
          Issue Type: Bug
          Components: runner-dataflow
            Reporter: Sam Whittle


Currently the page size of bags requested from the streaming dataflow backend is always 8MB.  In pipelines with large bags this can limit throughput as it results in more round-trips to the backend.  In particular with Streaming Engine this is noticable due to increased latency.

I propose using 8MB for the first bag fetch and then doubling the limit for subsequent paginations



--
This message was sent by Atlassian Jira
(v8.3.4#803005)