[jira] [Work logged] (BEAM-11910) Increase subsequent page size for bags after the first

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-11910?focusedWorklogId=561408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-561408 ]

ASF GitHub Bot logged work on BEAM-11910:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 05/Mar/21 13:51
            Start Date: 05/Mar/21 13:51
    Worklog Time Spent: 10m 
      Work Description: scwhittle commented on pull request #14154:
URL: https://github.com/apache/beam/pull/14154#issuecomment-791431827


   R: @reuvenlax 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 561408)
    Time Spent: 20m  (was: 10m)

> Increase subsequent page size for bags after the first
> ------------------------------------------------------
>
>                 Key: BEAM-11910
>                 URL: https://issues.apache.org/jira/browse/BEAM-11910
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>            Reporter: Sam Whittle
>            Assignee: Sam Whittle
>            Priority: P2
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently the page size of bags requested from the streaming dataflow backend is always 8MB.  In pipelines with large bags this can limit throughput as it results in more round-trips to the backend.  In particular with Streaming Engine this is noticable due to increased latency.
> I propose using 8MB for the first bag fetch and then doubling the limit for subsequent paginations



--
This message was sent by Atlassian Jira
(v8.3.4#803005)