You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Sam Whittle (Jira)" <ji...@apache.org> on 2021/11/17 12:02:00 UTC

[jira] [Assigned] (BEAM-13268) Reduce latency by parallelizing BQ inserts when flushing due to row limit

     [ https://issues.apache.org/jira/browse/BEAM-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sam Whittle reassigned BEAM-13268:
----------------------------------

    Assignee: Sam Whittle

> Reduce latency by parallelizing BQ inserts when flushing due to row limit
> -------------------------------------------------------------------------
>
>                 Key: BEAM-13268
>                 URL: https://issues.apache.org/jira/browse/BEAM-13268
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp
>            Reporter: Sam Whittle
>            Assignee: Sam Whittle
>            Priority: P2
>
> InsertBatchedElements consumes the output of GroupIntoBatches which flushes after default 500 elements to respect the max items per streaming insert request to BQ.
> However InsertBatchedElements flushes rows synchrounously meaning that latencies of writes accumulates.  It could instead initiate writes in ProcessElement and block on write completion in FinishBundle.  There could be some limited configurable parallelism if desired to limit memory usage.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)