[jira] [Assigned] (BEAM-11208) BigQuery storage streams fail with QUOTA_EXCEEDED errors in split

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Beam JIRA Bot reassigned BEAM-11208:
------------------------------------

    Assignee:     (was: Vachan Shetty)

> BigQuery storage streams fail with QUOTA_EXCEEDED errors in split
> -----------------------------------------------------------------
>
>                 Key: BEAM-11208
>                 URL: https://issues.apache.org/jira/browse/BEAM-11208
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp
>    Affects Versions: 2.25.0
>            Reporter: Kenneth Jung
>            Priority: P2
>              Labels: stale-assigned
>   Original Estimate: 168h
>          Time Spent: 1.5h
>  Remaining Estimate: 166.5h
>
> The BigQueryStorageStreamSource attempts to call [SplitReadStream](https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1#bigqueryread) for each splitAtFraction call. However, the storage API [limits](https://cloud.google.com/bigquery/quotas#storage-limits) the number of control plane operations per minute for a given project, which can lead to the service being overloaded in the event of a large pipeline. The stream reader should not attempt to split a stream once it has learned that the stream can no longer be split (e.g. once a SplitReadStream call returns an empty response).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)