[jira] [Comment Edited] (BEAM-5514) BigQueryIO doesn't handle quotaExceeded errors properly

["Chamikara Jayalath (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643834#comment-16643834 ] 

Chamikara Jayalath edited comment on BEAM-5514 at 10/9/18 6:06 PM:
-------------------------------------------------------------------

I'm trying to determine the priority at which this should be addressed.

[~reuvenlax] any reason why we rely on workitems retries instead of retrying BQ streaming write requests with exponential backoff ?


was (Author: chamikara):
I'm trying to determine the priority at which this should be addressed.

 

[~reuvenlax] any reason why do rely on workitems retries instead of retrying BQ streaming write requests with exponential backoff ?

> BigQueryIO doesn't handle quotaExceeded errors properly
> -------------------------------------------------------
>
>                 Key: BEAM-5514
>                 URL: https://issues.apache.org/jira/browse/BEAM-5514
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp
>            Reporter: Kevin Peterson
>            Assignee: Reuven Lax
>            Priority: Major
>
> When exceeding a streaming quota for BigQuery insertAll requests, BigQuery returns a 403 with reason "quotaExceeded".
> The current implementation of BigQueryIO does not consider this to be a rate limited exception, and therefore does not perform exponential backoff properly, leading to repeated calls to BQ.
> The actual error is in the [ApiErrorExtractor|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L739] class, which is called from [BigQueryServicesImpl|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/util/src/main/java/com/google/cloud/hadoop/util/ApiErrorExtractor.java#L263] to determine how to retry the failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)