[GitHub] [beam] kennknowles opened a new issue, #19120: Dataflow runner: rateLimit exceeded from BigQuery API in start_bundle

[GitBox <gi...@apache.org>]

kennknowles opened a new issue, #19120:
URL: https://github.com/apache/beam/issues/19120

   When many machines, or too big machines are used, BigQuery returns rateLimit exceeded errors in response to the GET requests from BigQueryWriteFn.start_bundle.
   
   Unsure if this causes performance issues, or is just noise. (Produced with a pipeline reading strings from Pub/Sub that are ~1000 characters long, and writing directly to BigQuery)
   
   {{... File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line 1087, in get_or_create_table found_table = self._get_table(project_id, dataset_id, table_id) File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 184, in wrapper return fun(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery.py", line 925, in _get_table response = self.client.tables.Get(request) File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 611, in Get config, request, global_params=global_params) File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 722, in _RunMethod return self.ProcessHttpResponse(method_config, http_response, request) File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config, http_response, request)) File "/usr/local/lib/
 python2.7/dist-packages/apitools/base/py/base_api.py", line 599, in __ProcessHttpResponse http_response, method_config=method_config, request=request) HttpForbiddenError: HttpError accessing <[https://www.googleapis.com/bigquery/v2/projects/<xxxxxx>/datasets/](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json) [<xxxxxx>](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json) [/tables/](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json) [<xxxxxx>](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json) [?alt=json](https://www.googleapis.com/bigquery/v2/projects/carrefour-170216/datasets/realtime_dashboard/tables/realtime_errors?alt=json)\>: response: <
   
   {'status': '403', 'content-length': '577', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'expires': 'Sun, 25 Nov 2018 14:36:24 GMT', 'vary': 'Origin, X-Origin', 'server': 'GSE', '-content-encoding': 'gzip', 'cache-control': 'private, max-age=0', 'date': 'Sun, 25 Nov 2018 14:36:24 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}
   
   \>, content <{ "error": { "errors": [
   
   { "domain": "global", "reason": "rateLimitExceeded", "message": "Exceeded rate limits: Your user_method exceeded quota for api requests per user per method. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors", "locationType": "other", "location": "helix_api.method_request" }
   
   ], "code": 403, "message": "Exceeded rate limits: Your user_method exceeded quota for api requests per user per method. For more information, see [https://cloud.google.com/bigquery/troubleshooting-errors](https://cloud.google.com/bigquery/troubleshooting-errors)"}}
   
   Imported from Jira [BEAM-6169](https://issues.apache.org/jira/browse/BEAM-6169). Original Jira may contain additional context.
   Reported by: tpilewicz.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org