[jira] [Comment Edited] (BEAM-8195) Quota exceeded for create requests

["Alan Myrvold (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000297#comment-17000297 ] 

Alan Myrvold edited comment on BEAM-8195 at 12/19/19 6:47 PM:
--------------------------------------------------------------

I also see inconsistent retry behavior between the Python SDK and Java SDK.

For 429 errors, Python retries 4 times and Java retries 5 times.

For 503 errors, Python retries 19 times and Java retries 5 times.

Python's submit_job_description uses the default retry_on_server_errors_filter which does not retry on 4xx errors.

Not sure where the extra python retry's are happening, but likely on the http transport level.
To verify, ran:
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 503 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null  ; echo "req $a $(date)"; done}}
or
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 429 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null  ; echo "req $a $(date)"; done}}

Locally, then ran dataflow pipeline with {{--dataflow_endpoint=http://localhost:1500 }}(Python) or {{--dataflowEndpoint=http://localhost:1500 }}(java)


was (Author: alanmyrvold):
I also see inconsistent retry behavior between the Python SDK and Java SDK.
For 429 errors, Python retries 4 times and Java retries 5 times.For 503 errors, Python retries 19 times and Java retries 5 times.
Python's submit_job_description uses the default retry_on_server_errors_filter which does not retry on 4xx errors.Not sure where the extra python retry's are happening, but likely on the http transport level.
To verify, ran:
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 503 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null  ; echo "req $a $(date)"; done}}
or
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 429 Message\n" | nc -q 1 -N -l -p 1500 > /dev/null  ; echo "req $a $(date)"; done}}

Locally, then ran dataflow pipeline with --dataflow_endpoint=http://localhost:1500 (Python) or --dataflowEndpoint=http://localhost:1500 (java)

> Quota exceeded for create requests
> ----------------------------------
>
>                 Key: BEAM-8195
>                 URL: https://issues.apache.org/jira/browse/BEAM-8195
>             Project: Beam
>          Issue Type: Sub-task
>          Components: sdk-py-core, testing
>            Reporter: Ahmet Altay
>            Assignee: Yifan Zou
>            Priority: Critical
>
> Post commits failied with the following error:
> HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'server': 'ESF', '-content-encoding': 'gzip', 'content-type': 'application/json; charset=UTF-8', 'content-length': '598', 'transfer-encoding': 'chunked', 'cache-control': 'private', 'x-xss-protection': '0', 'date': 'Tue, 10 Sep 2019 12:02:24 GMT', 'vary': 'Origin, X-Origin, Referer', 'x-frame-options': 'SAMEORIGIN', 'status': '429', 'x-content-type-options': 'nosniff'}>, content <{
>   "error": {
>     "code": 429,
>     "message": "Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'.",
>     "status": "RESOURCE_EXHAUSTED",
>     "details": [
>       {
> Could we increase the quota?
> /cc [~alanmyrvold] [~kenn]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)