[jira] [Commented] (BEAM-5053) CreateRequestsPerMinutePerUser quota limit exceeded

["Mikhail Gryzykhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-5053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567491#comment-16567491 ] 

Mikhail Gryzykhin commented on BEAM-5053:
-----------------------------------------

I suspect that the reason for this quota is that we allow concurrent runs for postcommit test jobs that are triggered on each commit.

I've disabled concurrency for automatically triggered postcommit jobs. This should resolve our current issue with quota. I'll monitor the state for couple of days and followup accordingly.

> CreateRequestsPerMinutePerUser quota limit exceeded
> ---------------------------------------------------
>
>                 Key: BEAM-5053
>                 URL: https://issues.apache.org/jira/browse/BEAM-5053
>             Project: Beam
>          Issue Type: Bug
>          Components: test-failures, testing
>            Reporter: Mikhail Gryzykhin
>            Assignee: Mikhail Gryzykhin
>            Priority: Major
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/553/consoleFull]
>  
> Dataflow has a CreateRequestsPerMinutePerUser quota that is 60 requests per minute.
> Our Java Validate Runner tests spin up more than 250 jobs in a run.
> We should look into best way to fix the issue. Currently, I see following options:
>  # Add retry logic. Although this limits us to 1 dataflow job start per second for whole Jenkins. In long scale this can also block one test job if other jobs take all the slots.
>  # Utilize different users to spin Dataflow jobs.
>  # Find way to rise quota limit on Dataflow. By default the field limits value to 60 requests per minute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)