You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Mikhail Gryzykhin <mi...@google.com> on 2019/10/28 21:05:09 UTC
Quota issues again
Hi everyone,
While validating release branch, I got failure due Quota again. Also,
current queue time for jobs is more than 1.5 hours.
I'm not sure if it is worth starting another thread on tests
efficiency, but still want to keep this mail to highlight the issues.
See PS for links.
Regards,
--Mikhail
PS:
https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
*13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed:
googleapi: Error 429: Quota exceeded for quota metric
'dataflow.googleapis.com/create_requests' and limit
'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com'
for consumer 'project_number:844138762903'., rateLimitExceeded
Queue time:
http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
Re: Quota issues again
Posted by Kenneth Knowles <ke...@apache.org>.
Post-commit runs all precommits. The builder for the Jenkins jobs creates
separate jobs with suffixes:
* _Commit (for when a commit is pushed to a PR)
* _Phrase (for when someone asks to run it)
* _Cron (run as a post-commit against master)
This way, the different jobs have independent configuration and the
post-commit version of them has a clear healthy/unhealthy signal.
My preference FWIW would be to use "precommit" and "postcommit" to refer
_only_ to when a suite of tests is run and _never_ as part of the name of
any suite of tests. There seems to be frequent confusion about things due
to the names. That would be a medium size change to gradle configs, Jenkins
configs, and cultural.
Kenn
On Tue, Oct 29, 2019 at 10:41 AM Mikhail Gryzykhin <mi...@google.com>
wrote:
> IIRC currently, post-commit doesn't run pre-commits. However we have
> precommit_cron jobs that run pre-commits periodically. However it sums up
> to dozens of jobs that is really hard to monitor.
>
> If we split things even further, we definitely need to combine result into
> something more easily trackable.
>
> Also making post-commits bigger is not that good idea either, since it
> will make them even more flaky and any PR that needs to run them can get
> stuck forever.
>
> Main point is that we want to do some work around improving monitoring,
> not simply make more post-commits, or bigger post-commits.
>
> On Tue, Oct 29, 2019 at 9:56 AM Chad Dombrova <ch...@gmail.com> wrote:
>
>>
>> +1 for splitting pre-commit tests into smaller modules. However in this
>>> case we need to run all the small tests periodically and have some combined
>>> flag or dashboard for regular monitoring. Otherwise we might not run/check
>>> on big amount of tests.
>>>
>>
>> post-commit seems like the best place for that, no?
>>
>>
>>
>
Re: Quota issues again
Posted by Mikhail Gryzykhin <mi...@google.com>.
IIRC currently, post-commit doesn't run pre-commits. However we have
precommit_cron jobs that run pre-commits periodically. However it sums up
to dozens of jobs that is really hard to monitor.
If we split things even further, we definitely need to combine result into
something more easily trackable.
Also making post-commits bigger is not that good idea either, since it will
make them even more flaky and any PR that needs to run them can get stuck
forever.
Main point is that we want to do some work around improving monitoring, not
simply make more post-commits, or bigger post-commits.
On Tue, Oct 29, 2019 at 9:56 AM Chad Dombrova <ch...@gmail.com> wrote:
>
> +1 for splitting pre-commit tests into smaller modules. However in this
>> case we need to run all the small tests periodically and have some combined
>> flag or dashboard for regular monitoring. Otherwise we might not run/check
>> on big amount of tests.
>>
>
> post-commit seems like the best place for that, no?
>
>
>
Re: Quota issues again
Posted by Chad Dombrova <ch...@gmail.com>.
> +1 for splitting pre-commit tests into smaller modules. However in this
> case we need to run all the small tests periodically and have some combined
> flag or dashboard for regular monitoring. Otherwise we might not run/check
> on big amount of tests.
>
post-commit seems like the best place for that, no?
Re: Quota issues again
Posted by Mikhail Gryzykhin <mi...@google.com>.
+1 for splitting pre-commit tests into smaller modules. However in this
case we need to run all the small tests periodically and have some combined
flag or dashboard for regular monitoring. Otherwise we might not run/check
on big amount of tests.
On Mon, Oct 28, 2019 at 6:39 PM Kenneth Knowles <ke...@apache.org> wrote:
> It may also be advantageous to separate most submodules to not run a giant
> generic Java precommit. Each IO really only needs its own, and to register
> itself in the global Java precommit run only for the core. The bookkeeping
> may become quite a lot, but this is the natural structure.
>
> Kenn
>
> On Mon, Oct 28, 2019 at 6:12 PM Chad Dombrova <ch...@gmail.com> wrote:
>
>> Can we get more aggressive about separating tests into groups by those
>> that are dependent on other languages and those that are not? I think we
>> could dramatically reduce our backlog if we didn’t run all of the Java
>> tests every time a commit is made that only affects python code, and vice
>> versa.
>>
>> -chad
>>
>>
>> On Mon, Oct 28, 2019 at 3:05 PM Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>>
>>> Quota jira issue:
>>> https://issues.apache.org/jira/browse/BEAM-8195
>>>
>>> On Mon, Oct 28, 2019 at 2:05 PM Mikhail Gryzykhin <mi...@google.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>>
>>>> While validating release branch, I got failure due Quota again. Also, current queue time for jobs is more than 1.5 hours.
>>>>
>>>>
>>>> I'm not sure if it is worth starting another thread on tests efficiency, but still want to keep this mail to highlight the issues.
>>>>
>>>>
>>>> See PS for links.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> --Mikhail
>>>>
>>>>
>>>> PS:
>>>>
>>>> https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
>>>>
>>>> *13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed: googleapi: Error 429: Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'., rateLimitExceeded
>>>>
>>>>
>>>> Queue time:
>>>>
>>>> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>>>>
>>>>
Re: Quota issues again
Posted by Kenneth Knowles <ke...@apache.org>.
It may also be advantageous to separate most submodules to not run a giant
generic Java precommit. Each IO really only needs its own, and to register
itself in the global Java precommit run only for the core. The bookkeeping
may become quite a lot, but this is the natural structure.
Kenn
On Mon, Oct 28, 2019 at 6:12 PM Chad Dombrova <ch...@gmail.com> wrote:
> Can we get more aggressive about separating tests into groups by those
> that are dependent on other languages and those that are not? I think we
> could dramatically reduce our backlog if we didn’t run all of the Java
> tests every time a commit is made that only affects python code, and vice
> versa.
>
> -chad
>
>
> On Mon, Oct 28, 2019 at 3:05 PM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Quota jira issue:
>> https://issues.apache.org/jira/browse/BEAM-8195
>>
>> On Mon, Oct 28, 2019 at 2:05 PM Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>>
>>> While validating release branch, I got failure due Quota again. Also, current queue time for jobs is more than 1.5 hours.
>>>
>>>
>>> I'm not sure if it is worth starting another thread on tests efficiency, but still want to keep this mail to highlight the issues.
>>>
>>>
>>> See PS for links.
>>>
>>>
>>> Regards,
>>>
>>> --Mikhail
>>>
>>>
>>> PS:
>>>
>>> https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
>>>
>>> *13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed: googleapi: Error 429: Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'., rateLimitExceeded
>>>
>>>
>>> Queue time:
>>>
>>> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>>>
>>>
Re: Quota issues again
Posted by Chad Dombrova <ch...@gmail.com>.
Can we get more aggressive about separating tests into groups by those that
are dependent on other languages and those that are not? I think we could
dramatically reduce our backlog if we didn’t run all of the Java tests
every time a commit is made that only affects python code, and vice versa.
-chad
On Mon, Oct 28, 2019 at 3:05 PM Mikhail Gryzykhin <mi...@google.com> wrote:
> Quota jira issue:
> https://issues.apache.org/jira/browse/BEAM-8195
>
> On Mon, Oct 28, 2019 at 2:05 PM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Hi everyone,
>>
>>
>> While validating release branch, I got failure due Quota again. Also, current queue time for jobs is more than 1.5 hours.
>>
>>
>> I'm not sure if it is worth starting another thread on tests efficiency, but still want to keep this mail to highlight the issues.
>>
>>
>> See PS for links.
>>
>>
>> Regards,
>>
>> --Mikhail
>>
>>
>> PS:
>>
>> https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
>>
>> *13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed: googleapi: Error 429: Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'., rateLimitExceeded
>>
>>
>> Queue time:
>>
>> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>>
>>
Re: Quota issues again
Posted by Mikhail Gryzykhin <mi...@google.com>.
Quota jira issue:
https://issues.apache.org/jira/browse/BEAM-8195
On Mon, Oct 28, 2019 at 2:05 PM Mikhail Gryzykhin <mi...@google.com> wrote:
> Hi everyone,
>
>
> While validating release branch, I got failure due Quota again. Also, current queue time for jobs is more than 1.5 hours.
>
>
> I'm not sure if it is worth starting another thread on tests efficiency, but still want to keep this mail to highlight the issues.
>
>
> See PS for links.
>
>
> Regards,
>
> --Mikhail
>
>
> PS:
>
> https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
>
> *13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed: googleapi: Error 429: Quota exceeded for quota metric 'dataflow.googleapis.com/create_requests' and limit 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for consumer 'project_number:844138762903'., rateLimitExceeded
>
>
> Queue time:
>
> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>
>