League table for flakey tests

[Ivan Kelly <iv...@apache.org>]

Hi folks,

We need to get a handle on the flakes we're in CI. I've yet to see a
build come back green on all runs. The flakes seem to be different
each time, so it's hard to see which to tackle first.

To this end, I've started a spreadsheet with a league table for flakes.
https://docs.google.com/spreadsheets/d/1CaYA7b4YlUGev8JqXBP8-Q3L_HAmfwY46FgzIb7Qpss/edit#gid=0

When you see a flake, and you're sure it's a flake (i.e. you've seen
the test pass on the same commit locally), either add a line, or
increment the number if it's already there.

This way we can identify which flakes are hurting us the most, and put
effort into dealing with them first.

Cheers,
Ivan

Re: League table for flakey tests

[Enrico Olivelli <eo...@gmail.com>]

Great idea.
Thank you

Enrico

Il mer 18 ott 2017, 22:26 Ivan Kelly <iv...@apache.org> ha scritto:

> Hi folks,
>
> We need to get a handle on the flakes we're in CI. I've yet to see a
> build come back green on all runs. The flakes seem to be different
> each time, so it's hard to see which to tackle first.
>
> To this end, I've started a spreadsheet with a league table for flakes.
>
> https://docs.google.com/spreadsheets/d/1CaYA7b4YlUGev8JqXBP8-Q3L_HAmfwY46FgzIb7Qpss/edit#gid=0
>
> When you see a flake, and you're sure it's a flake (i.e. you've seen
> the test pass on the same commit locally), either add a line, or
> increment the number if it's already there.
>
> This way we can identify which flakes are hurting us the most, and put
> effort into dealing with them first.
>
> Cheers,
> Ivan
>
-- 


-- Enrico Olivelli

Re: League table for flakey tests

[Sijie Guo <gu...@gmail.com>]

Yeah, agreed.

Sijie

On Oct 18, 2017 2:38 PM, "Ivan Kelly" <iv...@apache.org> wrote:

> We should also do that. But that alone doesn't give us an idea of
> which tests are flaking the most.
>
> On Wed, Oct 18, 2017 at 2:30 PM, Sijie Guo <gu...@gmail.com> wrote:
> > some of the tests were already marked @FlakyTest.
> > https://github.com/apache/bookkeeper/issues?q=flaky+is%3Aopen
> >
> > another idea is to have a issue label `FlakyTest`, create an issue when
> you
> > see a flake and label it as `FlakyTest`.
> >
> > - Sijie
> >
> > On Wed, Oct 18, 2017 at 1:26 PM, Ivan Kelly <iv...@apache.org> wrote:
> >
> >> Hi folks,
> >>
> >> We need to get a handle on the flakes we're in CI. I've yet to see a
> >> build come back green on all runs. The flakes seem to be different
> >> each time, so it's hard to see which to tackle first.
> >>
> >> To this end, I've started a spreadsheet with a league table for flakes.
> >> https://docs.google.com/spreadsheets/d/1CaYA7b4YlUGev8JqXBP8-Q3L_
> >> HAmfwY46FgzIb7Qpss/edit#gid=0
> >>
> >> When you see a flake, and you're sure it's a flake (i.e. you've seen
> >> the test pass on the same commit locally), either add a line, or
> >> increment the number if it's already there.
> >>
> >> This way we can identify which flakes are hurting us the most, and put
> >> effort into dealing with them first.
> >>
> >> Cheers,
> >> Ivan
> >>
>

Re: League table for flakey tests

[Ivan Kelly <iv...@apache.org>]

We should also do that. But that alone doesn't give us an idea of
which tests are flaking the most.

On Wed, Oct 18, 2017 at 2:30 PM, Sijie Guo <gu...@gmail.com> wrote:
> some of the tests were already marked @FlakyTest.
> https://github.com/apache/bookkeeper/issues?q=flaky+is%3Aopen
>
> another idea is to have a issue label `FlakyTest`, create an issue when you
> see a flake and label it as `FlakyTest`.
>
> - Sijie
>
> On Wed, Oct 18, 2017 at 1:26 PM, Ivan Kelly <iv...@apache.org> wrote:
>
>> Hi folks,
>>
>> We need to get a handle on the flakes we're in CI. I've yet to see a
>> build come back green on all runs. The flakes seem to be different
>> each time, so it's hard to see which to tackle first.
>>
>> To this end, I've started a spreadsheet with a league table for flakes.
>> https://docs.google.com/spreadsheets/d/1CaYA7b4YlUGev8JqXBP8-Q3L_
>> HAmfwY46FgzIb7Qpss/edit#gid=0
>>
>> When you see a flake, and you're sure it's a flake (i.e. you've seen
>> the test pass on the same commit locally), either add a line, or
>> increment the number if it's already there.
>>
>> This way we can identify which flakes are hurting us the most, and put
>> effort into dealing with them first.
>>
>> Cheers,
>> Ivan
>>

Re: League table for flakey tests

[Sijie Guo <gu...@gmail.com>]

some of the tests were already marked @FlakyTest.
https://github.com/apache/bookkeeper/issues?q=flaky+is%3Aopen

another idea is to have a issue label `FlakyTest`, create an issue when you
see a flake and label it as `FlakyTest`.

- Sijie

On Wed, Oct 18, 2017 at 1:26 PM, Ivan Kelly <iv...@apache.org> wrote:

> Hi folks,
>
> We need to get a handle on the flakes we're in CI. I've yet to see a
> build come back green on all runs. The flakes seem to be different
> each time, so it's hard to see which to tackle first.
>
> To this end, I've started a spreadsheet with a league table for flakes.
> https://docs.google.com/spreadsheets/d/1CaYA7b4YlUGev8JqXBP8-Q3L_
> HAmfwY46FgzIb7Qpss/edit#gid=0
>
> When you see a flake, and you're sure it's a flake (i.e. you've seen
> the test pass on the same commit locally), either add a line, or
> increment the number if it's already there.
>
> This way we can identify which flakes are hurting us the most, and put
> effort into dealing with them first.
>
> Cheers,
> Ivan
>