You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Ivan Kelly <iv...@apache.org> on 2018/01/24 13:20:55 UTC
[DISCUSS] Thoughts on enabling rerunFailedTests in surefire?
Hi folks,
Looking at open PRs right now, there's a lot of red Xs
https://screenshotscdn.firefoxusercontent.com/images/8df9daf7-d71a-4988-a9cc-69d9c10c2636.png
I see only 3 green. I know it's not all flakes, but a good amount of
it is. Enough to make a red X meaningless.
So I propose we enable the rerunFailingTest option in surefire. This
will rerun the flakes. I suggest we set to 2. Anything that it flaking
more than once per run needs to be looked at.
http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#rerunFailingTestsCount
As part of this, I suggest that we also institute a no submission on
red policy (i.e. tests have to get full green to be merged).
Regards,
Ivan
Re: [DISCUSS] Thoughts on enabling rerunFailedTests in surefire?
Posted by Sijie Guo <gu...@gmail.com>.
On Wed, Jan 24, 2018 at 5:20 AM, Ivan Kelly <iv...@apache.org> wrote:
> Hi folks,
>
> Looking at open PRs right now, there's a lot of red Xs
>
> https://screenshotscdn.firefoxusercontent.com/images/
> 8df9daf7-d71a-4988-a9cc-69d9c10c2636.png
>
> I see only 3 green. I know it's not all flakes, but a good amount of
> it is. Enough to make a red X meaningless.
>
just fyi. I think we are much better these days. I typically found one of
the jekinds CI passed at those red-Xs PRs.
>
> So I propose we enable the rerunFailingTest option in surefire. This
> will rerun the flakes. I suggest we set to 2. Anything that it flaking
> more than once per run needs to be looked at.
>
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#
> rerunFailingTestsCount
>
> As part of this, I suggest that we also institute a no submission on
> red policy (i.e. tests have to get full green to be merged).
>
> Regards,
> Ivan
>
Re: [DISCUSS] Thoughts on enabling rerunFailedTests in surefire?
Posted by Enrico Olivelli <eo...@gmail.com>.
Il mer 24 gen 2018, 17:56 Sijie Guo <gu...@gmail.com> ha scritto:
> On Wed, Jan 24, 2018 at 5:20 AM Ivan Kelly <iv...@apache.org> wrote:
>
> > Hi folks,
> >
> > Looking at open PRs right now, there's a lot of red Xs
> >
> >
> >
> https://screenshotscdn.firefoxusercontent.com/images/8df9daf7-d71a-4988-a9cc-69d9c10c2636.png
> >
> > I see only 3 green. I know it's not all flakes, but a good amount of
> > it is. Enough to make a red X meaningless.
> >
> > So I propose we enable the rerunFailingTest option in surefire. This
> > will rerun the flakes. I suggest we set to 2. Anything that it flaking
> > more than once per run needs to be looked at.
> >
> >
> >
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#rerunFailingTestsCount
>
>
> Good idea. +1
>
+1 let's try
>
> <
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#rerunFailingTestsCount
> >
> >
> > As part of this, I suggest that we also institute a no submission on
> > red policy (i.e. tests have to get full green to be merged).
>
>
> FYI. I created one issue for same purpose before:
>
> https://github.com/apache/bookkeeper/issues/1031
+1
Enrico
>
>
>
>
>
>
> >
> > Regards,
> > Ivan
> >
>
--
-- Enrico Olivelli
Re: [DISCUSS] Thoughts on enabling rerunFailedTests in surefire?
Posted by Ivan Kelly <iv...@apache.org>.
I've enabled the flag now
Search for Flake on [1] to see what a flake output looks like. I'll
write a script to collate these, so we can see what is flaking.
> Strictly tests (especially CI tests) must not be flaky. It really doesn't
> carry much use. As everyone tends to ignore them.
> Step-1: Separate them into a different pool, and you won't get a red mark
> for failing this pool/group.
> Step-2: Fix each test to be consistent and move it out of the optional pool.
> Step-3: If there are any tests that can't be consistent but makes sense
> (???) to keep them part of CI, we can leave them in this flaky/optional
> pool.
I like this idea of a quarantine for flakes. Do you know how to do
this with junit/maven?
-Ivan
[1] https://builds.apache.org/job/bookkeeper_precommit_pullrequest_java8/179/org.apache.bookkeeper$bookkeeper-server/consoleText
Re: [DISCUSS] Thoughts on enabling rerunFailedTests in surefire?
Posted by Venkateswara Rao Jujjuri <ju...@gmail.com>.
Great Idea. In addition, I would like to propose :
Strictly tests (especially CI tests) must not be flaky. It really doesn't
carry much use. As everyone tends to ignore them.
Step-1: Separate them into a different pool, and you won't get a red mark
for failing this pool/group.
Step-2: Fix each test to be consistent and move it out of the optional pool.
Step-3: If there are any tests that can't be consistent but makes sense
(???) to keep them part of CI, we can leave them in this flaky/optional
pool.
Thanks,
JV
On Wed, Jan 24, 2018 at 8:56 AM, Sijie Guo <gu...@gmail.com> wrote:
> On Wed, Jan 24, 2018 at 5:20 AM Ivan Kelly <iv...@apache.org> wrote:
>
> > Hi folks,
> >
> > Looking at open PRs right now, there's a lot of red Xs
> >
> >
> > https://screenshotscdn.firefoxusercontent.com/images/
> 8df9daf7-d71a-4988-a9cc-69d9c10c2636.png
> >
> > I see only 3 green. I know it's not all flakes, but a good amount of
> > it is. Enough to make a red X meaningless.
> >
> > So I propose we enable the rerunFailingTest option in surefire. This
> > will rerun the flakes. I suggest we set to 2. Anything that it flaking
> > more than once per run needs to be looked at.
> >
> >
> > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#
> rerunFailingTestsCount
>
>
> Good idea. +1
>
> <http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#
> rerunFailingTestsCount>
> >
> > As part of this, I suggest that we also institute a no submission on
> > red policy (i.e. tests have to get full green to be merged).
>
>
> FYI. I created one issue for same purpose before:
>
> https://github.com/apache/bookkeeper/issues/1031
>
>
>
> >
> > Regards,
> > Ivan
> >
>
--
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi
Re: [DISCUSS] Thoughts on enabling rerunFailedTests in surefire?
Posted by Sijie Guo <gu...@gmail.com>.
On Wed, Jan 24, 2018 at 5:20 AM Ivan Kelly <iv...@apache.org> wrote:
> Hi folks,
>
> Looking at open PRs right now, there's a lot of red Xs
>
>
> https://screenshotscdn.firefoxusercontent.com/images/8df9daf7-d71a-4988-a9cc-69d9c10c2636.png
>
> I see only 3 green. I know it's not all flakes, but a good amount of
> it is. Enough to make a red X meaningless.
>
> So I propose we enable the rerunFailingTest option in surefire. This
> will rerun the flakes. I suggest we set to 2. Anything that it flaking
> more than once per run needs to be looked at.
>
>
> http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#rerunFailingTestsCount
Good idea. +1
<http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#rerunFailingTestsCount>
>
> As part of this, I suggest that we also institute a no submission on
> red policy (i.e. tests have to get full green to be merged).
FYI. I created one issue for same purpose before:
https://github.com/apache/bookkeeper/issues/1031
>
> Regards,
> Ivan
>