You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Qingzhou Luo <qi...@google.com> on 2014/05/27 19:14:33 UTC
Extend surefire to rerun failing tests
Hi,
I am an intern at Google. The first step of my internship project is to add
the ability to Maven to automatically rerun failing tests a few times, to
see if they ever pass in any of the reruns. It is useful because in many
cases a test fails because it is flaky, not because there is a bug in the
new source code change.
We think the right way to achieve this is to modify surefire plugin of
maven. We want to add it as a part of the configuration of surefire, so
users can decide whether they want to enable this feature, and how many
times they want to rerun failing tests. We plan to open-source our
contribution, and hopefully can merge our code into surefire master branch
in the end. Therefore, we are wondering do you have any
comments/suggestions/opinions regarding this? We appreciate any input.
Thank you very much!
Regards,
Qingzhou
Re: Extend surefire to rerun failing tests
Posted by Kristian Rosenvold <kr...@gmail.com>.
2014-05-28 14:41 GMT+02:00 Jason van Zyl <ja...@takari.io>:
> Maybe a simple mechanism where you store the failed tests in a file and then have a flag to run only the failed tests. I think this would be very useful. Then possible a small, configurable loop around the main execution of tests if you wanted to specify how many times to run the tests.
Surefire already does this with the runorder=failedfirst. Most of the
other behaviour is handled quite all by junit rules, but since we
already store the failure/success information, I'm sure there's room
for other options :)
Kristian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Extend surefire to rerun failing tests
Posted by Andreas Gudian <an...@gmail.com>.
My first approach would also be to handle this in the test framework, e.g.
using the JUnit Rules as Kristian already suggested.
There you can decide on a much more fine-grained level what you want to do
in case of a failing test: re-run all the tests of the class? Or only those
test cases that failed? Perform the re-run of a test case immediately, or
after the other tests in the same class were executed?
2014-05-28 21:43 GMT+02:00 Kristian Rosenvold <kr...@gmail.com>
:
> I would really like to see and understand properly how this would
> collaborate with JUnit rules for retries. I suspect there might be
> some interesting issues regarding reporting (i.e. can the existing
> logic handle reporting of n different executions of a single test).
> How does this explode i parallel junit is selected :) ?
>
> Technically you can just pass a system property from your pom and into
> the surefire fork with the number of retries, you can read this
> proprety in the junit rule. I really think you should start with this
> approach and flesh out the details from there. It might turn out this
> is all you need, or that some other entirely unforseen issues pop up.
>
> As for the actual property in the plugin, I really think we should see
> where this leads before deciding on how to do this. On a similar note,
> I know some people with large corporate clouds sometimes run a test on
> 10000 different nodes in parallel, to investigate if there is any kind
> of 1/10000 type failures, just to be able to summarize in 20 seconds
> instead of 10000 linear runs.
>
> This might also be another one of those cases where we could consider
> picking up user-written code from the test classpath; but I think I
> need another year thinking about that :)
>
> Kristian
>
>
>
>
> 2014-05-28 19:58 GMT+02:00 Jason van Zyl <ja...@takari.io>:
> > I personally don't see an issue with those feature additions, but
> Kristian does the lions share of the work on the Surefire plugin so it
> would be his call. I would definitely support the additions of those
> features and would help test.
> >
> > On May 28, 2014, at 1:46 PM, Qingzhou Luo <qi...@google.com>
> wrote:
> >
> >> Hi Jason,
> >>
> >> Thanks for your reply. We plan to first extend surefire, so users can
> put a boolean parameter rerunFailintTests and an int parameter
> rerunFailintTestsCount, to tell surefire whether to rerun failing tests
> immediately after they fail. If a test passes in any of those reruns if
> will be marked as pass, but all the failing attempts will still be recorded
> and may be presented to users in a different way.
> >>
> >> We plan to finish everything in next 1-2 months. So we wonder would you
> be willing to accept our contribution in the end, and patch our code into
> surefire?
> >>
> >> Thanks,
> >>
> >> Qingzhou
> >>
> >>
> >> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <ja...@takari.io> wrote:
> >>
> >> On May 27, 2014, at 1:14 PM, Qingzhou Luo <qi...@google.com>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am an intern at Google. The first step of my internship project is
> to add
> >>> the ability to Maven to automatically rerun failing tests a few times,
> to
> >>> see if they ever pass in any of the reruns. It is useful because in
> many
> >>> cases a test fails because it is flaky, not because there is a bug in
> the
> >>> new source code change.
> >>>
> >>
> >> I think generally being able to re-run failed tests is useful. I use
> this feature in the IDE all the time in a fail/fix/run/repeat scenario. If
> you're trying to identify flaky tests I think that's useful especially if
> you can collect statistics. While In the long term you may not want to rely
> on the re-running of failed tests that later succeed without alteration as
> the basis for good testing, but it's a good identification tool, but a
> useful feature in its own right.
> >>
> >>> We think the right way to achieve this is to modify surefire plugin of
> >>> maven. We want to add it as a part of the configuration of surefire, so
> >>> users can decide whether they want to enable this feature, and how many
> >>> times they want to rerun failing tests. We plan to open-source our
> >>> contribution, and hopefully can merge our code into surefire master
> branch
> >>> in the end. Therefore, we are wondering do you have any
> >>> comments/suggestions/opinions regarding this? We appreciate any input.
> >>>
> >>
> >> Maybe a simple mechanism where you store the failed tests in a file and
> then have a flag to run only the failed tests. I think this would be very
> useful. Then possible a small, configurable loop around the main execution
> of tests if you wanted to specify how many times to run the tests.
> >>
> >>> Thank you very much!
> >>>
> >>> Regards,
> >>>
> >>> Qingzhou
> >>
> >> Thanks,
> >>
> >> Jason
> >>
> >> ----------------------------------------------------------
> >> Jason van Zyl
> >> Founder, Apache Maven
> >> http://twitter.com/jvanzyl
> >> http://twitter.com/takari_io
> >> ---------------------------------------------------------
> >>
> >> The modern conservative is engaged in one of man's oldest exercises in
> moral philosophy; that is,
> >> the search for a superior moral justification for selfishness.
> >>
> >> -- John Kenneth Galbraith
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> > Thanks,
> >
> > Jason
> >
> > ----------------------------------------------------------
> > Jason van Zyl
> > Founder, Apache Maven
> > http://twitter.com/jvanzyl
> > http://twitter.com/takari_io
> > ---------------------------------------------------------
> >
> > Selfish deeds are the shortest path to self destruction.
> >
> > -- The Seven Samuari, Akira Kurosawa
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Extend surefire to rerun failing tests
Posted by Kristian Rosenvold <kr...@gmail.com>.
I would really like to see and understand properly how this would
collaborate with JUnit rules for retries. I suspect there might be
some interesting issues regarding reporting (i.e. can the existing
logic handle reporting of n different executions of a single test).
How does this explode i parallel junit is selected :) ?
Technically you can just pass a system property from your pom and into
the surefire fork with the number of retries, you can read this
proprety in the junit rule. I really think you should start with this
approach and flesh out the details from there. It might turn out this
is all you need, or that some other entirely unforseen issues pop up.
As for the actual property in the plugin, I really think we should see
where this leads before deciding on how to do this. On a similar note,
I know some people with large corporate clouds sometimes run a test on
10000 different nodes in parallel, to investigate if there is any kind
of 1/10000 type failures, just to be able to summarize in 20 seconds
instead of 10000 linear runs.
This might also be another one of those cases where we could consider
picking up user-written code from the test classpath; but I think I
need another year thinking about that :)
Kristian
2014-05-28 19:58 GMT+02:00 Jason van Zyl <ja...@takari.io>:
> I personally don't see an issue with those feature additions, but Kristian does the lions share of the work on the Surefire plugin so it would be his call. I would definitely support the additions of those features and would help test.
>
> On May 28, 2014, at 1:46 PM, Qingzhou Luo <qi...@google.com> wrote:
>
>> Hi Jason,
>>
>> Thanks for your reply. We plan to first extend surefire, so users can put a boolean parameter rerunFailintTests and an int parameter rerunFailintTestsCount, to tell surefire whether to rerun failing tests immediately after they fail. If a test passes in any of those reruns if will be marked as pass, but all the failing attempts will still be recorded and may be presented to users in a different way.
>>
>> We plan to finish everything in next 1-2 months. So we wonder would you be willing to accept our contribution in the end, and patch our code into surefire?
>>
>> Thanks,
>>
>> Qingzhou
>>
>>
>> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <ja...@takari.io> wrote:
>>
>> On May 27, 2014, at 1:14 PM, Qingzhou Luo <qi...@google.com> wrote:
>>
>>> Hi,
>>>
>>> I am an intern at Google. The first step of my internship project is to add
>>> the ability to Maven to automatically rerun failing tests a few times, to
>>> see if they ever pass in any of the reruns. It is useful because in many
>>> cases a test fails because it is flaky, not because there is a bug in the
>>> new source code change.
>>>
>>
>> I think generally being able to re-run failed tests is useful. I use this feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're trying to identify flaky tests I think that's useful especially if you can collect statistics. While In the long term you may not want to rely on the re-running of failed tests that later succeed without alteration as the basis for good testing, but it's a good identification tool, but a useful feature in its own right.
>>
>>> We think the right way to achieve this is to modify surefire plugin of
>>> maven. We want to add it as a part of the configuration of surefire, so
>>> users can decide whether they want to enable this feature, and how many
>>> times they want to rerun failing tests. We plan to open-source our
>>> contribution, and hopefully can merge our code into surefire master branch
>>> in the end. Therefore, we are wondering do you have any
>>> comments/suggestions/opinions regarding this? We appreciate any input.
>>>
>>
>> Maybe a simple mechanism where you store the failed tests in a file and then have a flag to run only the failed tests. I think this would be very useful. Then possible a small, configurable loop around the main execution of tests if you wanted to specify how many times to run the tests.
>>
>>> Thank you very much!
>>>
>>> Regards,
>>>
>>> Qingzhou
>>
>> Thanks,
>>
>> Jason
>>
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder, Apache Maven
>> http://twitter.com/jvanzyl
>> http://twitter.com/takari_io
>> ---------------------------------------------------------
>>
>> The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is,
>> the search for a superior moral justification for selfishness.
>>
>> -- John Kenneth Galbraith
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> http://twitter.com/jvanzyl
> http://twitter.com/takari_io
> ---------------------------------------------------------
>
> Selfish deeds are the shortest path to self destruction.
>
> -- The Seven Samuari, Akira Kurosawa
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Extend surefire to rerun failing tests
Posted by Jason van Zyl <ja...@takari.io>.
I personally don't see an issue with those feature additions, but Kristian does the lions share of the work on the Surefire plugin so it would be his call. I would definitely support the additions of those features and would help test.
On May 28, 2014, at 1:46 PM, Qingzhou Luo <qi...@google.com> wrote:
> Hi Jason,
>
> Thanks for your reply. We plan to first extend surefire, so users can put a boolean parameter rerunFailintTests and an int parameter rerunFailintTestsCount, to tell surefire whether to rerun failing tests immediately after they fail. If a test passes in any of those reruns if will be marked as pass, but all the failing attempts will still be recorded and may be presented to users in a different way.
>
> We plan to finish everything in next 1-2 months. So we wonder would you be willing to accept our contribution in the end, and patch our code into surefire?
>
> Thanks,
>
> Qingzhou
>
>
> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <ja...@takari.io> wrote:
>
> On May 27, 2014, at 1:14 PM, Qingzhou Luo <qi...@google.com> wrote:
>
>> Hi,
>>
>> I am an intern at Google. The first step of my internship project is to add
>> the ability to Maven to automatically rerun failing tests a few times, to
>> see if they ever pass in any of the reruns. It is useful because in many
>> cases a test fails because it is flaky, not because there is a bug in the
>> new source code change.
>>
>
> I think generally being able to re-run failed tests is useful. I use this feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're trying to identify flaky tests I think that's useful especially if you can collect statistics. While In the long term you may not want to rely on the re-running of failed tests that later succeed without alteration as the basis for good testing, but it's a good identification tool, but a useful feature in its own right.
>
>> We think the right way to achieve this is to modify surefire plugin of
>> maven. We want to add it as a part of the configuration of surefire, so
>> users can decide whether they want to enable this feature, and how many
>> times they want to rerun failing tests. We plan to open-source our
>> contribution, and hopefully can merge our code into surefire master branch
>> in the end. Therefore, we are wondering do you have any
>> comments/suggestions/opinions regarding this? We appreciate any input.
>>
>
> Maybe a simple mechanism where you store the failed tests in a file and then have a flag to run only the failed tests. I think this would be very useful. Then possible a small, configurable loop around the main execution of tests if you wanted to specify how many times to run the tests.
>
>> Thank you very much!
>>
>> Regards,
>>
>> Qingzhou
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> http://twitter.com/jvanzyl
> http://twitter.com/takari_io
> ---------------------------------------------------------
>
> The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is,
> the search for a superior moral justification for selfishness.
>
> -- John Kenneth Galbraith
>
>
>
>
>
>
>
>
>
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
http://twitter.com/jvanzyl
http://twitter.com/takari_io
---------------------------------------------------------
Selfish deeds are the shortest path to self destruction.
-- The Seven Samuari, Akira Kurosawa
Re: Extend surefire to rerun failing tests
Posted by Qingzhou Luo <qi...@google.com>.
Hi Jason,
Thanks for your reply. We plan to first extend surefire, so users can put a
boolean parameter rerunFailintTests and an int parameter
rerunFailintTestsCount, to tell surefire whether to rerun failing tests
immediately after they fail. If a test passes in any of those reruns if
will be marked as pass, but all the failing attempts will still be recorded
and may be presented to users in a different way.
We plan to finish everything in next 1-2 months. So we wonder would you be
willing to accept our contribution in the end, and patch our code into
surefire?
Thanks,
Qingzhou
On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <ja...@takari.io> wrote:
>
> On May 27, 2014, at 1:14 PM, Qingzhou Luo <qi...@google.com> wrote:
>
> Hi,
>
> I am an intern at Google. The first step of my internship project is to add
> the ability to Maven to automatically rerun failing tests a few times, to
> see if they ever pass in any of the reruns. It is useful because in many
> cases a test fails because it is flaky, not because there is a bug in the
> new source code change.
>
>
> I think generally being able to re-run failed tests is useful. I use this
> feature in the IDE all the time in a fail/fix/run/repeat scenario. If
> you're trying to identify flaky tests I think that's useful especially if
> you can collect statistics. While In the long term you may not want to rely
> on the re-running of failed tests that later succeed without alteration as
> the basis for good testing, but it's a good identification tool, but a
> useful feature in its own right.
>
> We think the right way to achieve this is to modify surefire plugin of
> maven. We want to add it as a part of the configuration of surefire, so
> users can decide whether they want to enable this feature, and how many
> times they want to rerun failing tests. We plan to open-source our
> contribution, and hopefully can merge our code into surefire master branch
> in the end. Therefore, we are wondering do you have any
> comments/suggestions/opinions regarding this? We appreciate any input.
>
>
> Maybe a simple mechanism where you store the failed tests in a file and
> then have a flag to run only the failed tests. I think this would be very
> useful. Then possible a small, configurable loop around the main execution
> of tests if you wanted to specify how many times to run the tests.
>
> Thank you very much!
>
> Regards,
>
> Qingzhou
>
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> http://twitter.com/jvanzyl
> http://twitter.com/takari_io
> ---------------------------------------------------------
>
>
> The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is,
> the search for a superior moral justification for selfishness.
>
> -- John Kenneth Galbraith
>
>
>
>
>
>
>
>
>
>
Re: Extend surefire to rerun failing tests
Posted by Jason van Zyl <ja...@takari.io>.
On May 27, 2014, at 1:14 PM, Qingzhou Luo <qi...@google.com> wrote:
> Hi,
>
> I am an intern at Google. The first step of my internship project is to add
> the ability to Maven to automatically rerun failing tests a few times, to
> see if they ever pass in any of the reruns. It is useful because in many
> cases a test fails because it is flaky, not because there is a bug in the
> new source code change.
>
I think generally being able to re-run failed tests is useful. I use this feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're trying to identify flaky tests I think that's useful especially if you can collect statistics. While In the long term you may not want to rely on the re-running of failed tests that later succeed without alteration as the basis for good testing, but it's a good identification tool, but a useful feature in its own right.
> We think the right way to achieve this is to modify surefire plugin of
> maven. We want to add it as a part of the configuration of surefire, so
> users can decide whether they want to enable this feature, and how many
> times they want to rerun failing tests. We plan to open-source our
> contribution, and hopefully can merge our code into surefire master branch
> in the end. Therefore, we are wondering do you have any
> comments/suggestions/opinions regarding this? We appreciate any input.
>
Maybe a simple mechanism where you store the failed tests in a file and then have a flag to run only the failed tests. I think this would be very useful. Then possible a small, configurable loop around the main execution of tests if you wanted to specify how many times to run the tests.
> Thank you very much!
>
> Regards,
>
> Qingzhou
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
http://twitter.com/jvanzyl
http://twitter.com/takari_io
---------------------------------------------------------
The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is,
the search for a superior moral justification for selfishness.
-- John Kenneth Galbraith
Re: Extend surefire to rerun failing tests
Posted by Alexander Kriegisch <Al...@Kriegisch.name>.
I have not tried, Kristian, but it looks promising. Both your link and the randomised testing one I have forwarded to my old Scrum team (I am no longer coaching them, thus unable to access their code base and try for myself).
Thank you. :-)
--
Alexander Kriegisch
> Am 28.05.2014 um 09:50 schrieb Kristian Rosenvold <kr...@gmail.com>:
>
> Does this fit the bill for you ?
>
> http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately
>
> Kristian
>
>
> 2014-05-28 9:14 GMT+02:00 Alexander Kriegisch <Al...@kriegisch.name>:
>>
>>
>> --
>> Alexander Kriegisch
>> http://scrum-master.de
>>
>>> Am 28.05.2014 um 05:13 schrieb Benson Margulies <bi...@gmail.com>:
>>>
>>> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <qi...@google.com>wrote:
>>>
>>>> I am an intern at Google. The first step of my internship project is to add
>>>> the ability to Maven to automatically rerun failing tests a few times, to
>>>> see if they ever pass in any of the reruns. It is useful because in many
>>>> cases a test fails because it is flaky, not because there is a bug in the
>>>> new source code change.
>>>
>>> I entirely disagree. 99.9999% of programs are completely deterministic. If
>>> you feed them the same inputs, you get the same output.
>>
>> Well, in my previous project (I am an Agile Coach) my teams were working on a distributed, multi-threaded system and around 10% of all unit tests and next to all integration tests were affected by the exception you mentioned:
>>
>>> The exceptions are tests that use threads, and it's quite hard to do a good
>>> job writing them.
>>
>> One can argue that several of the unit tests were not real unit tests because they mocked some, but not all of the infrastructure they needed for running. Anyway, they were running in Surefire (the integration tests in Failsafe), and some lf them were flaky.
>>
>>> Just 'rerunning a few times' does not lead to useful
>>> information in my view.
>>
>> Only the information that indeed they are flaky and a rough percentage of cases in which they fail, which would be rather dependend on the test environment though. Better than nothing information-wise. So far, so good.
>>
>> Having said all that, I still agree to Benson that a failing test should just fail, period. I do not want developers to think, "it is only failing only in 5% of all runs, so it is fine / low priority to fix". OTOH I understand Qingzhou from a perspective of wishing to "test the test" or rather wishing to harden a flaky test into a stable one, either by fixing the test or the threading bug(s) it exposes. Benson's suggestion below looks promising though, so thanks for that.
>>
>>> if you want random testing, there's a fine alternative at
>>> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine
>>> maven plugin. It can already 'rerun your tests a few times' very easily.
>>>
>>> I wonder what the more involved surefire maintainers think, but as a PMC
>>> member I'd be -1 on this proposal.
>>>
>>>> We think the right way to achieve this is to modify surefire plugin of
>>>> maven. We want to add it as a part of the configuration of surefire, so
>>>> users can decide whether they want to enable this feature, and how many
>>>> times they want to rerun failing tests. We plan to open-source our
>>>> contribution, and hopefully can merge our code into surefire master branch
>>>> in the end. Therefore, we are wondering do you have any
>>>> comments/suggestions/opinions regarding this? We appreciate any input.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Extend surefire to rerun failing tests
Posted by Kristian Rosenvold <kr...@gmail.com>.
Does this fit the bill for you ?
http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately
Kristian
2014-05-28 9:14 GMT+02:00 Alexander Kriegisch <Al...@kriegisch.name>:
>
>
> --
> Alexander Kriegisch
> http://scrum-master.de
>
> Am 28.05.2014 um 05:13 schrieb Benson Margulies <bi...@gmail.com>:
>
>> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <qi...@google.com>wrote:
>>
>>> I am an intern at Google. The first step of my internship project is to add
>>> the ability to Maven to automatically rerun failing tests a few times, to
>>> see if they ever pass in any of the reruns. It is useful because in many
>>> cases a test fails because it is flaky, not because there is a bug in the
>>> new source code change.
>>
>> I entirely disagree. 99.9999% of programs are completely deterministic. If
>> you feed them the same inputs, you get the same output.
>
> Well, in my previous project (I am an Agile Coach) my teams were working on a distributed, multi-threaded system and around 10% of all unit tests and next to all integration tests were affected by the exception you mentioned:
>
>> The exceptions are tests that use threads, and it's quite hard to do a good
>> job writing them.
>
> One can argue that several of the unit tests were not real unit tests because they mocked some, but not all of the infrastructure they needed for running. Anyway, they were running in Surefire (the integration tests in Failsafe), and some lf them were flaky.
>
>> Just 'rerunning a few times' does not lead to useful
>> information in my view.
>
> Only the information that indeed they are flaky and a rough percentage of cases in which they fail, which would be rather dependend on the test environment though. Better than nothing information-wise. So far, so good.
>
> Having said all that, I still agree to Benson that a failing test should just fail, period. I do not want developers to think, "it is only failing only in 5% of all runs, so it is fine / low priority to fix". OTOH I understand Qingzhou from a perspective of wishing to "test the test" or rather wishing to harden a flaky test into a stable one, either by fixing the test or the threading bug(s) it exposes. Benson's suggestion below looks promising though, so thanks for that.
>
>> if you want random testing, there's a fine alternative at
>> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine
>> maven plugin. It can already 'rerun your tests a few times' very easily.
>>
>> I wonder what the more involved surefire maintainers think, but as a PMC
>> member I'd be -1 on this proposal.
>>
>>> We think the right way to achieve this is to modify surefire plugin of
>>> maven. We want to add it as a part of the configuration of surefire, so
>>> users can decide whether they want to enable this feature, and how many
>>> times they want to rerun failing tests. We plan to open-source our
>>> contribution, and hopefully can merge our code into surefire master branch
>>> in the end. Therefore, we are wondering do you have any
>>> comments/suggestions/opinions regarding this? We appreciate any input.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Extend surefire to rerun failing tests
Posted by Alexander Kriegisch <Al...@Kriegisch.name>.
--
Alexander Kriegisch
http://scrum-master.de
Am 28.05.2014 um 05:13 schrieb Benson Margulies <bi...@gmail.com>:
> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <qi...@google.com>wrote:
>
>> I am an intern at Google. The first step of my internship project is to add
>> the ability to Maven to automatically rerun failing tests a few times, to
>> see if they ever pass in any of the reruns. It is useful because in many
>> cases a test fails because it is flaky, not because there is a bug in the
>> new source code change.
>
> I entirely disagree. 99.9999% of programs are completely deterministic. If
> you feed them the same inputs, you get the same output.
Well, in my previous project (I am an Agile Coach) my teams were working on a distributed, multi-threaded system and around 10% of all unit tests and next to all integration tests were affected by the exception you mentioned:
> The exceptions are tests that use threads, and it's quite hard to do a good
> job writing them.
One can argue that several of the unit tests were not real unit tests because they mocked some, but not all of the infrastructure they needed for running. Anyway, they were running in Surefire (the integration tests in Failsafe), and some lf them were flaky.
> Just 'rerunning a few times' does not lead to useful
> information in my view.
Only the information that indeed they are flaky and a rough percentage of cases in which they fail, which would be rather dependend on the test environment though. Better than nothing information-wise. So far, so good.
Having said all that, I still agree to Benson that a failing test should just fail, period. I do not want developers to think, "it is only failing only in 5% of all runs, so it is fine / low priority to fix". OTOH I understand Qingzhou from a perspective of wishing to "test the test" or rather wishing to harden a flaky test into a stable one, either by fixing the test or the threading bug(s) it exposes. Benson's suggestion below looks promising though, so thanks for that.
> if you want random testing, there's a fine alternative at
> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine
> maven plugin. It can already 'rerun your tests a few times' very easily.
>
> I wonder what the more involved surefire maintainers think, but as a PMC
> member I'd be -1 on this proposal.
>
>> We think the right way to achieve this is to modify surefire plugin of
>> maven. We want to add it as a part of the configuration of surefire, so
>> users can decide whether they want to enable this feature, and how many
>> times they want to rerun failing tests. We plan to open-source our
>> contribution, and hopefully can merge our code into surefire master branch
>> in the end. Therefore, we are wondering do you have any
>> comments/suggestions/opinions regarding this? We appreciate any input.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Extend surefire to rerun failing tests
Posted by Benson Margulies <bi...@gmail.com>.
On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <qi...@google.com>wrote:
> Hi,
>
> I am an intern at Google. The first step of my internship project is to add
> the ability to Maven to automatically rerun failing tests a few times, to
> see if they ever pass in any of the reruns. It is useful because in many
> cases a test fails because it is flaky, not because there is a bug in the
> new source code change.
>
I entirely disagree. 99.9999% of programs are completely deterministic. If
you feed them the same inputs, you get the same output.
The exceptions are tests that use threads, and it's quite hard to do a good
job writing them. Just 'rerunning a few times' does not lead to useful
information in my view.
if you want random testing, there's a fine alternative at
http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine
maven plugin. It can already 'rerun your tests a few times' very easily.
I wonder what the more involved surefire maintainers think, but as a PMC
member I'd be -1 on this proposal.
>
> We think the right way to achieve this is to modify surefire plugin of
> maven. We want to add it as a part of the configuration of surefire, so
> users can decide whether they want to enable this feature, and how many
> times they want to rerun failing tests. We plan to open-source our
> contribution, and hopefully can merge our code into surefire master branch
> in the end. Therefore, we are wondering do you have any
> comments/suggestions/opinions regarding this? We appreciate any input.
>
> Thank you very much!
>
> Regards,
>
> Qingzhou
>