You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Carl Tsai <da...@gmail.com> on 2018/07/31 00:42:44 UTC
Automated Flaky Test Detector
Hi all,
My name is Carl Tsai, I’m an intern working with the Amazon MXNet engine
team. For the past several weeks, I’ve been working with my mentor Hao Jin
on an automatic flaky test detection system. As you may know, there’s an
ongoing effort to reduce the number of flaky tests in the code base, and
this flaky test detector is meant to help ensure that these efforts will
have a long-lasting impact on the quality of our tests.
This is an automated tool that will be run on PR checks to detect flaky
tests before we merge them. It will check PRs for new and modified test
cases and run them through a flakiness checker so that we can have high
confidence that our tests are not flaky. There’s a design overview on
confluence
<https://cwiki.apache.org/confluence/display/MXNET/Automated+Flaky+Test+Detector>—
I’d be happy to hear any feedback!
In the meantime, one of the components, the flakiness checker, has been
merged under the tools/ folder and I’d encourage those of you who are
working on flaky tests or writing new tests to try it out. Documentation
can be found on the confluence page for the project as well as on the
Reproducing
Test Results Page
<https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results>
under tips and tricks, and I’m open to suggestions on how to improve it.
Thanks,
Carl
Re: Automated Flaky Test Detector
Posted by Pedro Larroy <pe...@gmail.com>.
Hi Carl
This is great, I like a lot the design for the tool. I think it will
prevent addition of flaky tests once is implemented.
I don't have major concerns with the design, please keep us updated with
PRs about this tool.
Thanks!
Pedro.
On Tue, Jul 31, 2018 at 3:01 AM Carl Tsai <da...@gmail.com> wrote:
> Minor correction:
>
> I'm working with the *Apache* MXNet engine team at Amazon!
>
> Carl
>
> On Mon, Jul 30, 2018 at 5:42 PM Carl Tsai <da...@gmail.com> wrote:
>
> > Hi all,
> >
> >
> >
> > My name is Carl Tsai, I’m an intern working with the Amazon MXNet engine
> > team. For the past several weeks, I’ve been working with my mentor Hao
> Jin
> > on an automatic flaky test detection system. As you may know, there’s an
> > ongoing effort to reduce the number of flaky tests in the code base, and
> > this flaky test detector is meant to help ensure that these efforts will
> > have a long-lasting impact on the quality of our tests.
> >
> >
> >
> > This is an automated tool that will be run on PR checks to detect flaky
> > tests before we merge them. It will check PRs for new and modified test
> > cases and run them through a flakiness checker so that we can have high
> > confidence that our tests are not flaky. There’s a design overview on
> > confluence
> > <
> https://cwiki.apache.org/confluence/display/MXNET/Automated+Flaky+Test+Detector
> >—
> > I’d be happy to hear any feedback!
> >
> >
> >
> > In the meantime, one of the components, the flakiness checker, has been
> > merged under the tools/ folder and I’d encourage those of you who are
> > working on flaky tests or writing new tests to try it out. Documentation
> > can be found on the confluence page for the project as well as on the
> Reproducing
> > Test Results Page
> > <
> https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results
> >
> > under tips and tricks, and I’m open to suggestions on how to improve it.
> >
> >
> >
> > Thanks,
> >
> > Carl
> >
>
Re: Automated Flaky Test Detector
Posted by Carl Tsai <da...@gmail.com>.
Minor correction:
I'm working with the *Apache* MXNet engine team at Amazon!
Carl
On Mon, Jul 30, 2018 at 5:42 PM Carl Tsai <da...@gmail.com> wrote:
> Hi all,
>
>
>
> My name is Carl Tsai, I’m an intern working with the Amazon MXNet engine
> team. For the past several weeks, I’ve been working with my mentor Hao Jin
> on an automatic flaky test detection system. As you may know, there’s an
> ongoing effort to reduce the number of flaky tests in the code base, and
> this flaky test detector is meant to help ensure that these efforts will
> have a long-lasting impact on the quality of our tests.
>
>
>
> This is an automated tool that will be run on PR checks to detect flaky
> tests before we merge them. It will check PRs for new and modified test
> cases and run them through a flakiness checker so that we can have high
> confidence that our tests are not flaky. There’s a design overview on
> confluence
> <https://cwiki.apache.org/confluence/display/MXNET/Automated+Flaky+Test+Detector>—
> I’d be happy to hear any feedback!
>
>
>
> In the meantime, one of the components, the flakiness checker, has been
> merged under the tools/ folder and I’d encourage those of you who are
> working on flaky tests or writing new tests to try it out. Documentation
> can be found on the confluence page for the project as well as on the Reproducing
> Test Results Page
> <https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results>
> under tips and tricks, and I’m open to suggestions on how to improve it.
>
>
>
> Thanks,
>
> Carl
>