You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/22 02:05:03 UTC
[GitHub] [incubator-mxnet] DickJC123 opened a new issue #19207: How to keep CI reliable- extra 'shake-out' of newly added tests?
DickJC123 opened a new issue #19207:
URL: https://github.com/apache/incubator-mxnet/issues/19207
## Description
This is a suggestion for a new CI feature. The problem addressed is known to most- when one submits a PR, one often has to rerun the CI many times to get past flaky unittests unrelated to the PR. But is a test that fails 0.1% of the time 'flaky'? Well, if our unittest suite accumulates 100 such 'semi-flaky' tests, and those tests are run in 10 jobs, there will be an average of 1 failure on every full CI run!
This suggestion of a remedy consists of two part:
- a script that would analyze the PR's diff, and deduce the set of newly introduced tests
- a script that given a set of tests and a value N, runs the tests N times [or perhaps runs the test repeatedly for N minutes]
With this proposed 'new test shake-out' facility integrated with the CI, PR contributors would have increased confidence that their PRs, once merged, would not make the CI unstable for the greater development community. I'm hopeful this won't discourage creating new tests- after all, reviewers generally demand test coverage of new features. However, it might discourage excessively long tests- if someone adds a test that takes 5 minutes, and N=10 reps, that would add 50 minutes to the CI runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #19207: [RFC] How to keep CI reliable- extra 'shake-out' of newly added tests?
Posted by GitBox <gi...@apache.org>.
szha commented on issue #19207:
URL: https://github.com/apache/incubator-mxnet/issues/19207#issuecomment-696943612
Thanks for the proposal on addressing the flaky test problem. I've been also thinking the same and here are a couple of suggestions that may help:
- if we introduce a new mxnet-bot command that can trigger requested tests using MXNET_TEST_COUNT=1000 that contributors/committers/reviewers can use, this can help reduce the cost of verifying flaky tests and thus prevent new flaky tests from being merged.
- we can adopt [hypothesis](https://hypothesis.readthedocs.io/en/latest/numpy.html) package for more clever and reproducible fuzzy testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #19207: [RFC] How to keep CI reliable- extra 'shake-out' of newly added tests?
Posted by GitBox <gi...@apache.org>.
szha commented on issue #19207:
URL: https://github.com/apache/incubator-mxnet/issues/19207#issuecomment-696943612
Thanks for the proposal on addressing the flaky test problem. I've been also thinking the same and here are a couple of suggestions that may help:
- if we introduce a new mxnet-bot command that can trigger requested tests using MXNET_TEST_COUNT=1000 that contributors/committers/reviewers can use, this can help reduce the cost of verifying flaky tests and thus prevent new flaky tests from being merged.
- we can adopt [hypothesis](https://hypothesis.readthedocs.io/en/latest/numpy.html) package for more clever and reproducible fuzzy testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org