You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/08/03 01:48:47 UTC
[GitHub] zachgk commented on issue #11695: test_loss.test_hinge_loss has
fixed seed that can mask flakiness
zachgk commented on issue #11695: test_loss.test_hinge_loss has fixed seed that can mask flakiness
URL: https://github.com/apache/incubator-mxnet/issues/11695#issuecomment-410119461
I ran the test numerous times. It only failed at one instance with the original tolerance, however I did not record the seed that caused the failure. After ~40k additional iterations, I have been unable to reproduce another failure with the original tolerance. However, the increased tolerance would have accepted that failure, so I updated it.
I then ran `tests/python/unittest/test_loss.py:test_hinge_loss` and `tests/python/gpu/test_operator_gpu.py:test_hinge_loss` 10k times each without producing the error at an increased tolerance
I raised a PR #12020 to increase the tolerance and remove the fixed seed
CPU Run with increased tolerance:
```
ubuntu@ip-172-31-27-232:~/incubator-mxnet$ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s tests/python/unittest/test_loss.py:test_hinge_loss
Produces:
[DEBUG] 9999 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2140149943 to reproduce.
0.0
[DEBUG] 10000 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=552569265 to reproduce.
0.0
ok
----------------------------------------------------------------------
Ran 1 test in 7793.215s
OK
```
GPU Run with increased tolerance:
```
ubuntu@ip-172-31-27-232:~/incubator-mxnet$ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s tests/python/gpu/test_operator_gpu.py:test_hinge_loss
Produces:
[DEBUG] 9999 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=175424733 to reproduce.
0.0
[DEBUG] 10000 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=745050216 to reproduce.
0.0
ok
----------------------------------------------------------------------
Ran 1 test in 8036.187s
OK
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services