You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/04/28 05:59:01 UTC
[GitHub] [incubator-mxnet] szha opened a new issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
szha opened a new issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184
## Description
slogdet seems to produce wrong results on windows in a flaky way even with `rtol=atol=1e-1`.
## Occurrences
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/windows-gpu/branches/PR-18146/runs/21/nodes/109/steps/155/log/?start=0
## What have you tried to solve it?
1. added retry
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-625624201
@vexilligera just making sure that you're running it on windows. I observed that the error only happens on windows.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] vexilligera edited a comment on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
vexilligera edited a comment on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-625047127
Hi, I tried to reproduce the error by running
`pytest tests/python/gpu/test_operator_gpu.py::test_np_linalg_slogdet`
for 20 times and didn't run into the error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-621501407
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/windows-gpu/branches/PR-18146/runs/30/nodes/109/steps/155/log/?start=0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-622227153
Some more failures
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/windows-gpu/branches/PR-18202/runs/2/nodes/109/steps/155/log/?start=0
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/windows-gpu/branches/PR-18202/runs/3/nodes/109/steps/155/log/?start=0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-625081994
Did you try on G3 instances? The other question is why the `rtol=atol=1e-1` and not lower
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-625917939
Thanks. Sounds like it's safe to re-enable the test then. Let's do that and run the CI for three times to make sure it's no longer flaky.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] vexilligera commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
vexilligera commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-625612824
> Did you try on G3 instances? The other question is why the `rtol=atol=1e-1` and not lower
I later reran the test on a G3 instance for 100 times with tolerance at 1e-2, still didn't get the error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] vexilligera commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
vexilligera commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-625047127
Hi, I tried to reproduce the error by running
`pytest tests/python/gpu/test_operator_gpu.py::test_np_linalg_slogdet`
for 20 times both on CPU and GPU, and didn't run into the error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] vexilligera commented on issue #18184: Flaky Incorrect Results test_numpy_op.py:: test_np_linalg_slogdet
Posted by GitBox <gi...@apache.org>.
vexilligera commented on issue #18184:
URL: https://github.com/apache/incubator-mxnet/issues/18184#issuecomment-625638615
> @vexilligera just making sure that you're running it on windows. I observed that the error only happens on windows.
Of course, on G3 with Windows Server 2016.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org