You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/01/17 13:31:17 UTC
[GitHub] [incubator-mxnet] TaoLv opened a new issue #17362: [flaky]
test_higher_order_grad.test_arcsin
TaoLv opened a new issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362
## Description
test_higher_order_grad.test_arcsin
## Occurrences
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-17356/1/pipeline
```
======================================================================
FAIL: test_higher_order_grad.test_arcsin
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/work/mxnet/tests/python/unittest/common.py", line 221, in test_new
orig_test(*args, **kwargs)
File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 153, in test_arcsin
check_second_order_unary(array, arcsin, grad_grad_op)
File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 507, in check_second_order_unary
check_nth_order_unary(x, op, grad_grad_op, 2, rtol, atol)
File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 577, in check_nth_order_unary
expected_grad, computed_grad.asnumpy(), rtol=rtol, atol=atol)
File "/work/mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
Error nan exceeds tolerance rtol=1.000000e-05, atol=1.000000e-20 (mismatch 0.154321%).
Location of maximum error: (1, 6, 2, 1), a=0.00000000, b=nan
ACTUAL: array([[[[-0.03453423, 0.1168007 , -0.00583075],
[ 0.02170432, 0.37017354, 0.09864384],
[-0.02544087, -0.06709251, -0.34824234]],...
DESIRED: array([[[[-0.03453423, 0.1168007 , -0.00583075],
[ 0.02170432, 0.37017348, 0.09864385],
[-0.02544087, -0.0670925 , -0.34824237]],...
-------------------- >> begin captured stdout << ---------------------
*** Maximum errors for vector of size 648: rtol=1e-05, atol=1e-20
1: Error nan Location of error: (1, 6, 2, 1), a=0.00000000, b=nan
--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=402611671 to reproduce.
--------------------- >> end captured logging << ---------------------
```
## What have you tried to solve it?
None
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
apeforest commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-575752318
@kshitij12345 Could you please take a look? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] kshitij12345 edited a comment on issue #17362:
[flaky] test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
kshitij12345 edited a comment on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-601200135
@zixuanweeei Good to know that the issue is solved.
@TaoLv @haojin2 I guess we can close the issue as it is taken care of by https://github.com/apache/incubator-mxnet/pull/17325
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] TaoLv commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
TaoLv commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-575626698
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-17355/1/pipeline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] zixuanweeei commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
zixuanweeei commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-600946386
Hi @kshitij12345 , thanks for your quick response. Finally, I found that I need to `make clean` and manually clean the compiling assets, and then got it passed. BTW, the error in the backtrace is mainly about `std::basic_string`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] kshitij12345 commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
kshitij12345 commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-575978053
#17325
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] TaoLv commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
TaoLv commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-601216473
Thanks @kshitij12345 . Sure I'm closing this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] kshitij12345 commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
kshitij12345 commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-600767627
Hi,
I built on the mentioned commit with
```
cmake -DDEBUG=1 -DENABLE_TESTCOVERAGE=0 -DUSE_CPP_PACKAGE=0 -DUSE_MKLDNN=1 -DUSE_BLAS=mkl -DUSE_SIGNAL_HANDLER=1 ..
```
I couldn't get the `-DUSE_BLAS=blas` to work as I was getting `Could not find #include <mkl_blas.h>`.
However I couldn't reproduce the issue.
```
MXNET_SUBGRAPH_VERBOSE=0 nosetests -s -v --logging-level=DEBUG tests/python/unittest/test_higher_order_grad.py:test_arctanh
test_higher_order_grad.test_arctanh ... [DEBUG] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=343422092 to reproduce.
ok
----------------------------------------------------------------------
Ran 1 test in 0.073s
OK
```
@zixuanweeei Can you try to get the backtrace?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] kshitij12345 commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
kshitij12345 commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-578377185
@haojin2
#17325 takes care of this one as well as others.
However I am facing issues with the CI. It has timed out multiple times.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] haojin2 commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
haojin2 commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-578314928
Another example of failure to assist the debug:
```
======================================================================
FAIL: test_higher_order_grad.test_arcsin
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/home/ubuntu/5-mxnet/tests/python/unittest/common.py", line 215, in test_new
orig_test(*args, **kwargs)
File "/home/ubuntu/5-mxnet/tests/python/unittest/test_higher_order_grad.py", line 153, in test_arcsin
check_second_order_unary(array, arcsin, grad_grad_op)
File "/home/ubuntu/5-mxnet/tests/python/unittest/test_higher_order_grad.py", line 507, in check_second_order_unary
check_nth_order_unary(x, op, grad_grad_op, 2, rtol, atol)
File "/home/ubuntu/5-mxnet/tests/python/unittest/test_higher_order_grad.py", line 577, in check_nth_order_unary
expected_grad, computed_grad.asnumpy(), rtol=rtol, atol=atol, equal_nan=True)
File "/home/ubuntu/5-mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
Error nan exceeds tolerance rtol=1.000000e-05, atol=1.000000e-20 (mismatch 0.173611%).
Location of maximum error: (2, 0, 2, 2), a=-0.00000000, b=nan
ACTUAL: array([[[[ 0.04022623, 0.06529827, 0.04777484, 0.0758371 ,
0.06068928, -0.18494761],
[-0.01754379, 0.06515067, 0.00206734, -0.01263793,...
DESIRED: array([[[[ 0.04022623, 0.06529827, 0.04777484, 0.07583709,
0.06068929, -0.18494764],
[-0.01754379, 0.06515064, 0.00206734, -0.01263793,...
-------------------- >> begin captured stdout << ---------------------
*** Maximum errors for vector of size 576: rtol=1e-05, atol=1e-20
1: Error nan Location of error: (2, 0, 2, 2), a=-0.00000000, b=nan
--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: 6939 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1754124593 to reproduce.
--------------------- >> end captured logging << ---------------------
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] TaoLv closed issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
TaoLv closed issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] kshitij12345 commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
kshitij12345 commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-601200135
@zixuanweeei Good to know that the issue is solved.
@TaoLv @haojin2 I guess we can close the issue as it is taken care of.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] zixuanweeei commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
zixuanweeei commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-600378095
Took a test for 1efb747ad5cf695f181c64bfb16daccd6ca4c456, and built it from source using the following command:
```
make -j DEBUG=1 ENABLE_TESTCOVERAGE=0 USE_CPP_PACKAGE=0 USE_MKLDNN=1 USE_BLAS=blas USE_SIGNAL_HANDLER=1
```
Several tests like `test_arcsin`, `test_arctan`, `test_arcsinh`, `test_arccosh`, `test_arctanh`, etc report *Segmentation Fault: 11*.
Example:
```
$ MXNET_SUBGRAPH_VERBOSE=0 nosetests -s -v --logging-level=DEBUG tests/python/unittest/test_higher_order_grad.py:test_arcsin
test_higher_order_grad.test_arcsin ... [DEBUG] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1219729000 to reproduce.
Segmentation fault: 11
```
@haojin2 @kshitij12345
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] kshitij12345 commented on issue #17362: [flaky]
test_higher_order_grad.test_arcsin
Posted by GitBox <gi...@apache.org>.
kshitij12345 commented on issue #17362: [flaky] test_higher_order_grad.test_arcsin
URL: https://github.com/apache/incubator-mxnet/issues/17362#issuecomment-600676474
Will have a look over the weekend.
Thanks for informing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services