You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/01/21 17:52:46 UTC
[GitHub] [incubator-mxnet] access2rohit opened a new issue #17397:
log_softmax for large tensor is flaky
access2rohit opened a new issue #17397: log_softmax for large tensor is flaky
URL: https://github.com/apache/incubator-mxnet/issues/17397
## Description
log_softmax fails randomly when using very large tensors.
## Occurrences
*** Maximum errors for vector of size 5000000000: rtol=0.0001, atol=1e-20
1: Error 228.818179 Location of error: (1, 69287122), EXPECTED_log_softmax0_output=-17.01618748, FORWARD_log_softmax0_output=-16.63553619
2: Error 228.818174 Location of error: (1, 66902942), EXPECTED_log_softmax0_output=-17.01618358, FORWARD_log_softmax0_output=-16.63553238
3: Error 228.818163 Location of error: (1, 42527178), EXPECTED_log_softmax0_output=-17.01618356, FORWARD_log_softmax0_output=-16.63553238
4: Error 228.818160 Location of error: (1, 36123476), EXPECTED_log_softmax0_output=-17.01618356, FORWARD_log_softmax0_output=-16.63553238
5: Error 228.818143 Location of error: (1, 52855011), EXPECTED_log_softmax0_output=-17.01618353, FORWARD_log_softmax0_output=-16.63553238
6: Error 228.818141 Location of error: (1, 16069295), EXPECTED_log_softmax0_output=-17.01618353, FORWARD_log_softmax0_output=-16.63553238
7: Error 228.818126 Location of error: (1, 3578168), EXPECTED_log_softmax0_output=-17.01618546, FORWARD_log_softmax0_output=-16.63553429
8: Error 228.818122 Location of error: (1, 37536126), EXPECTED_log_softmax0_output=-17.01618349, FORWARD_log_softmax0_output=-16.63553238
9: Error 228.818113 Location of error: (1, 61821668), EXPECTED_log_softmax0_output=-17.01618737, FORWARD_log_softmax0_output=-16.63553619
10: Error 228.818111 Location of error: (1, 42215697), EXPECTED_log_softmax0_output=-17.01618737, FORWARD_log_softmax0_output=-16.63553619
FAIL
======================================================================
FAIL: test_large_array.test_nn
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/ubuntu/incubator-mxnet/tests/nightly/test_large_array.py", line 447, in test_nn
check_log_softmax()
File "/home/ubuntu/incubator-mxnet/tests/nightly/test_large_array.py", line 138, in check_log_softmax
check_symbolic_forward(sym, [data], [np.log(np_softmax(data, axis=axis)+1e-20)])
File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1202, in check_symbolic_forward
equal_nan=equal_nan)
File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
Error 228.818179 exceeds tolerance rtol=1.000000e-04, atol=1.000000e-20 (mismatch at least 0.000000%).
Location of maximum error: (1, 69287122), EXPECTED_log_softmax0_output=-17.01618748, FORWARD_log_softmax0_output=-16.63553619
ACTUAL: array([[-17.9350638 , -17.96627981, -19.72406303, ..., -19.82472448,
-17.56042908, -18.59646628],
[-20.59984509, -18.51675027, -19.10637481, ..., -19.50312141,...
DESIRED: array([[-17.554623, -17.585838, -19.343622, ..., -19.444284, -17.179989,
-18.216026],
[-20.219194, -18.1361 , -18.725725, ..., -19.12247 , -19.1559 ,...
----------------------------------------------------------------------
Ran 1 test in 3267.809s
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] access2rohit commented on issue #17397:
log_softmax for large tensor is flaky
Posted by GitBox <gi...@apache.org>.
access2rohit commented on issue #17397: log_softmax for large tensor is flaky
URL: https://github.com/apache/incubator-mxnet/issues/17397#issuecomment-576801129
@mxnet-label-bot add [large-tensor]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services