You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2022/03/23 21:11:53 UTC
[GitHub] [incubator-mxnet] barry-jin opened a new issue #20978: test_conv2d_16c[224-256] fails for cu101, cu110, cu112
barry-jin opened a new issue #20978:
URL: https://github.com/apache/incubator-mxnet/issues/20978
## Description
tests/python/gpu/test_gluon_gpu.py::test_conv2d_16c[224-256] will cause worker thread to crash on cu101, cu110 and cu112 but will pass on cu102
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/restricted-mxnet-cd%2Fmxnet-cd-release-job/detail/mxnet-cd-release-job/4064/pipeline/340
error message:
```
[2022-03-20T11:07:01.372Z] Thread 0x00007fed6143a700 (most recent call first):
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 400 in read
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 432 in from_io
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 220 in run
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
[2022-03-20T11:07:01.372Z]
[2022-03-20T11:07:01.372Z] Thread 0x00007fed64bca740 (most recent call first):
[2022-03-20T11:07:01.372Z] File "/work/mxnet/python/mxnet/ndarray/ndarray.py", line 2640 in asnumpy
[2022-03-20T11:07:01.372Z] File "/work/mxnet/tests/python/unittest/test_gluon.py", line 1808 in check_layer_forward_withinput
[2022-03-20T11:07:01.372Z] File "/work/mxnet/tests/python/unittest/test_gluon.py", line 1830 in test_conv2d_16c
[2022-03-20T11:07:01.372Z] File "/work/mxnet/python/mxnet/util.py", line 486 in _with_np_array
[2022-03-20T11:07:01.372Z] File "/work/mxnet/python/mxnet/util.py", line 304 in _with_np_shape
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/python.py", line 184 in pytest_pyfunc_call
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/python.py", line 1627 in runtest
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 163 in pytest_runtest_call
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 256 in <lambda>
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 310 in from_call
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py", line 138 in call_and_report
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 127 in runtestprotocol
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 110 in pytest_runtest_protocol
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py", line 94 in pytest_runtest_protocol
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/xdist/remote.py", line 87 in run_one_test
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/xdist/remote.py", line 70 in pytest_runtestloop
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 313 in _main
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 257 in wrap_session
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/_pytest/main.py", line 306 in pytest_cmdline_main
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/xdist/remote.py", line 237 in <module>
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 1084 in executetask
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 220 in run
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 1060 in serve
[2022-03-20T11:07:01.372Z] File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/execnet/gateway_base.py", line 1554 in serve
[2022-03-20T11:07:01.372Z] File "<string>", line 8 in <module>
[2022-03-20T11:07:01.372Z] File "<string>", line 1 in <module>
[2022-03-20T11:07:01.372Z]
[2022-03-20T11:07:01.372Z] [gw0] [ 31%] PASSED tests/python/gpu/test_gluon_gpu.py::test_export
[2022-03-20T11:07:01.372Z] tests/python/gpu/test_gluon_gpu.py::test_import
[2022-03-20T11:07:01.372Z] [gw2] node down: Not properly terminated
[2022-03-20T11:07:01.372Z] [gw2] [ 31%] FAILED tests/python/gpu/test_gluon_gpu.py::test_conv2d_16c[224-256]
[2022-03-20T11:07:01.372Z]
[2022-03-20T11:07:01.372Z] replacing crashed worker gw2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org