You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/01/11 22:07:05 UTC
[GitHub] eric-haibin-lin opened a new issue #9382: Failed kvstore nightly test
eric-haibin-lin opened a new issue #9382: Failed kvstore nightly test
URL: https://github.com/apache/incubator-mxnet/issues/9382
http://jenkins-master-elb-1979848568.us-east-1.elb.amazonaws.com/job/KVStore_SingleNode_Tests/77/console
The nightly test "test_kvstore.py" has been failing for the past week.
```
local_update_cpu
local_allreduce_cpu
local_allreduce_device
[04:20:59] src/kvstore/././comm.h:655: only 0 out of 12 GPU pairs are enabled direct access. It may affect the performance. You can set MXNET_ENABLE_GPU_P2P=0 to turn it off
[04:20:59] src/kvstore/././comm.h:664: ....
[04:20:59] src/kvstore/././comm.h:664: ....
[04:20:59] src/kvstore/././comm.h:664: ....
[04:20:59] src/kvstore/././comm.h:664: ....
local_allreduce_device with 2bit compression
[04:21:03] src/kvstore/././comm.h:655: only 0 out of 12 GPU pairs are enabled direct access. It may affect the performance. You can set MXNET_ENABLE_GPU_P2P=0 to turn it off
[04:21:03] src/kvstore/././comm.h:664: ....
[04:21:03] src/kvstore/././comm.h:664: ....
[04:21:03] src/kvstore/././comm.h:664: ....
[04:21:03] src/kvstore/././comm.h:664: ....
terminate called after throwing an instance of 'dmlc::Error'
what(): [04:21:06] src/engine/./threaded_engine.h:359: [04:21:06] /home/ec2-user/workspace/KVStore_SingleNode_Tests/mshadow/mshadow/./stream_gpu-inl.h:62: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered
Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc10StackTraceEv+0x42) [0x7f241ee0d792]
[bt] (1) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x18) [0x7f241ee0dd88]
[bt] (2) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN7mshadow6StreamINS_3gpuEE4WaitEv+0xd8) [0x7f2420feb138]
[bt] (3) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZZN5mxnet10imperative12PushFComputeERKSt8functionIFvRKN4nnvm9NodeAttrsERKNS_9OpContextERKSt6vectorINS_5TBlobESaISA_EERKS9_INS_9OpReqTypeESaISF_EESE_EEPKNS2_2OpES5_RKNS_7ContextERKS9_IPNS_6engine3VarESaISW_EES10_RKS9_INS_8ResourceESaIS11_EERKS9_IPNS_7NDArrayESaIS17_EES1B_RKS9_IjSaIjEESJ_ENKUlNS_10RunContextEE_clES1G_+0x78a) [0x7f242103caba]
[bt] (4) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(+0x2bb9ca3) [0x7f2420faeca3]
[bt] (5) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x9d) [0x7f2420fb71fd]
[bt] (6) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine23ThreadedEnginePerDevice9GPUWorkerILN4dmlc19ConcurrentQueueTypeE0EEEvNS_7ContextEbPNS1_17ThreadWorkerBlockIXT_EEESt10shared_ptrINS0_10ThreadPool11SimpleEventEE+0x10b) [0x7f2420fbbdab]
[bt] (7) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_+0x56) [0x7f2420fbbf86]
[bt] (8) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x3b) [0x7f2420fb8acb]
[bt] (9) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f240a047a60]
A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 9 entries:
[bt] (0) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc10StackTraceEv+0x42) [0x7f241ee0d792]
[bt] (1) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x18) [0x7f241ee0dd88]
[bt] (2) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x3a0) [0x7f2420fb7500]
[bt] (3) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine23ThreadedEnginePerDevice9GPUWorkerILN4dmlc19ConcurrentQueueTypeE0EEEvNS_7ContextEbPNS1_17ThreadWorkerBlockIXT_EEESt10shared_ptrINS0_10ThreadPool11SimpleEventEE+0x10b) [0x7f2420fbbdab]
[bt] (4) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_+0x56) [0x7f2420fbbf86]
[bt] (5) /home/ec2-user/workspace/KVStore_SingleNode_Tests/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x3b) [0x7f2420fb8acb]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f240a047a60]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f242d061184]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f242cd8e37d]
bash: line 1: 11585 Aborted python test_kvstore.py
Build step 'Execute shell' marked build as failure
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
Finished: FAILURE
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services