You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/09/20 04:20:55 UTC
[GitHub] Harold-Zhang opened a new issue #7958: Engine shutdown
Harold-Zhang opened a new issue #7958: Engine shutdown
URL: https://github.com/apache/incubator-mxnet/issues/7958
## Environment info
Operating System:
Ubuntu 14.04
Compiler:
gcc 4.8.4
Package used (Python/R/Scala/Julia):
Python 2.7
MXNet version:
The latest version
GPU:
Tesla K40m
## Error Message:
[19:43:24] src/nnvm/legacy_json_util.cc:198: Symbol successfully upgraded!
2017-09-19 19:43:45,834 - Epoch[0] Batch [20] Speed: 4.56 samples/sec accuracy=0.964286
2017-09-19 19:44:03,379 - Epoch[0] Batch [40] Speed: 4.56 samples/sec accuracy=1.000000
2017-09-19 19:44:20,942 - Epoch[0] Batch [60] Speed: 4.56 samples/sec accuracy=1.000000
2017-09-19 19:44:38,747 - Epoch[0] Batch [80] Speed: 4.49 samples/sec accuracy=1.000000
2017-09-19 19:44:56,319 - Epoch[0] Batch [100] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:45:13,862 - Epoch[0] Batch [120] Speed: 4.56 samples/sec accuracy=1.000000
2017-09-19 19:45:31,494 - Epoch[0] Batch [140] Speed: 4.54 samples/sec accuracy=1.000000
2017-09-19 19:45:49,110 - Epoch[0] Batch [160] Speed: 4.54 samples/sec accuracy=1.000000
2017-09-19 19:46:06,677 - Epoch[0] Batch [180] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:46:24,257 - Epoch[0] Batch [200] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:46:41,886 - Epoch[0] Batch [220] Speed: 4.54 samples/sec accuracy=1.000000
2017-09-19 19:46:59,501 - Epoch[0] Batch [240] Speed: 4.54 samples/sec accuracy=1.000000
2017-09-19 19:47:17,085 - Epoch[0] Batch [260] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:47:34,667 - Epoch[0] Batch [280] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:47:52,273 - Epoch[0] Batch [300] Speed: 4.54 samples/sec accuracy=1.000000
2017-09-19 19:48:09,861 - Epoch[0] Batch [320] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:48:27,503 - Epoch[0] Batch [340] Speed: 4.53 samples/sec accuracy=1.000000
2017-09-19 19:48:45,085 - Epoch[0] Batch [360] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:49:02,700 - Epoch[0] Batch [380] Speed: 4.54 samples/sec accuracy=1.000000
2017-09-19 19:49:20,358 - Epoch[0] Batch [400] Speed: 4.53 samples/sec accuracy=1.000000
2017-09-19 19:49:37,943 - Epoch[0] Batch [420] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:49:55,530 - Epoch[0] Batch [440] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:50:13,105 - Epoch[0] Batch [460] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:50:30,683 - Epoch[0] Batch [480] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:50:48,265 - Epoch[0] Batch [500] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:51:05,903 - Epoch[0] Batch [520] Speed: 4.54 samples/sec accuracy=1.000000
2017-09-19 19:51:23,492 - Epoch[0] Batch [540] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:51:41,176 - Epoch[0] Batch [560] Speed: 4.52 samples/sec accuracy=1.000000
2017-09-19 19:51:58,766 - Epoch[0] Batch [580] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:52:16,347 - Epoch[0] Batch [600] Speed: 4.55 samples/sec accuracy=1.000000
2017-09-19 19:52:33,933 - Epoch[0] Batch [620] Speed: 4.55 samples/sec accuracy=1.000000
[19:52:38] /home/harold/mxnet/dmlc-core/include/dmlc/logging.h:308: [19:52:38] src/io/image_io.cc:165: Check failed: !dst.empty()
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f9450196c8c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2io12ImdecodeImplEibPvmPNS_7NDArrayE+0x67a) [0x7f9451a3eefa]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvN5mxnet10RunContextENS0_6engine18CallbackOnCompleteEEZNS0_6Engine8PushSyncESt8functionIFvS1_EENS0_7ContextERKSt6vectorIPNS2_3VarESaISC_EESG_NS0_10FnPropertyEiPKcEUlS1_S3_E_E9_M_invokeERKSt9_Any_dataS1_S3_+0x23) [0x7f9450284963]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine11NaiveEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKc+0x8b) [0x7f94519daf4b]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6Engine8PushSyncESt8functionIFvNS_10RunContextEEENS_7ContextERKSt6vectorIPNS_6engine3VarESaIS9_EESD_NS_10FnPropertyEiPKc+0x124) [0x7f9450285814]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2io8ImdecodeERKN4nnvm9NodeAttrsERKSt6vectorINS_7NDArrayESaIS6_EEPS8_+0xc90) [0x7f9451a40a10]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_Z20ImperativeInvokeImplRKN5mxnet7ContextEON4nnvm9NodeAttrsEPSt6vectorINS_7NDArrayESaIS7_EESA_PS6_IbSaIbEESD_+0x3cf) [0x7f94519a91ff]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_Z22MXImperativeInvokeImplPviPS_PiPS0_iPPKcS5_+0x25b) [0x7f94519bb43b]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(MXImperativeInvokeEx+0x2f) [0x7f94519a982f]
[bt] (9) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f9468629adc]
Traceback (most recent call last):
File "train.py", line 142, in <module>
image_shape='3,224,224', epoch=0, num_epoch=args.num_epoch, kv=kv)
File "train.py", line 106, in train_model
epoch_end_callback=checkpoint)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/module/base_module.py", line 491, in fit
next_data_batch = next(data_iter)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/image/image.py", line 1151, in next
data = self.imdecode(s)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/image/image.py", line 1183, in imdecode
return imdecode(s)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/image/image.py", line 136, in imdecode
return _internal._cvimdecode(buf, *args, **kwargs)
File "<string>", line 16, in _cvimdecode
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
ctypes.byref(out_stypes)))
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/base.py", line 143, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [19:52:38] src/io/image_io.cc:165: Check failed: !dst.empty()
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f9450196c8c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2io12ImdecodeImplEibPvmPNS_7NDArrayE+0x67a) [0x7f9451a3eefa]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvN5mxnet10RunContextENS0_6engine18CallbackOnCompleteEEZNS0_6Engine8PushSyncESt8functionIFvS1_EENS0_7ContextERKSt6vectorIPNS2_3VarESaISC_EESG_NS0_10FnPropertyEiPKcEUlS1_S3_E_E9_M_invokeERKSt9_Any_dataS1_S3_+0x23) [0x7f9450284963]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine11NaiveEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKc+0x8b) [0x7f94519daf4b]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6Engine8PushSyncESt8functionIFvNS_10RunContextEEENS_7ContextERKSt6vectorIPNS_6engine3VarESaIS9_EESD_NS_10FnPropertyEiPKc+0x124) [0x7f9450285814]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet2io8ImdecodeERKN4nnvm9NodeAttrsERKSt6vectorINS_7NDArrayESaIS6_EEPS8_+0xc90) [0x7f9451a40a10]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_Z20ImperativeInvokeImplRKN5mxnet7ContextEON4nnvm9NodeAttrsEPSt6vectorINS_7NDArrayESaIS7_EESA_PS6_IbSaIbEESD_+0x3cf) [0x7f94519a91ff]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_Z22MXImperativeInvokeImplPviPS_PiPS0_iPPKcS5_+0x25b) [0x7f94519bb43b]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(MXImperativeInvokeEx+0x2f) [0x7f94519a982f]
[bt] (9) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f9468629adc]
[19:52:38] src/engine/naive_engine.cc:53: Engine shutdown
## please provide the commands you have run that lead to the error.
I used the pretrained model from https://github.com/cypw/DPNs
commands:
python train.py --epoch 0 --model ./models/dpn92-extra --batch-size 4 --num-classes 2 --data-train ./lst_train.lst --image-train ./data/ --data-val ./lst_val.lst --image-val ./data/ --num-examples 2000 --lr 0.001 --gpus 0 --num-epoch 20 --save-result ./output
I have tried --batch-size 16/32, and I got the same result.
## What have you tried to solve it?
At first, I got a result: An fatal error occurred in asynchronous engine operation.
According to a guide, I set environment MXNET_CUDNN_AUTOTUNE_DEFAULT=0 and MXNET_ENGINE_TYPE=NaiveEngine, then I got the above result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services