You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/08/24 08:10:03 UTC
[GitHub] Ume07 opened a new issue #7590: Gradient function not returning enough gradient
Ume07 opened a new issue #7590: Gradient function not returning enough gradient
URL: https://github.com/apache/incubator-mxnet/issues/7590
For bugs or installation issues, please provide the following information.
The more information you provide, the more likely people will be able to help you.
## Environment info
Operating System: Ubuntu 16.04
Package used (Python/R/Scala/Julia): python
MXNet version: 0.10.1/ 0.11.0
MXNet commit hash (`git rev-parse HEAD`):
If you are using python package, please provide
Python version and distribution: Python 2.7.12
## Error Message:
batchsize is 100
[16:07:10] src/io/iter_mnist.cc:112: MNISTIter: load 60000 images, shuffle=1, shape=(100,1,28,28)
[16:07:10] src/io/iter_mnist.cc:112: MNISTIter: load 10000 images, shuffle=1, shape=(100,1,28,28)
training model ...
dev is [gpu(0)]
/home/emptytaboo/mxnet_center_loss/train_model.py:133: DeprecationWarning: mxnet.model.FeedForward has been deprecated. Please use mxnet.mod.Module instead.
**model_args)
/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/initializer.py:370: DeprecationWarning: Calling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.
init(name, arr)
[16:07:11] /home/mxnet/dmlc-core/include/dmlc/./logging.h:308: [16:07:11] src/pass/gradient.cc:159: Check failed: (*rit)->inputs.size() == input_grads.size() (5 vs. 2) Gradient function not returning enough gradient
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f34139007cc]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(+0x2a996e0) [0x7f3415c466e0]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFN4nnvm5GraphES1_EPS2_E9_M_invokeERKSt9_Any_dataOS1_+0x111) [0x7f34146a0be1]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm11ApplyPassesENS_5GraphERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x32c) [0x7f3415c766dc]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm9ApplyPassENS_5GraphERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x3c9) [0x7f3414a81fe9]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm4pass8GradientENS_5GraphESt6vectorINS_9NodeEntryESaIS3_EES5_S5_St8functionIFS3_OS5_EES6_IFiRKNS_4NodeEEES6_IFS3_RKS3_SG_EES2_IPKNS_2OpESaISL_EENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x6fc) [0x7f3414b0bbbc]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor13InitFullGraphEN4nnvm6SymbolERKSt6vectorINS_9OpReqTypeESaIS5_EE+0x863) [0x7f3414af3a43]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor9InitGraphEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorIS4_SaIS4_EESR_SR_RKSN_INS_9OpReqTypeESaISS_EE+0x82) [0x7f3414af46d2]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorINS_7NDArrayESaISO_EESS_RKSN_INS_9OpReqTypeESaIST_EESS_PNS_8ExecutorERKSt13unordered_mapINS2_9NodeEntryESO_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISG_IKS11_SO_EEE+0x8b7) [0x7f3414b00997]
[bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet8Executor4BindEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_St4lessISC_ESaISt4pairIKSC_S3_EEERKSt6vectorINS_7NDArrayESaISN_EESR_RKSM_INS_9OpReqTypeESaISS_EESR_PS0_+0xe0) [0x7f3414b01d70]
Traceback (most recent call last):
File "train.py", line 98, in <module>
main()
File "train.py", line 95, in main
train_model.fit(args, net, (train, val), data_shape)
File "/home/emptytaboo/mxnet_center_loss/train_model.py", line 153, in fit
epoch_end_callback = checkpoint)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/model.py", line 847, in fit
sym_gen=self.sym_gen)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/model.py", line 227, in _train_multi_device
logger=logger)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/executor_manager.py", line 343, in __init__
self.slices, train_data)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/executor_manager.py", line 255, in __init__
input_types=data_types)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/executor_manager.py", line 201, in _bind_exec
grad_req=grad_req, shared_exec=base_exec)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/symbol/symbol.py", line 1661, in bind
ctypes.byref(handle)))
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/base.py", line 143, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [16:07:11] src/pass/gradient.cc:159: Check failed: (*rit)->inputs.size() == input_grads.size() (5 vs. 2) Gradient function not returning enough gradient
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f34139007cc]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(+0x2a996e0) [0x7f3415c466e0]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFN4nnvm5GraphES1_EPS2_E9_M_invokeERKSt9_Any_dataOS1_+0x111) [0x7f34146a0be1]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm11ApplyPassesENS_5GraphERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x32c) [0x7f3415c766dc]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm9ApplyPassENS_5GraphERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x3c9) [0x7f3414a81fe9]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN4nnvm4pass8GradientENS_5GraphESt6vectorINS_9NodeEntryESaIS3_EES5_S5_St8functionIFS3_OS5_EES6_IFiRKNS_4NodeEEES6_IFS3_RKS3_SG_EES2_IPKNS_2OpESaISL_EENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x6fc) [0x7f3414b0bbbc]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor13InitFullGraphEN4nnvm6SymbolERKSt6vectorINS_9OpReqTypeESaIS5_EE+0x863) [0x7f3414af3a43]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor9InitGraphEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorIS4_SaIS4_EESR_SR_RKSN_INS_9OpReqTypeESaISS_EE+0x82) [0x7f3414af46d2]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorINS_7NDArrayESaISO_EESS_RKSN_INS_9OpReqTypeESaIST_EESS_PNS_8ExecutorERKSt13unordered_mapINS2_9NodeEntryESO_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISG_IKS11_SO_EEE+0x8b7) [0x7f3414b00997]
[bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet-0.11.1-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet8Executor4BindEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_St4lessISC_ESaISt4pairIKSC_S3_EEERKSt6vectorINS_7NDArrayESaISN_EESR_RKSM_INS_9OpReqTypeESaISS_EESR_PS0_+0xe0) [0x7f3414b01d70]
## Minimum reproducible example
https://github.com/pangyupo/mxnet_center_loss/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services