You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/10/25 19:09:57 UTC
[GitHub] [incubator-mxnet] leezu opened a new issue #16628: RNNOp regression
on CPU when using a GPU-enabled build
leezu opened a new issue #16628: RNNOp regression on CPU when using a GPU-enabled build
URL: https://github.com/apache/incubator-mxnet/issues/16628
## Description
Using a master build (27f708222828f1f8f62c24a33764ce75c3a7588e), I observe the following error when invoking `RNNOp` on a CPU machine. The line causing the error was added by @DickJC123 in #16391.
@DickJC123 should all `CUDA_CALL`s and `CUDNN_CALL`s be put in an extra `if ctx_.dev_-ype == kGPU)` check? It seems they are currently run even if the `RNNOp` is executed on CPU.
### Error Message
```
E mxnet.base.MXNetError: [18:23:30] /opt/brazil-pkg-cache/packages/DeepMXNet/DeepMXNet-master.322.0/AL2012/generic-flavor/src/src/operator/./rnn-inl.h:496: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: CUDA driver version is insufficient for CUDA runtime version
E Stack trace:
[bt] E (0) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x24) [0x7fb186c7e144]
[bt] E (1) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::op::RNNOp<mshadow::cpu, float, float>::RNNOp(mxnet::op::RNNParam, mxnet::Context)+0xff6) [0x7fb18a684e26]
[bt] E (2) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::OpStatePtr mxnet::OpStatePtr::Create<mxnet::op::RNNOp<mshadow::cpu, float, float>, mxnet::op::RNNParam const&, mxnet::Context const&>(mxnet::op::RNNParam const&, mxnet::Context const&)+0xd3) [0x7fb18a6858b3]
[bt] E (3) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(+0x4f2f786) [0x7fb18a58f786]
[bt] E (4) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(std::_Function_handler<mxnet::OpStatePtr (nnvm::NodeAttrs const&, mxnet::Context, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, std::vector<int, std::allocator<int> > const&), mxnet::OpStatePtr (*)(nnvm::NodeAttrs const&, mxnet::Context, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, std::vector<int, std::allocator<int> > const&)>::_M_invoke(std::_Any_data const&, nnvm::NodeAttrs const&, mxnet::Context, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, std::vector<int, std::allocator<int> > const&)+0x15) [0x7fb189352e65]
[bt] E (5) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(+0x3e17f7d) [0x7fb189477f7d]
[bt] E (6) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::imperative::RunGraph(bool, nnvm::IndexedGraph const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, unsigned long, unsigned long, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> >&&, std::vector<unsigned int, std::allocator<unsigned int> >&&, std::vector<mxnet::OpStatePtr, std::allocator<mxnet::OpStatePtr> >*, std::vector<mxnet::DispatchMode, std::allocator<mxnet::DispatchMode> > const&, bool, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::function<void (char const*, char const*, void*)> const&, bool)+0x1fd) [0x7fb18947846d]
[bt] E (7) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::CachedOp::DynamicForward(mxnet::Context const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, bool)+0xc55) [0x7fb18943ed95]
[bt] E (8) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::CachedOp::Forward(std::shared_ptr<mxnet::CachedOp> const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0xcee) [0x7fb18944723e]
/opt/brazil-pkg-cache/packages/DeepMXNet/DeepMXNet-master.322.0/AL2012/DEV.STD.PTHREAD/build/lib/python3.6/site-packages/mxnet/base.py:254: MXNetError
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services