[GitHub] [incubator-mxnet] MrRaghav commented on pull request #16487: Fix learning rate scheduler being unexpectedly overwritten by optimizer's default value

[GitBox <gi...@apache.org>]

MrRaghav commented on pull request #16487:
URL: https://github.com/apache/incubator-mxnet/pull/16487#issuecomment-653756605


   Hello,
   I am still getting this error while using mxnet with sockeye. Since, this was fixed in the new release, I didn't open a new bug.
   Please find the details in following points:
   
   1) I'm using following versions of mxnet and sockeye 2.1.7 (on CUDA 10.1) 
       [username]@[server]:~/username/sockeye/dir1$ pip3 list | grep mxnet
       mxnet               1.6.0
       **mxnet-cu101mkl      1.6.0**
       mxnet-mkl           1.6.0
       [username]@[server]:~/username/sockeye/dir1$ pip3 list | grep sockeye
       **sockeye             2.1.7**
   
   2) When I run the **sockeye.train** command with arguments, I get following log:
   
      _[username]@[server]:~/username/sockeye$ tail -30 77233.out
     File "/home/username/.local/lib/python3.7/site-packages/sockeye/train.py", line 997, in <module>
       main()
     File "/home/username/.local/lib/python3.7/site-packages/sockeye/train.py", line 764, in main
       train(args)
     File "/home/username/.local/lib/python3.7/site-packages/sockeye/train.py", line 992, in train
       training_state = trainer.fit(train_iter=train_iter, validation_iter=eval_iter, checkpoint_decoder=cp_decoder)
     File "/home/username/.local/lib/python3.7/site-packages/sockeye/training.py", line 242, in fit_
       self._step(batch=train_iter.next())
     File "/home/username/.local/lib/python3.7/site-packages/sockeye/training.py", line 346, in _step
       loss_func.metric.update(loss_value.asscalar(), num_samples.asscalar())
     File "/home/username/.local/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 2553, in asscalar
       return self.asnumpy()[0]
     File "/home/username/.local/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 2535, in asnumpy
       ctypes.c_size_t(data.size)))
     File "/home/username/.local/lib/python3.7/site-packages/mxnet/base.py", line 255, in check_call_
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   **mxnet.base.MXNetError: [09:58:26] src/storage/./pooled_storage_manager.h:161: cudaMalloc retry failed: out of memory**
   Stack trace:
     [bt] (0) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x6d554b) [0x7f6c5b3d054b]
     [bt] (1) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x41a0c72) [0x7f6c5ee9bc72]
     [bt] (2) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x41a694f) [0x7f6c5eea194f]
     [bt] (3) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x3972e10) [0x7f6c5e66de10]
     [bt] (4) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x39730c7) [0x7f6c5e66e0c7]
     [bt] (5) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::RunContext)#1}::operator()(mxnet::RunContext) const+0x281) [0x7f6c5e66e4d1]
     [bt] (6) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x3896f19) [0x7f6c5e591f19]
     [bt] (7) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x38a3c31) [0x7f6c5e59ec31]
     [bt] (8) /home/username/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x38a7170) [0x7f6c5e5a2170]_
   
   
     **learning rate from ``lr_scheduler`` has been overwritten by ``learning_rate`` in optimizer.**
   
   
   3) Also, I can see another error "cudaMalloc retry failed: out of memory" in above log and I checked https://github.com/deepinsight/insightface/issues/257 to find a fix. They've mentioned that reducing the batches solves the issue but I am not using any such argument in **sockeye.train**
   
   4) The arguments used with sockeye are as follows:
       _python3 -m sockeye.train -d training_data \
                           -vs dev.BPE.de \
                           -vt dev.BPE.en \
                           --shared-vocab \
                           -o parallel/wmt_model_
   
   5) I found the code https://mxnet.apache.org/api/python/docs/_modules/mxnet/optimizer/optimizer.html which says learning_rate should be assigned to self.lr_scheduler.base_lr with above warning. But, I am getting it as an error and the output comes as failed.
       
   6) Moreover, I checked the release notes of mxnet 1.6.0 from below link and can see that this issue has been fixed.
       https://cwiki.apache.org/confluence/display/MXNET/1.6.0+Release+notes#id-1.6.0Releasenotes-Bugfixes
   
   I hope I didn't miss anything before coming to you. can you please suggest what should be done in such scenario?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org