[GitHub] [incubator-mxnet] hetong007 opened a new pull request #16487: Fix learning rate scheduler being unexpectedly overwritten by optimizer's default value

[GitBox <gi...@apache.org>]

hetong007 opened a new pull request #16487: Fix learning rate scheduler being unexpectedly overwritten by optimizer's default value
URL: https://github.com/apache/incubator-mxnet/pull/16487
 
 
   ## Description ##
   
   In concept, learning rate scheduler takes care of everything related to the learning rate changing bahavior during the training, which of course should have a higher level of control of the learning rate other than the optimizer.
   
   Right now the following example (from https://github.com/dmlc/gluon-cv/issues/750) yields surprising and unexpected output:
   
   ```python
   import mxnet as mx 
   from mxnet import LRScheduler
   lr_scheduler = mx.lr_scheduler.CosineScheduler(1000, base_lr=10, final_lr=0) 
   optim = mx.optimizer.create('sgd', lr_scheduler=lr_scheduler)
   
   print(optim.learning_rate) # 0.01
   ```
   
   In the current implementation of optimizer, mxnet overwrite the learning rate scheduler's learning rate with the optimizer's learning rate by 
   
   ```python
   if lr_scheduler is not None:	
       self.lr_scheduler.base_lr = learning_rate
   ```
   
   Therefore, when users define a learning rate scheduler, and use it as the parameter of the optimizer, they are going to surprisingly find that the learning rate has been overwritten by the default value, `0.01`.
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [X] Changes are complete (i.e. I finished coding on this PR)
   - [X] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
   - [X] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services