[GitHub] [incubator-mxnet] Caenorst opened a new pull request #16122: Add fast implementation of LARS

[GitBox <gi...@apache.org>]

Caenorst opened a new pull request #16122: Add fast implementation of LARS
URL: https://github.com/apache/incubator-mxnet/pull/16122
 
 
   ## Description ##
   Add a Fast implementation of LARS.
   
   ## Checklist ##
   ### Essentials ###
   - [X] Changes are complete (i.e. I finished coding on this PR)
   - [X] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
   - [X] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments are documented. 
   - [X] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - Add multi_sum_sq Op: computing sum of squares of multiple arrays (1 sum per array).
   - Add multi_lars Op: computing LARS with the sum of squares of weights and gradients.
   - Add preloaded_multi_sgd which a version of SGD where learning rate are MXNet array instead of list of parameters.
   - Add SGDwFastLARS optimizer
   - Add test for the new Ops and optimizer.
   
   ## Comments ##
   This code have been used for MLPerf v0.6 benchmarks. It's especially pertinent if you are training with small local batch size as optimizers usually don't scale with batch size.
   
   ## Credits ##
   - The CPU part is from @drivanov
   - As said in the comments of multi_sum_sq files, the main approach is coming from https://github.com/NVIDIA/apex

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services