[GitHub] ptrendx opened a new pull request #13346: Aggregate SGD

[GitBox <gi...@apache.org>]

ptrendx opened a new pull request #13346: Aggregate SGD
URL: https://github.com/apache/incubator-mxnet/pull/13346
 
 
   ## Description ##
   Currently MXNet optimizers are invoked 1 weight at a time. This leads to a lot of synchronization overhead, as updates (especially for convolutions and batchnorm) tend to be small, but each one needs to by synchronized upon.
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [x] Changes are complete (i.e. I finished coding on this PR)
   - [x] All changes have test coverage:
   - [x] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments are documented. 
   - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [x] ability to control `update_on_kvstore` value via environment variable `MXNET_UPDATE_ON_KVSTORE` (default is 1, which is consistent with the current behavior)
   - [x] if `update_on_kvstore` is False, in the case of SGD optimizer it attempts to bundle updates of multiple weights together and launches a single kernel to perform them all, reducing the number of kernel calls and synchronizations.
   
   ## Comments ##
   - Current test_sgd automatically uses the new code paths, so no new tests are needed.
   - Code does not support sparse arrays and it will fall back to not aggregated calls when it encounters sparse array in the bundle of weights/gradients
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services