[GitHub] StephanieYuan opened a new pull request #12376: [MXNET-854] SVRG Optimization in Python Module API

[GitBox <gi...@apache.org>]

StephanieYuan opened a new pull request #12376: [MXNET-854] SVRG Optimization in Python Module API
URL: https://github.com/apache/incubator-mxnet/pull/12376
 
 
   ## Description ##
   SVRG stands for Stochastic Variance Reduced Gradient, which is an optimization technique that complements SGD. It is introduced in the paper _[Accelerating Stochastic Gradient Descent using Predicative Variance Reduction](https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf)_ in 2013. 
   Key Characteristics of SVRG: 
   - Explicit Variance Reduction
   - Converge much faster compared to SGD on strongly convex smooth functions   
   
   SVRG Module should:
   - seamlessly supports both dense and sparse data, run on CPU and GPU instances on single machine and in distributed setting.
   - minimize the API differences with Python Module API.
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-854], where MXNET-854 refers to the relevant [JIRA issue](https://issues.apache.org/jira/browse/MXNET-854) created (except PRs with tiny changes)
   - [ ] All changes have test coverage:
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - SVRGModule inherits from Python Module and minimizes the API changes of Module API. Both intermediate level and high level APIs are supported
   - Functional tests and unittests have been tested for SVRGModule and SVRG Optimizer.
   
   ## Comments ##
   - Design decisions of why using Python Module instead of Optimizer can be found in the design doc [here](https://cwiki.apache.org/confluence/display/MXNET/SVRG+Optimization+in+MXNet+Python+Module)
   - This version of SVRGModule supports single machine run with single cpu, single gpu and multiple-gpus. Distributed SVRGModule is soon to be updated.
   - Benchmarking has been performed on this single machine version SVRGModule on single cpu on linear regression model. Distributed SVRGModule to be updated soon. Benchmarking results can be found in contrib/svrg_optimization_python/benchmarks.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services