You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/07/02 06:36:02 UTC

[GitHub] [incubator-mxnet] roywei opened a new issue #15429: Operator Performance regression on CPU

roywei opened a new issue #15429: Operator Performance regression on CPU
URL: https://github.com/apache/incubator-mxnet/issues/15429
 
 
   Follow up on dev list discussion:
   
   https://lists.apache.org/thread.html/154ef1e4010671e7375c7a7cbedb413d5a4a3677321488440fb32a3a@%3Cdev.mxnet.apache.org%3E
   
   We have found some operators to have performance regression using the operator benchmark module here:
   https://github.com/apache/incubator-mxnet/tree/master/benchmark/opperf
   
   @sandeep-krishnamurthy has helped to run the benchmark and this is the result:
   https://gist.github.com/sandeep-krishnamurthy/e0a2be893c8c4d484390c9c8813bdf50
   
   The above result is using training mode (`autograd.record()`) and calculating both forward and backward time.
   
   To further investigate the impact on inference I have run the scripts 
   
   Please find the results here: 
   https://docs.google.com/spreadsheets/d/1_eezNWbrBAm3s3i6G1m0Rd3YYdTEnmKlYtn4klqdyN0/edit?usp=sharing
   
   I have calculated the regression percentage and sorted them, thanks to @aaronmarkham for providing the first version.
   
   Although there are variances on perf numbers between runs, we observe the following commonly used operators be slower consistently.
   
   We need to fix them
   
   - [ ] BatchNorm
   - [ ] Dropout
   - [ ] relu
   - [ ] LeakyReLU
   - [ ] dot
   - [ ] element wise ops (mul, div, sub)
   - [ ] broadcast ops (mul, sub)
   
   Some ops regression seems only to happen on mxnet-mkl version (refer to 4th sheet of the google sheet)
   
   Environment:
   
   AWS C5.18xLarge
   Deep Learning Base AMI (Ubuntu) Version 18.1
   Python 3.6
   
   Scripts:
   https://github.com/apache/incubator-mxnet/tree/master/benchmark/opperf
   Notes: to run operators in inference mode, you need to set `False` at this line
   https://github.com/apache/incubator-mxnet/blob/master/benchmark/opperf/utils/op_registry_utils.py#L73
   
   and change `run_backward` to `False` in all files under: 
   https://github.com/apache/incubator-mxnet/tree/master/benchmark/opperf/nd_operations
   for example:
   https://github.com/apache/incubator-mxnet/blob/master/benchmark/opperf/nd_operations/gemm_operators.py#L59

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services