[GitHub] nicklhy commented on issue #9055: Re-enable all op segments when in batch mode

[GitBox <gi...@apache.org>]

nicklhy commented on issue #9055: Re-enable all op segments when in batch mode
URL: https://github.com/apache/incubator-mxnet/pull/9055#issuecomment-357575692
 
 
   @KellenSunderland  Thanks a lot for your fix. After reading what you discussed above, I tested your mxnet fork(branch: batched_op_perf_regression) with a script from [here](https://github.com/apache/incubator-mxnet/issues/9396). Though the number of cudaStreamSynchronize calls returned to a small one which is exactly the same as the old mxnet version(i.e. 0.10.0), I also found the avg time cost of this function increased a little bit. In my case, the network runs faster than the original mxnet 1.0.0 but a bit slower than mxnet 0.10.0. At the same time, the gpu usage increased from 83% to 95%, but is still not fully utilized. The details is like below:
   
   ```
   # mxnet 0.10.0
   # GPU usage: 99~100%
   ==21421== API calls:
   Time(%)      Time     Calls       Avg       Min       Max  Name
    52.04%  7.76526s      3691  2.1038ms  1.4510us  9.5652ms  cudaStreamSynchronize
    21.31%  3.17943s    659205  4.8230us  3.3450us  1.2354ms  cudaLaunch
    11.92%  1.77868s         4  444.67ms  60.328us  889.60ms  cudaStreamCreate
   ```
   
   ```
   # mxnet 1.0.0
   # GPU usage: 95%
   ==3718== API calls:
   Time(%)      Time     Calls       Avg       Min       Max  Name
    53.11%  8.31733s      3691  2.2534ms  1.5270us  9.8319ms  cudaStreamSynchronize
    19.48%  3.05095s    631908  4.8280us  2.8860us  1.6870ms  cudaLaunch
     8.01%  1.25366s        16  78.354ms  7.7330us  627.02ms  cudaStreamCreateWithFlags
     4.66%  729.40ms       291  2.5065ms     314ns  178.82ms  cudaFree
   ```
   
   Software env: Ubuntu 16.04
   GPU: GTX 1080 Ti
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services