[GitHub] githubfragments opened a new issue #7380: strange phenomenon: 1080Ti is slower than maxwell gtx titan x. How to analysis the results of Profiler?

[gi...@git.apache.org]

githubfragments opened a new issue #7380: strange phenomenon: 1080Ti is slower than maxwell gtx titan x. How to analysis the results of Profiler?
URL: https://github.com/apache/incubator-mxnet/issues/7380
 
 
   
   For bugs or installation issues, please provide the following information.
   The more information you provide, the more likely people will be able to help you.
   
   ## Environment info
   Operating System:
   
   Compiler:
   
   Package used (Python/R/Scala/Julia):
   
   MXNet version:
   
   Or if installed from source:
   
   MXNet commit hash (`git rev-parse HEAD`):
   
   If you are using python package, please provide
   
   Python version and distribution:
   
   
   I use 1080Ti and maxwell titan x to train the same network with the same mxnet, but 1080Ti is strangely slower. I use the profiler to analyze the results, as following figures. One iteration cycle of 1080ti takes 1.343s; One iteration cycle of maxwell titan x  takes 1.005s. In the figures we can see the forward and backward time of 1080ti is much smaller than maxwell titan x. But for 1080ti, there is a large unknown time consumption after each syncCopyGpu2Cpu operation(the green block).  Do you know what is unknown time doing in the mxnet? Why the 1080ti is slower than maxwell Titan x?  @piiswrong 
   Thank you!
   Evironments:
   both: cuda 8.0, cudnnv6.0, mxnet Version 0.10.0 Release. I have recompiled the mxnet.
   for 1080ti:  cpu: e5-2620, ubuntu 16.04
   for maxwell titan x: cpu:i7-4790k, ubuntu 14.04
   
   1080ti-profiler:
   ![1080ti-profiler2](https://user-images.githubusercontent.com/16164389/29058204-bbd12740-7c41-11e7-86a0-335aaca8ce53.png)
   maxwell gtx titan x profiler:
   ![max_well_titan_x](https://user-images.githubusercontent.com/16164389/29058587-acff940c-7c43-11e7-999e-6c3170920fa8.png)
   
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services