[GitHub] idealboy opened a new issue #12029: The performance between different version

[GitBox <gi...@apache.org>]

idealboy opened a new issue #12029: The performance between different version
URL: https://github.com/apache/incubator-mxnet/issues/12029
 
 
   Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.
   
   For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io 
   
   ## Description
   (Brief description of the problem in no more than 2 sentences.)
   
   when I do inferrence in 1.2.1(cuda9.1+cudnn7.1+nccl2.2) with model trained in 0.9.3(cuda8.0+cudnn5.1), it has a low performance(~1500ms on cpu,  ~300ms on gpu), 
   
   but, when I do inference in 1.2.1(cuda9.1+cudnn7.1+nccl2.2) with model trained by itself, it's (<300ms on cpu, <100ms on gpu)
   
   I use resnet-34, inferrence on v100 gpu
   
   ## Environment info (Required)
   
   ```
   What to do:
   1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
   2. Run the script using `python diagnose.py` and paste its output here.
   
   ```
   ----------Python Info----------
   ('Version      :', '2.7.15')
   ('Compiler     :', 'GCC 7.2.0')
   ('Build        :', ('default', 'May  1 2018 23:32:55'))
   ('Arch         :', ('64bit', ''))
   ------------Pip Info-----------
   ('Version      :', '18.0')
   ('Directory    :', '##anaconda2/lib/python2.7/site-packages/pip')
   ----------MXNet Info-----------
   ##/anaconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
     from ._conv import register_converters as _register_converters
   ('Version      :', '1.2.1')
   ('Directory    :', '##/anaconda2/lib/python2.7/site-packages/mxnet-1.2.1-py2.7.egg/mxnet')
   Hashtag not found. Not installed from pre-built package.
   ----------System Info----------
   ('Platform     :', 'Linux-3.18.6-2.el7.centos.x86_64-x86_64-with-centos-7.3.1611-Core')
   ('system       :', 'Linux')
   ('node         :', '##')
   ('release      :', '3.18.6-2.el7.centos.x86_64')
   ('version      :', '#1 SMP Mon Oct 24 13:01:33 CST 2016')
   ----------Hardware Info----------
   ('machine      :', 'x86_64')
   ('processor    :', 'x86_64')
   Architecture:          x86_64
   CPU op-mode(s):        32-bit, 64-bit
   Byte Order:            Little Endian
   CPU(s):                56
   On-line CPU(s) list:   0-55
   Thread(s) per core:    2
   Core(s) per socket:    14
   Socket(s):             2
   NUMA node(s):          2
   Vendor ID:             GenuineIntel
   CPU family:            6
   Model:                 79
   Model name:            Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
   Stepping:              1
   CPU MHz:               2600.074
   BogoMIPS:              5206.47
   Virtualization:        VT-x
   L1d cache:             32K
   L1i cache:             32K
   L2 cache:              256K
   L3 cache:              35840K
   NUMA node0 CPU(s):     0-13,28-41
   NUMA node1 CPU(s):     14-27,42-55
   ----------Network Test----------
   Setting timeout: 10
   Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0751 sec, LOAD: 0.9542 sec.
   Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0009 sec, LOAD: 16.9515 sec.
   Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.2202 sec, LOAD: 2.1099 sec.
   Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.2300 sec, LOAD: 1.5281 sec.
   Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 4.1006 sec, LOAD: 1.4700 sec.
   Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 1.5907 sec, LOAD: 1.4789 sec.
   
   
   Package used (Python/R/Scala/Julia):
   (I'm using Python)
   
   For Scala user, please provide:
   1. Java version: (`java -version`)
   2. Maven version: (`mvn -version`)
   3. Scala runtime if applicable: (`scala -version`)
   
   For R user, please provide R `sessionInfo()`:
   
   ## Build info (Required if built from source)
   
   Compiler (gcc/clang/mingw/visual studio):gcc4.8.5
   
   MXNet commit hash:
   (Paste the output of `git rev-parse HEAD` here.)
   
   Build config:
   (Paste the content of config.mk, or the build command.)
   
   ## Error Message:
   (Paste the complete error message, including stack trace.)
   
   ## Minimum reproducible example
   (If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)
   
   ## Steps to reproduce
   (Paste the commands you ran that produced the error.)
   
   1.
   2.
   
   ## What have you tried to solve it?
   
   1.
   2.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services