[GitHub] [incubator-mxnet] wkcn opened a new issue #13928: MXNet 1.5.0 is slower than 1.3.0 when intputs are variant

[GitBox <gi...@apache.org>]

wkcn opened a new issue #13928: MXNet 1.5.0 is slower than 1.3.0 when intputs are variant
URL: https://github.com/apache/incubator-mxnet/issues/13928
 
 
   ## Description
   Hi! I have an experiment about Object Counting, which needs variant inputs.
   I write the code with Gluon, and hybridize the model with `static_alloc=True`
   I found there is obvious difference between MXNet 1.5.0 and MXNet 1.3.0, and I checked it on two servers.
   
   I think the method of memory allocation for Gluon may be changed after MXNet 1.3.0.
   
   Thanks!
   
   ## Environment info (Required)
   OS: ubuntu 14.04
   GPU: Tesla M40 x 4
   
   ## Minimum reproducible example
   I write a minimum reproducible example without dataset.
   [Code](https://gist.githubusercontent.com/wkcn/69f0f6d2ca467816dc481a00c225104f/raw/2899896f42a920ff0fde5ff93b9a16d16aec507f/test_fcn_for_mxnet.py)
   
   - Performance for test code [a fully convolutional model(vgg16 without FC layer), variant inputs]:
   MXNet 1.5.0: 10 images / sec
   MXNet 1.3.0: 40+ images / sec
   
   The performances are the same when input shape is fixed.
   
   Input shape: (9, 3, 300\~512, 300\~512) in NCHW order
   
   Package used (Python/R/Scala/Julia):
   Python 2.7.12, 3.7.1
   
   MXNet is installed by pip:
   ```
   # MXNet 1.5.0
   pip install mxnet-cu80 --pre
   # MXNet 1.3.0
   pip install mxnet-cu80==1.3.0
   ```
   
   ## Steps to reproduce
   Download [the test code](https://gist.githubusercontent.com/wkcn/69f0f6d2ca467816dc481a00c225104f/raw/2899896f42a920ff0fde5ff93b9a16d16aec507f/test_fcn_for_mxnet.py).
   Run the test code in different version (1.3.0 and 1.5.0) of MXNet.
   
   ## Performance
   
   I test several versions of MXNet.
   
   version|performance
   ----------|----------------
   1.4.0b20181207|slow
   1.3.1b20181101|slow
   1.3.1b20181010|slow
   1.3.1b20181004|fast
   1.3.1b20181001|fast
   
   Some pre-build versions don't  support CUDA9.0, so I cound't test it.
   The performance drops during 20181004 to 20181010.
   
   If changing the dilation of dilated conv to 1, the performance will be normal.
   It seems the problem occurs in dilated conv.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services