You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/05/11 23:30:07 UTC
[GitHub] hetong007 opened a new issue #10912: nn.GlobalAvgPool2D is slow in
backpropagation
hetong007 opened a new issue #10912: nn.GlobalAvgPool2D is slow in backpropagation
URL: https://github.com/apache/incubator-mxnet/issues/10912
## Description
As stated in this PR for GluonCV (https://github.com/dmlc/gluon-cv/pull/113), I tested performance of `se_resnext50_32x4d` with dummy data (to get rid of possible side effect of data loader), and found that
- With `nn.GlobalAvgPool2D`, training speed is 462 images/sec
- When replaced with `contrib.AdaptiveAvgPooling2D`, training speed increased to 643 images/sec
## Results with Profiler
With `nn.GlobalAvgPool2D`, from one single batch in the result of `mxnet.profiler`:
- The operator `_backward_Pooling` consumes a significant amount of time (152 ms in one batch), being the second most time consuming operator.
- As a comparison, `_backward_Convolution` consumes 181 ms in one batch. However in the network, the number of`nn.GlobalAvgPool2D` and `nn.Conv2D` is 1:3, and convolution is expected to be more computational expensive.
It could be the case that `nn.GlobalAvgPool2D` doesn't call cudnn, which affects the performance.
## Environment info (Required)
```
----------Python Info----------
('Version :', '2.7.12')
('Compiler :', 'GCC 5.4.0 20160609')
('Build :', ('default', 'Nov 20 2017 18:23:56'))
('Arch :', ('64bit', 'ELF'))
------------Pip Info-----------
('Version :', '9.0.1')
('Directory :', '/usr/local/lib/python2.7/dist-packages/pip')
----------MXNet Info-----------
('Version :', '1.2.0')
('Directory :', '/usr/local/lib/python2.7/dist-packages/mxnet')
('Commit Hash :', 'f02caae5a06c84d2e764f3a4c4283cf838e3d69f')
----------System Info----------
('Platform :', 'Linux-4.4.0-1052-aws-x86_64-with-Ubuntu-16.04-xenial')
('system :', 'Linux')
('node :', 'ip-172-31-16-166')
('release :', '4.4.0-1052-aws')
('version :', '#61-Ubuntu SMP Mon Feb 12 23:05:58 UTC 2018')
----------Hardware Info----------
('machine :', 'x86_64')
('processor :', 'x86_64')
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping: 1
CPU MHz: 1420.699
CPU max MHz: 3000.0000
CPU min MHz: 1200.0000
BogoMIPS: 4600.13
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 46080K
NUMA node0 CPU(s): 0-31
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single retpoline kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0015 sec, LOAD: 0.4066 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0022 sec, LOAD: 0.3150 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0213 sec, LOAD: 0.0889 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0078 sec, LOAD: 0.3392 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1028 sec, LOAD: 0.1176 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 2.1753 sec, LOAD: 0.1001 sec.
```
Package used (Python/R/Scala/Julia):
`python`
## What have you tried to solve it?
Replace it with `contrib.AdaptiveAvgPooling2D`, training speed is boosted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services