You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/02/25 23:00:03 UTC
[GitHub] johnbroughton2017 opened a new issue #9884: How to speed up prediction run time? Copying gpu->cpu takes a long time
johnbroughton2017 opened a new issue #9884: How to speed up prediction run time? Copying gpu->cpu takes a long time
URL: https://github.com/apache/incubator-mxnet/issues/9884
Hi all,
Doing prediction using mxnet has two major part: forward pass and copy results from gpu to cpu memory, as
```
mod.forward(Batch([mx.nd.array(data)]))
prob = mod.get_outputs(0)[0][0].asnumpy()
```
I did a quick timing based on batch size (see below). It seems like the second operation above takes a lot of time when batch size increases.
batch size mod.forward() (ms) mod.get_outputs...asnumpy() (ms)
------------------------------------------------------------------------------------------------
16 5.8 30.1
32 10.5 51.1
48 14 78.7
64 17.8 95.6
80 33.2 121.3
96 36.2 147.5
112 41.3 174.3
128 46.4 245.5
144 52 219
160 56.9 241.2
176 64.9 267.4
192 69.5 329.1
208 73.4 317.1
224 80.7 337.4
240 83.4 446.7
256 93.4 380.7
I don't understand this because copying data from gpu to cpu should be really fast. For example, the following code takes only 0.1ms to run.
```
# speed test
import time
import mxnet as mx
a = mx.nd.random_uniform(shape=(256, 3, 224, 224), ctx=mx.cpu())
b = mx.nd.random_uniform(shape=(256, 3, 224, 224), ctx=mx.gpu())
t0 = time.time()
b.copyto(a)
print time.time()-t0
```
Am I doing this in a wrong way? Any help is highly appreciated. Thanks.
-- John
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services