You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/03/21 19:00:54 UTC
[GitHub] [incubator-mxnet] samskalicky opened a new issue #14496:
performance degradation from 1.3.1 to 1.4.0
samskalicky opened a new issue #14496: performance degradation from 1.3.1 to 1.4.0
URL: https://github.com/apache/incubator-mxnet/issues/14496
There appears to be some performance degradation between the 1.3.1 and 1.4.0 releases. So far we know of imdecode and tranpose operators having reduced performance.
We've tracked the responsible PRs for these operators to:
Transpose:
https://github.com/dmlc/mshadow/pull/359/files
https://github.com/apache/incubator-mxnet/pull/11742
Imdecode:
https://github.com/apache/incubator-mxnet/pull/8757
Im using the cat.jpg from here as input data:
https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/cat.jpg
Heres the current performance benchmarking script for imdecode:
```
import time
import mxnet as mx
from mxnet import image as img
import numpy as np
flag=1
to_rgb=True
out=None
dtimes = []
for i in range(1000):
buf = open("cat.jpg", 'rb').read()
start = time.time()
img.imdecode(buf, flag, to_rgb, out)
decode = (time.time() - start)*1000
dtimes.append(decode)
print('mxnet version: %s' % mx.__version__)
print('decode:---------')
print('p50: %4.2f ms' % np.percentile(dtimes,50))
print('p90: %4.2f ms' % np.percentile(dtimes,90))
print('p99: %4.2f ms' % np.percentile(dtimes,99))
```
And here are the performance results using mxnet-cu90mkl for 1.4.0 and 1.3.1:
mxnet version: 1.4.0
decode:---------
p50: 13.75 ms
p90: 15.74 ms
p99: 19.89 ms
mxnet version: 1.3.1
decode:---------
p50: 12.82 ms
p90: 13.59 ms
p99: 13.88 ms
Setting the flag = 1 + 128 (instead of just 1) as an argument to imdecode results in the following results:
mxnet version: 1.4.0
load:---------
p50: 0.10 ms
p90: 0.16 ms
p99: 0.18 ms
decode:---------
p50: 13.27 ms
p90: 13.83 ms
p99: 15.63 ms
So there is some additional work thats going on that makes the imdecode take longer. This is a "feature" of the v1.4 release and changed the defaults which is where some performance degradation is happening at least in the imdecode function.
Heres the current performance benchmarking script for transpose:
```
import mxnet as mx
import time
import numpy as np
sizes = [10, 50, 100,200,500]
iters = [10000,1000,500,200,20]
times = []
for size in range(len(sizes)):
data = []
s = sizes[size]
print(s)
for i in range(iters[size]):
x = mx.nd.ones((s,s,s))
mx.nd.waitall()
start = time.time()
y = mx.nd.transpose(x,(2,0,1))
mx.nd.waitall()
data.append((time.time() - start)*1000)
#print(data[-1])
times.append(data)
print('mxnet version: %s' % mx.__version__)
for s in range(len(sizes)):
print('--------------------')
print('size: %s' % str(sizes[s]))
print('p50: %4.2f ms' % np.percentile(times[s],50))
print('p90: %4.2f ms' % np.percentile(times[s],90))
print('p99: %4.2f ms' % np.percentile(times[s],99))
```
And here are the performance results using mxnet-cu90mkl for 1.4.0 and 1.3.1:
mxnet version: 1.4.0
--------------------
size: 10
p50: 0.04 ms
p90: 0.04 ms
p99: 0.05 ms
--------------------
size: 50
p50: 1.08 ms
p90: 1.09 ms
p99: 1.13 ms
--------------------
size: 100
p50: 12.50 ms
p90: 12.53 ms
p99: 13.52 ms
--------------------
size: 200
p50: 123.06 ms
p90: 125.63 ms
p99: 125.95 ms
--------------------
size: 500
p50: 2768.49 ms
p90: 2797.46 ms
p99: 2809.45 ms
mxnet version: 1.3.1
--------------------
size: 10
p50: 0.03 ms
p90: 0.04 ms
p99: 0.04 ms
--------------------
size: 50
p50: 0.36 ms
p90: 0.37 ms
p99: 0.37 ms
--------------------
size: 100
p50: 2.79 ms
p90: 2.90 ms
p99: 3.97 ms
--------------------
size: 200
p50: 46.05 ms
p90: 50.07 ms
p99: 50.11 ms
--------------------
size: 500
p50: 1094.50 ms
p90: 1095.89 ms
p99: 1096.44 ms
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services