You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/05/16 18:21:59 UTC
[GitHub] [incubator-mxnet] Ishitori commented on issue #14938: Performance
issue when use_bias=True
Ishitori commented on issue #14938: Performance issue when use_bias=True
URL: https://github.com/apache/incubator-mxnet/issues/14938#issuecomment-493178787
There is definitely something strange going on. I didn't touch PyTorch, and I think we can just stay with MXNet to see something weird going on.
I took @NRauschmayr code and made it to print out metrics for various values of `flatten`, `use_bias` and context. Here is the code:
```
import re
import sys
from io import StringIO
import mxnet as mx
from mxnet import gluon
import numpy as np
import time
class net(mx.gluon.HybridBlock):
def __init__(self, flatten, bias):
super(net, self).__init__()
self.fr = gluon.nn.HybridSequential()
self.fr.add(gluon.nn.Dense(50, activation='relu', flatten=flatten, use_bias=bias))
self.fr.add(gluon.nn.Dense(10))
def hybrid_forward(self, F, x):
return self.fr(x)
def run(flatten, bias, ctx):
data = np.random.uniform(0, 10000, (100, 10000, 100))
label = np.random.randint(0, 50, (100, 1))
label = label.reshape(100, 1)
batch_size = 100
ctx = ctx
model = net(flatten, bias)
model.collect_params().initialize(mx.init.Xavier(), ctx=ctx)
optimizer = gluon.Trainer(model.collect_params(), 'sgd', {'learning_rate': 1e-2})
loss = gluon.loss.SoftmaxCrossEntropyLoss()
data = mx.nd.array(data, ctx=ctx)
label = mx.nd.array(label, ctx=ctx)
sys.stdout = StringIO()
model.summary(data)
params_count = re.findall('Total params: (\d+)', sys.stdout.getvalue())[0]
sys.stdout = sys.__stdout__
total_forward = 0
total_backward = 0
for i in range(100):
tic = time.time()
with mx.autograd.record():
out = model(data)
l = loss(out, label)
mx.nd.waitall()
total_forward += time.time() - tic
tic = time.time()
l.backward()
optimizer.step(batch_size)
mx.nd.waitall()
total_backward += time.time() - tic
mx.nd.waitall()
print("Flatten = {}, Bias = {}, Parameters count = {:10}. Forward time: {}, Backward time: {}, Total time: {}".format(
'T' if flatten else 'F', 'T' if bias else 'F', params_count, total_forward, total_backward, total_forward + total_backward))
if __name__ == '__main__':
print("GPU version:")
run(flatten=True, bias=True, ctx=mx.gpu())
run(flatten=True, bias=False, ctx=mx.gpu())
run(flatten=False, bias=True, ctx=mx.gpu())
run(flatten=False, bias=False, ctx=mx.gpu())
print("\nCPU version:")
run(flatten=True, bias=True, ctx=mx.cpu())
run(flatten=True, bias=False, ctx=mx.cpu())
run(flatten=False, bias=True, ctx=mx.cpu())
run(flatten=False, bias=False, ctx=mx.cpu())
```
The result I receive is on a P3 instance:
```
GPU version:
Flatten = T, Bias = T, Parameters count = 50000560 . Forward time: 0.544151782989502, Backward time: 0.3901991844177246, Total time: 0.9343509674072266
Flatten = T, Bias = F, Parameters count = 50000510 . Forward time: 0.509131669998169, Backward time: 0.3707692623138428, Total time: 0.8799009323120117
Flatten = F, Bias = T, Parameters count = 5005060 . Forward time: 0.43353724479675293, Backward time: 1.8251211643218994, Total time: 2.2586584091186523
Flatten = F, Bias = F, Parameters count = 5005010 . Forward time: 0.37809014320373535, Backward time: 0.48314833641052246, Total time: 0.8612384796142578
CPU version:
Flatten = T, Bias = T, Parameters count = 50000560 . Forward time: 1.9246060848236084, Backward time: 2.7853872776031494, Total time: 4.709993362426758
Flatten = T, Bias = F, Parameters count = 50000510 . Forward time: 1.8545055389404297, Backward time: 2.801023006439209, Total time: 4.655528545379639
Flatten = F, Bias = T, Parameters count = 5005060 . Forward time: 3.8630404472351074, Backward time: 8.411863565444946, Total time: 12.274904012680054
Flatten = F, Bias = F, Parameters count = 5005010 . Forward time: 3.154486656188965, Backward time: 4.549597263336182, Total time: 7.7040839195251465
```
As I can see from this output, backward time for the case `Flatten = F, Bias = T` becomes much bigger compare to other cases. It is reproducible on both CPU and GPU. Moreover, number of parameters is actually smaller for that case, then in a case of `Flatten = True`, so I can't explain it with "network just has more parameters" statement.
I believe the roots of the problem is somewhere in C++ code, so would appreciate someone more familiar with the engine to look into that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services