You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/05/16 18:21:59 UTC

[GitHub] [incubator-mxnet] Ishitori commented on issue #14938: Performance issue when use_bias=True

Ishitori commented on issue #14938: Performance issue when use_bias=True
URL: https://github.com/apache/incubator-mxnet/issues/14938#issuecomment-493178787
 
 
   There is definitely something strange going on. I didn't touch PyTorch, and I think we can just stay with MXNet to see something weird going on.
   
   I took @NRauschmayr code and made it to print out metrics for various values of `flatten`, `use_bias` and context. Here is the code:
   
   ```
   import re
   import sys
   from io import StringIO
   
   import mxnet as mx
   from mxnet import gluon
   import numpy as np
   import time
   
   
   class net(mx.gluon.HybridBlock):
       def __init__(self, flatten, bias):
           super(net, self).__init__()
           self.fr = gluon.nn.HybridSequential()
           self.fr.add(gluon.nn.Dense(50, activation='relu', flatten=flatten, use_bias=bias))
           self.fr.add(gluon.nn.Dense(10))
   
       def hybrid_forward(self, F, x):
           return self.fr(x)
   
   
   def run(flatten, bias, ctx):
       data = np.random.uniform(0, 10000, (100, 10000, 100))
       label = np.random.randint(0, 50, (100, 1))
       label = label.reshape(100, 1)
       batch_size = 100
       ctx = ctx
   
       model = net(flatten, bias)
       model.collect_params().initialize(mx.init.Xavier(), ctx=ctx)
       optimizer = gluon.Trainer(model.collect_params(), 'sgd', {'learning_rate': 1e-2})
       loss = gluon.loss.SoftmaxCrossEntropyLoss()
   
       data = mx.nd.array(data, ctx=ctx)
       label = mx.nd.array(label, ctx=ctx)
   
       sys.stdout = StringIO()
       model.summary(data)
       params_count = re.findall('Total params: (\d+)', sys.stdout.getvalue())[0]
       sys.stdout = sys.__stdout__
   
       total_forward = 0
       total_backward = 0
   
       for i in range(100):
           tic = time.time()
           with mx.autograd.record():
               out = model(data)
               l = loss(out, label)
   
           mx.nd.waitall()
           total_forward += time.time() - tic
           tic = time.time()
   
           l.backward()
           optimizer.step(batch_size)
           mx.nd.waitall()
           total_backward += time.time() - tic
   
       mx.nd.waitall()
       print("Flatten = {}, Bias = {}, Parameters count = {:10}. Forward time: {}, Backward time: {}, Total time: {}".format(
           'T' if flatten else 'F', 'T' if bias else 'F', params_count, total_forward, total_backward, total_forward + total_backward))
   
   
   if __name__ == '__main__':
       print("GPU version:")
       run(flatten=True, bias=True, ctx=mx.gpu())
       run(flatten=True, bias=False, ctx=mx.gpu())
       run(flatten=False, bias=True, ctx=mx.gpu())
       run(flatten=False, bias=False, ctx=mx.gpu())
   
       print("\nCPU version:")
       run(flatten=True, bias=True, ctx=mx.cpu())
       run(flatten=True, bias=False, ctx=mx.cpu())
       run(flatten=False, bias=True, ctx=mx.cpu())
       run(flatten=False, bias=False, ctx=mx.cpu())
   ```
   
   The result I receive is on a P3 instance:
   
   ```
   GPU version:
   Flatten = T, Bias = T, Parameters count = 50000560  . Forward time: 0.544151782989502, Backward time: 0.3901991844177246, Total time: 0.9343509674072266
   Flatten = T, Bias = F, Parameters count = 50000510  . Forward time: 0.509131669998169, Backward time: 0.3707692623138428, Total time: 0.8799009323120117
   Flatten = F, Bias = T, Parameters count = 5005060   . Forward time: 0.43353724479675293, Backward time: 1.8251211643218994, Total time: 2.2586584091186523
   Flatten = F, Bias = F, Parameters count = 5005010   . Forward time: 0.37809014320373535, Backward time: 0.48314833641052246, Total time: 0.8612384796142578
   
   CPU version:
   Flatten = T, Bias = T, Parameters count = 50000560  . Forward time: 1.9246060848236084, Backward time: 2.7853872776031494, Total time: 4.709993362426758
   Flatten = T, Bias = F, Parameters count = 50000510  . Forward time: 1.8545055389404297, Backward time: 2.801023006439209, Total time: 4.655528545379639
   Flatten = F, Bias = T, Parameters count = 5005060   . Forward time: 3.8630404472351074, Backward time: 8.411863565444946, Total time: 12.274904012680054
   Flatten = F, Bias = F, Parameters count = 5005010   . Forward time: 3.154486656188965, Backward time: 4.549597263336182, Total time: 7.7040839195251465
   ```
   
   As I can see from this output, backward time for the case `Flatten = F, Bias = T` becomes much bigger compare to other cases. It is reproducible on both CPU and GPU. Moreover, number of parameters is actually smaller for that case, then in a case of `Flatten = True`, so I can't explain it with "network just has more parameters" statement.
   
   I believe the roots of the problem is somewhere in C++ code, so would appreciate someone more familiar with the engine to look into that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services