[GitHub] rocketbear opened a new issue #10972: Gluon's PReLU is very slow and a fix to it

[GitBox <gi...@apache.org>]

rocketbear opened a new issue #10972: Gluon's PReLU is very slow and a fix to it
URL: https://github.com/apache/incubator-mxnet/issues/10972
 
 
   I have experienced significantly slow training speed when using PReLU activation instead of using ReLU activation with model composed using the gluon API. The speed of PReLU over ReLU is about 1/5 on GPU when measuring the number of samples processed per second.
   
   Finally I managed to bring the speed of gluon's PReLU back to normal by the following modifications:
   Following is the original init function of PReLU:
   
       def __init__(self, alpha_initializer=initializer.Constant(0.25), **kwargs):
           super(PReLU, self).__init__(**kwargs)
           with self.name_scope():
               self.alpha = self.params.get('alpha', shape=(1,), init=alpha_initializer)
   
   Following is the modified init function of PReLU:
   
       def __init__(self, in_channels=1, alpha_initializer=initializer.Constant(0.25), **kwargs):
           super(PReLU, self).__init__(**kwargs)
           with self.name_scope():
               self.alpha = self.params.get('alpha', shape=(in_channels,), init=alpha_initializer)
   
   The key is to pass in the expected number of channels to the PReLU block, so that it does not share the negative slope among the channels. The downside of this solution is that you need to pass in the number of channels every time.
   
   I don't know why the two settings (shared vs. non-shared) have so drastically different performance. The contributors of mxnet should investigate this issue in further depth.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services