You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/03/01 22:19:28 UTC

[GitHub] sxjscience opened a new issue #9950: [OP] LayerNorm in MXNet

sxjscience opened a new issue #9950: [OP] LayerNorm in MXNet
URL: https://github.com/apache/incubator-mxnet/issues/9950
 
 
   ## Description
   
   [Layer Normalization](https://arxiv.org/pdf/1607.06450.pdf) is becoming more and more common in deep learning models, especially RNNs. We need to support the layer normalization layer. Although we can implement it by combining the existing symbols, e.g, [Implementation in Sockeye](https://github.com/awslabs/sockeye/blob/master/sockeye/layers.py#L53-L112), the speed and memory cost would be large due to the various broadcasting operators that cannot be calculated inplace. Thus, we need to write our own C++ implementation of this operator.
   
   Some reference implementations:
   1. https://github.com/pytorch/pytorch/issues/1959
   2. https://github.com/MycChiu/fast-LayerNorm-TF

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services