You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/07/06 16:10:59 UTC
[GitHub] [incubator-mxnet] sammieghabra opened a new issue #18665: Using ShiftScale operation (Similar to BatchNorm) in MXNet (Customer question/inquiry)
sammieghabra opened a new issue #18665:
URL: https://github.com/apache/incubator-mxnet/issues/18665
## Description
Hi MXNet team,
My team wants to implement a layer in MXNet to implement a ShiftScale operator. This layer is similar to batchnorm - however, we want `moving_mean` and `moving_var` to be used instead of `data_mean` and `data_var` to compute the output of the layer. I see that batch norm has a flag `use_global_stats`, and in the [mxnet docs](http://beta.mxnet.io/r/api/mx.symbol.BatchNorm.html), it seems that setting this flag to be true would be something similar to what I'm trying to do. However, upon inspecting the [batch-norm code](https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/batch_norm.cc#L260-L271), it seems that running_mean and running_var won't be updated if that flag is set to true for training.
1. Is there a reason why from a design perspective setting this `use_global_stats` flag to be true wont update the running mean and running var?
2. We would like to support this shift scale layer during training. So what my proposal is to do is to add another flag to the `batchNorm` operator to be "use_shift_scale", which would simply replace mean and var with running mean and running var when updating the weights. Is this something that MXNet team would be ok with?
3. We also plan to train with more than one instance - will the running_mean and running_var parameters be the same across instances?
Thanks
Sammie
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #18665: Using ShiftScale operation (Similar to BatchNorm) in MXNet (Customer question/inquiry)
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on issue #18665:
URL: https://github.com/apache/incubator-mxnet/issues/18665#issuecomment-657893291
@rondogency and this is for training right? for inference MXNet supports running mean/var right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] rondogency commented on issue #18665: Using ShiftScale operation (Similar to BatchNorm) in MXNet (Customer question/inquiry)
Posted by GitBox <gi...@apache.org>.
rondogency commented on issue #18665:
URL: https://github.com/apache/incubator-mxnet/issues/18665#issuecomment-654368718
Hi, are you using Gluon API? according to the doc it is supported https://github.com/apache/incubator-mxnet/blob/v1.7.x/python/mxnet/gluon/nn/basic_layers.py#L306, if it's not then it would be a bug in Gluon
Also which MXNet version are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #18665: Using ShiftScale operation (Similar to BatchNorm) in MXNet (Customer question/inquiry)
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on issue #18665:
URL: https://github.com/apache/incubator-mxnet/issues/18665#issuecomment-662016100
Discussed offline. This one is for training.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] rondogency commented on issue #18665: Using ShiftScale operation (Similar to BatchNorm) in MXNet (Customer question/inquiry)
Posted by GitBox <gi...@apache.org>.
rondogency commented on issue #18665:
URL: https://github.com/apache/incubator-mxnet/issues/18665#issuecomment-654546346
can confirm currently MXNet doesn't support both update global running mean/var and use running mean/var to calculate grads in one pass of Batchnorm, while Keras-MXNet can support it by passing mean/var as arguments to Batchnorm as a workaround
adding the support would still have constraints, especially cudnn doesn't support it so it cannot run with the best performance; some other frameworks like caffe doesn't support this either
it can be implemented by modifying batch_norm.cc/cu and add "use_shift_scale" option to control it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sammieghabra commented on issue #18665: Using ShiftScale operation (Similar to BatchNorm) in MXNet (Customer question/inquiry)
Posted by GitBox <gi...@apache.org>.
sammieghabra commented on issue #18665:
URL: https://github.com/apache/incubator-mxnet/issues/18665#issuecomment-654371413
@rondogency
Thank you so much for your response.
As mentioned in the description of this issue, I am aware of the `use_global_stats` flag in batch norm. However, as I stated in the description, running mean and running var do not get updated if that flag is true for [training](https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/batch_norm.cc#L260-L271). For our case, we need those values to be updated for training.
I am using version 1.5 of MXNet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] rondogency commented on issue #18665: Using ShiftScale operation (Similar to BatchNorm) in MXNet (Customer question/inquiry)
Posted by GitBox <gi...@apache.org>.
rondogency commented on issue #18665:
URL: https://github.com/apache/incubator-mxnet/issues/18665#issuecomment-654546431
@zhreshold @sxjscience FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org