[GitHub] [incubator-mxnet] kexinyu edited a comment on issue #17164: net.Cast("float16") doesn't work: Check failed: (*in_type)[i] == dtype_param (2 vs. 0) : This layer requires uniform type. Expected 'float32' v.s. given 'float16' at 'gamma'

[GitBox <gi...@apache.org>]

kexinyu edited a comment on issue #17164: net.Cast("float16") doesn't work: Check failed: (*in_type)[i] == dtype_param (2 vs. 0) : This layer requires uniform type. Expected 'float32' v.s. given 'float16' at 'gamma'
URL: https://github.com/apache/incubator-mxnet/issues/17164#issuecomment-569778847
 
 
   > This is a known issue with BatchNorm operator not supporting float16. It needs some work to support, not currently a bug but not-implemented feature
   > @mxnet-label-bot update [Feature ]
   
   Then how does the current BatchNorm operator handle **float16** inputs in mixed precision training?
   https://github.com/apache/incubator-mxnet/blob/b6972bb055fc44481b072db3abb90e26ee27c787/src/operator/nn/batch_norm-inl.h#L292
   Here, if the inputs use float16, `DType` becomes float16 but `AccReal` is float32.
   https://github.com/apache/incubator-mxnet/blob/5fb29167a1a66480864486bf59c6b4e980ce7daa/src/operator/nn/batch_norm.cu#L241
   Is `gamma * (inp - mean) * invstd + beta` okay with mixed precision operands? 
   https://github.com/apache/incubator-mxnet/blob/692f49f2b1b9df1bb226c586405226291c6095cf/src/operator/nn/convolution.cc#L297
   Or are they the reason why `UNIFORM_TYPE_CHECK` fails with error
   `Check failed: (*in_type)[i] == dtype_param (2 vs. 0) : This layer requires uniform type. Expected 'float32' v.s. given 'float16' at 'gamma'`
   ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services