[GitHub] [incubator-mxnet] matteosal commented on issue #21111: cuDNN batchnorm behaviour is not consistent and it can output nan

[GitBox <gi...@apache.org>]

matteosal commented on issue #21111:
URL: https://github.com/apache/incubator-mxnet/issues/21111#issuecomment-1235710738

   @DickJC123 you were in fact right about biased vs unbiased variance computation. This script tests such claim by letting a non-cudnn batchnorm and a cudnn-batchnorm update their moving variance, and checking that they are updated differently and that they respectively correspond to the biased (non-cudnn) and the unbiased (cudnn) computations:
   ```
   import mxnet as mx
   import numpy as np
   from mxnet import autograd
   
   print("**** cudnn batchnorm variance")
   
   shapes = {'input': [1, 6, 5], 'gamma': [6], 'beta': [6], 'mean': [6], 'var': [6]}
   
   # Define batchnorms with identical specs except cudnn_off
   # Note that momentum is 0, so moving arrays are replaced everytime with the latest one
   sym1 = mx.symbol.BatchNorm(
   	*[mx.symbol.Variable(name) for name in shapes.keys()],
   	eps=0.001,
   	momentum=0,
   	fix_gamma=False,
   	use_global_stats=False,
   	axis=1,
   	cudnn_off=True
   )
   sym2 = mx.symbol.BatchNorm(
   	*[mx.symbol.Variable(name) for name in shapes.keys()],
   	eps=0.001,
   	momentum=0,
   	fix_gamma=False,
   	use_global_stats=False,
   	axis=1,
   	cudnn_off=False
   )
   op1 = mx.ndarray.CachedOp(sym1)
   op2 = mx.ndarray.CachedOp(sym2)
   
   # Define arrays for op1 and 
   # They are identical now, but they will be changed differently by the ops
   args1 = [mx.np.random.uniform(size=shape, ctx=mx.gpu()) for shape in shapes.values()]
   args2 = [mx.np.array(array, ctx=mx.gpu()) for array in args1]
   
   data, gamma, beta, mean, var = args1
   
   # Evaluation in training mode with backward that rewrites moving mean and var
   with autograd.record(train_mode=True):
   	[arg.attach_grad() for arg in args1]
   	[arg.attach_grad() for arg in args2]
   	dummy1 = op1(*args1, default_ctx=mx.gpu())
   	dummy2 = op2(*args2, default_ctx=mx.gpu())
   autograd.backward(dummy1, head_grads=mx.np.ones(shapes['input'], ctx=mx.gpu()))
   autograd.backward(dummy2, head_grads=mx.np.ones(shapes['input'], ctx=mx.gpu()))
   
   # Check that outputs are the same
   print()
   print("difference between training mode outputs")
   print(mx.np.max(mx.np.abs(dummy1 - dummy2)))	
   
   # Check updated moving vars and observe they are different
   print()
   print("variance updated by the non-cudnn batchnorm")
   print(args1[-1])
   print("variance updated by the cudnn batchnorm")
   print(args2[-1])
   
   # Manually compute biased and unbiased variance
   data_mean = mx.np.mean(data, axis=(-1))
   data_zeromean = data - data_mean[:, :, np.newaxis]
   var1 = mx.np.mean((data_zeromean * data_zeromean), axis=(-1))
   var2 = var1 * shapes['input'][-1] / (shapes['input'][-1] - 1)
   
   print()
   print("manual biased variance")
   print(var1)
   print("manual unbiased variance")
   print(var2)
   ```
   output is:
   ```
   **** cudnn batchnorm variance
   
   difference between training mode outputs
   2.3841858e-07
   
   variance updated by the non-cudnn batchnorm
   [0.12171984 0.03338415 0.03920404 0.04988261 0.02153183 0.02420242] @gpu(0)
   variance updated by the cudnn batchnorm
   [0.15214981 0.04173018 0.04900505 0.06235326 0.02691478 0.03025302] @gpu(0)
   
   manual biased variance
   [[0.12171984 0.03338414 0.03920404 0.04988261 0.02153182 0.02420242]] @gpu(0)
   manual unbiased variance
   [[0.1521498  0.04173018 0.04900505 0.06235326 0.02691478 0.03025302]] @gpu(0)
   ```
   
   So this shows that:
   1) The training mode output is the same between non-cudnn and cudnn implementations ("difference between training mode outputs"), so they are computing the data variance in the same way at this step.It can be checked manually that their result corresponds to using the biased variance
   2) However the way the end up changing their moving variance is different. In particular, the non-cudnn case uses the biased variance as before but the cudnn case uses the non-biased variance this time. Note that the momentum is set to 0 for both ops, which means that moving arrays are replaced with the latest ones, that makes it easy to check the results
   3) This explains the numerical error found in my original report. For a spatial size of 1, the unbiased variance gets multiplied by a factor **1 / (1 - 1) = nan** which would make a subsequent evaluation fail for the cudnn case


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org