[GitHub] [incubator-mxnet] Neutron3529 commented on issue #19649: Results are significant different between RTX 2080Ti and RTX 3090

[GitBox <gi...@apache.org>]

Neutron3529 commented on issue #19649:
URL: https://github.com/apache/incubator-mxnet/issues/19649#issuecomment-758323589


   > A torch test case. Torch has lower difference between 2080Ti and 3090. However, MXNet on RTX3090 will have difference up to 0.3 in some cases.
   > 
   > ```python
   >     import torch 
   >     import torchvision as tv
   >     torch.backends.cudnn.benchmark=True
   > 
   >     model = tv.models.resnet18(pretrained=True)
   >     model.cuda(0)
   >     model.eval()
   > 
   >     # y is always 948.1921 on CPU
   >     # y is always 948.1919 on RTX2080Ti whenever cudnn.benchmark is True or False
   >     # y is 948.19165 on RTX3090 when cudnn.benchmark=False
   >     # y varies on RTX3090 when cudnn.benchmark=True: 948.19147, 948.1919
   >     x = torch.ones(1,3,224,224).cuda(0)
   >     y = model(x)
   >     y = y.abs().sum()
   >     print(y.detach().cpu().numpy())
   > ```
   
   have you ever tried `NVIDIA_TF32_OVERRIDE=0 python`?
   3090 using tf32 to accelerate training&testing by default, and using `NVIDIA_TF32_OVERRIDE=0` will disable it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org