[GitHub] rahul003 edited a comment on issue #11842: Strange results from fp16 benchmarks on Alexnet on CIFAR10

[GitBox <gi...@apache.org>]

rahul003 edited a comment on issue #11842: Strange results from fp16 benchmarks on Alexnet on CIFAR10
URL: https://github.com/apache/incubator-mxnet/issues/11842#issuecomment-406748642
 
 
   Hi Ray,
   Cifar doesn't lend itself well for fp16 computation. 
   
   Nvidia Tensor Cores essentially perform the computation D = A * B + C, where A and B are half precision matrices, while C and D could be either half precision or full precision. The tensor cores can be leveraged when dimensions of these matrices are multiples of 8. This means that Tensor Cores can not be used in all cases for fast float16 computation. When training Cifar10 dataset, the tensors involved are sometimes smaller, and Tensor Cores can not always be used. The computation in that case falls back to slower algorithms and using float16 turns out to be slower than float32 on a single GPU. Note that when using multiple GPUs, using float16 can still be faster than float32 because of reduction in communication costs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services