FW: Talk proposal - MXNet AMP (Automatic Mixed Precision)

[apachemxnetday <ap...@nvidia.com>]

From: Vladimir Cherepanov <vc...@nvidia.com>
Date: Friday, November 20, 2020 at 11:23 AM
To: apachemxnetday <ap...@nvidia.com>
Subject: Talk proposal - MXNet AMP (Automatic Mixed Precision)

MXNet AMP (Automatic Mixed Precision)

Automatic Mixed Precision allows some parts of a model to run in fp16 mode, which improves performance without loosing numerical precision. In MXNet 2.0 it is now a core feature. MXNet AMP is implemented by classifying all ops with respect to their ability to be executed in fp16 mode, and casting their inputs accordingly. The casting is currently implemented by monkey-patching Python op functions. We'll present some benchmarking / performance gain examples. AMP benefits are very model dependent. Some real challenges - significantly increased memory consumption in the case of RNNs - a fix is understood, but is still to be implemented. Some perceived - vanishing loss scale - can happen during automatic loss scaling - is an indication of numerical instability, rather than an AMP-specific problem - AMP just exposes it. Future development includes, among other things, layout management (implemented in the container, but not yet upstreamed). There's also a strong early feedback to add API to be able to turn AMP off - a natural way appears to be to make it controlled by a Python context manager.

Vladimir