[GitHub] yifeim opened a new issue #14048: `take` does not support `grad_stype='row_sparse'`

[GitBox <gi...@apache.org>]

yifeim opened a new issue #14048: `take` does not support `grad_stype='row_sparse'`
URL: https://github.com/apache/incubator-mxnet/issues/14048
 
 
   ## Description
   The `take` operator currently does not support `grad_stype='row_sparse'`, which can impact performance. This post describes the issue and possible work-arounds until it is implemented.
   
   ## Environment info (Required)
   
   ```
   ----------Python Info----------
   Version      : 3.6.5
   Compiler     : GCC 7.2.0
   Build        : ('default', 'Apr 29 2018 16:14:56')
   Arch         : ('64bit', '')
   ------------Pip Info-----------
   Version      : 10.0.1
   Directory    : /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pip
   ----------MXNet Info-----------
   Version      : 1.3.1
   Directory    : /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet
   Commit Hash   : 19c501680183237d52a862e6ae1dc4ddc296305b
   ----------System Info----------
   Platform     : Linux-4.14.77-70.82.amzn1.x86_64-x86_64-with-glibc2.9
   system       : Linux
   node         : ip-172-16-72-155
   release      : 4.14.77-70.82.amzn1.x86_64
   version      : #1 SMP Mon Dec 3 20:01:27 UTC 2018
   ----------Hardware Info----------
   machine      : x86_64
   processor    : x86_64
   Architecture:          x86_64
   CPU op-mode(s):        32-bit, 64-bit
   Byte Order:            Little Endian
   CPU(s):                8
   On-line CPU(s) list:   0-7
   Thread(s) per core:    2
   Core(s) per socket:    4
   Socket(s):             1
   NUMA node(s):          1
   Vendor ID:             GenuineIntel
   CPU family:            6
   Model:                 79
   Model name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
   Stepping:              1
   CPU MHz:               2700.116
   BogoMIPS:              4600.14
   Hypervisor vendor:     Xen
   Virtualization type:   full
   L1d cache:             32K
   L1i cache:             32K
   L2 cache:              256K
   L3 cache:              46080K
   NUMA node0 CPU(s):     0-7
   ----------Network Test----------
   Setting timeout: 10
   Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0022 sec, LOAD: 0.6019 sec.
   Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1398 sec, LOAD: 0.1484 sec.
   Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.4542 sec, LOAD: 0.2610 sec.
   Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0148 sec, LOAD: 0.0896 sec.
   Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0104 sec, LOAD: 0.4013 sec.
   Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0102 sec, LOAD: 0.0597 sec.
   ```
   
   Package used (Python/R/Scala/Julia): Python
   
   ## Error Message:
   ```
   Storage type fallback detected:
   operator = _backward_take
   input storage types = [default, default, ]
   output storage types = [row_sparse, default, ]
   params = {}
   context.dev_mask = gpu
   The operator with default storage type will be dispatched for execution. You're seeing this warning message because the operator above is unable to process the given ndarrays with specified storage types, context and parameter. Temporary dense ndarrays are generated in order to execute the operator. This does not affect the correctness of the programme. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.
   ```
   
   ## Minimum reproducible example
   This is the original approach that triggers the warning
   ```
   import mxnet as mx
   from mxnet import gluon
   
   class Model(gluon.HybridBlock):
       def __init__(self, **kwargs):
           super().__init__(**kwargs)
           with self.name_scope():
               self.weight = self.params.get('weight', shape=(100,2), grad_stype='row_sparse')
       
       def hybrid_forward(self, F, data, weight):
           return weight.take(data)
   
   model = Model()
   
   model.collect_params().initialize()
   trainer = gluon.Trainer(model.collect_params(), 'sgd')
   
   for _ in range(10):
       data = mx.nd.zeros((20,2))
       with mx.autograd.record():
           L = model(data)
       L.backward()
       trainer.step(1)
   ```
   This is the work-around that preserves `grad_stype='row_sparse'`:
   ```
   import mxnet as mx
   from mxnet import gluon
   
   model = gluon.nn.Embedding(100, 2, sparse_grad=True)
   
   model.collect_params().initialize()
   trainer = gluon.Trainer(model.collect_params(), 'sgd')
   
   for _ in range(10):
       data = mx.nd.zeros((20,2))
       with mx.autograd.record():
           L = model(data)
       L.backward()
       trainer.step(1)
   ```
   
   ## What have you tried to solve it?
   
   Use the alternative example in case row_sparse gradients are desired.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services