[GitHub] eric-haibin-lin opened a new pull request #9219: Add option to throw exception on storage fallback to help debugging

[GitBox <gi...@apache.org>]

eric-haibin-lin opened a new pull request #9219: Add option to throw exception on storage fallback to help debugging
URL: https://github.com/apache/incubator-mxnet/pull/9219
 
 
   ## Description ##
   #8902 
   ```
   ubuntu@ip-172-31-9-180:~/mxnet/python$ MXNET_STORAGE_FALLBACK_DEBUG=1 python
   Python 2.7.12 (default, Nov 20 2017, 18:23:56)
   [GCC 5.4.0 20160609] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import mxnet as mx
   >>> def func():
   ...     a = mx.nd.ones((2,2)).tostype('csr')
   ...     b = mx.nd.ones((2,2)).tostype('csr')
   ...     c = a * b
   ...     d = c + 1
   ...
   >>> func()
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "<stdin>", line 4, in func
     File "mxnet/ndarray/ndarray.py", line 235, in __mul__
       return multiply(self, other)
     File "mxnet/ndarray/ndarray.py", line 2566, in multiply
       None)
     File "mxnet/ndarray/ndarray.py", line 2379, in _ufunc_helper
       return fn_array(lhs, rhs)
     File "<string>", line 46, in broadcast_mul
     File "mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
       ctypes.byref(out_stypes)))
     File "mxnet/base.py", line 146, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError:
   Storage fallback detected:
   operator = broadcast_mul
   input storage types = [csr, csr, ]
   output storage types = [default, ]
   params = {}
   context.dev_mask = cpu
   The operator with default storage type will be dispatched for execution. You're seeing this warning message because the operator above is unable to process the given ndarrays with specified storage types, context and parameter. Temporary dense ndarrays are generated in order to execute the operator. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress the warnings. If you do not know what caused this error, you can try set environment variable MXNET_STORAGE_FALLBACK_DEBUG to 1. This will give you the series of calls that lead to this error. Remember to set MXNET_STORAGE_FALLBACK_DEBUG back to empty after debugging.
   
   ```
   
   ```
   ubuntu@ip-172-31-9-180:~/mxnet/python$ MXNET_STORAGE_FALLBACK_DEBUG=1 python
   Python 2.7.12 (default, Nov 20 2017, 18:23:56)
   [GCC 5.4.0 20160609] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import mxnet as mx
   >>> a = mx.nd.ones((2,2)).tostype('csr')
   >>> b = mx.nd.ones((2,2))
   >>> b.copyto(a)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "mxnet/ndarray/ndarray.py", line 1876, in copyto
       return _internal._copyto(self, out=other)
     File "<string>", line 25, in _copyto
     File "mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
       ctypes.byref(out_stypes)))
     File "mxnet/base.py", line 146, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError:
   Storage fallback detected:
   Copy from default storage type on cpu to csr storage type on cpu
   A temporary ndarray with csr storage type will be generated in order to perform the copy. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning. If you do not know what caused this error, you can try set environment variable MXNET_STORAGE_FALLBACK_DEBUG to 1. This will give you the series of calls that lead to this error. Remember to set MXNET_STORAGE_FALLBACK_DEBUG back to empty after debugging.
   ```
   
   
   ## Checklist ##
   ### Essentials ###
   - [ ] Passed code style checking (`make lint`)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments are documented. 
   - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be made.
   - Interesting edge cases to note here
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services