[GitHub] drivanov opened a new pull request #14154: Improving the run time of the tests which use assert_almost_equal OR check_numeric_gradient functions

[GitBox <gi...@apache.org>]

drivanov opened a new pull request #14154: Improving the run time of the tests which use assert_almost_equal OR check_numeric_gradient functions
URL: https://github.com/apache/incubator-mxnet/pull/14154
 
 
   ## Description ##
   - The analog of `numpy.allclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False)` is implemented as a MxNet operator:
   ```
       mx.nd.contrib.allclose(a, b, rtol, atol, equal_nan)
   ```
   - For now, besides the unit test `test_allclose_function()`, this method is used only in
   ```
   def assert_almost_equal(a, b, rtol=None, atol=None, names=('a', 'b'), equal_nan=False)
   ```
   where parameter **a** or/and **b** could be defined also as  `mx.nd.array`(s):
   ```
       Parameters
       ----------
       a : np.ndarray or mx.nd.array
       b : np.ndarray or mx.nd.array
   ```
   - When calling `assert_almost_equal`, no more `asnumpy()` conversion needed. It will be done automatically (in `assert_almost_equal`), if 
        (a) **a** or **b** has no attribute "context" OR 
        (b) these attributes are different.
   
   - The elimination of `asnumpy()` conversions and the use of `mx.nd.contrib.allclose(...)` for `mx.nd.array`'s allows to achieve 5-7x speedup for GPU tests that use long arrays.
   
   - The MxNet operator `mx.nd.contrib.approx_gradient(...)`, for calculation of **one** OR **all** coordinates of the numerical approximation of the gradient vector was implemented. This operator is called in the `numeric_grad(...)`, which is called by widely used `check_numeric_gradient(...)`
    
   - New parameter  `use_batch` which was added to
   ```
       def check_numeric_gradient(... use_batch=False):
   ```
   allows to calculate the approximation of gradient in a _batch mode_. 
   - Unfortunately our current implementation of _batch mode_ **DOES NOT** allow to use it for all operations. For instance, it is impossible to use it   
       (a) for operation, which are similar to matrix multiplications;  
       (b) when the shape of the output of the operation is NOT the same as the shapes of its input parameters;   
       (c) some other cases;   
   
   Currently we are using the _batch mode_ **only** when we call
   ```
   def test_elemwise_binary_ops():
           . . .
           if skip_gradient_check is not True:
               check_numeric_gradient(test, location,
                                      grad_stype_dict=grad_stypes, use_batch=True)
   ```
   - For _elemwise_ operations the _batch mode_ used in `check_numeric_gradient(...)` gives significant (3.5-5x on CPU and 6-10x on GPU) speedup. 
   
   - Even when we cannot use  _batch mode_, we still see the run time improvement associated with the usage of new MxNet operators:
   ```
        mx.nd.contrib.approx_gradient(...)
        mx.nd.contrib.allclose(...)
   ```
   There are two causes of this improvement:
       (a) now the number of `asnumpy()` conversions, used inside `check_numeric_gradient(...)`, is significantly less than before;
       (b) more calculations which are made on GPUs.  
   
   - In general, we see 1.58x improvement in runtime for the test suite `L0_self_test`:
   ```
   Now:      Ran 520 tests in  994.935s
   Before:   Ran 520 tests in 1574.341s 
   ```
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [x] Changes are complete (i.e. I finished coding on this PR)
   - [x] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
   - [x] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments are documented. 
   - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   
   ## Comments ##
   Perhaps, it makes sense:
   - to make possible to use (OR just start to use)  _batch mode_ for other operations;
   - eliminate `asnumpy()` conversions used when `assert_almost_equal` function is called.
   
   Unfortunately, it's not an easy task, because `assert_almost_equal` is used in many different places. But, it should make the Python code cleaner and some tests will run much faster, especially, those, where both parameters **a** and **b** are the  `mx.nd.array`'s
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services