You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/07/02 00:14:35 UTC

[GitHub] eric-haibin-lin edited a comment on issue #11173: Flaky test: test_deformable_convolution_options

eric-haibin-lin edited a comment on issue #11173: Flaky test: test_deformable_convolution_options
URL: https://github.com/apache/incubator-mxnet/issues/11173#issuecomment-401642974
 
 
   I'm able to reproduce the err with the following example in test_operator_gpu.py:
   ```
   @with_seed()
   def test_deformable_convolution_options():
   
       ctx_list = [{'ctx': mx.gpu(0),
                    'deformable_conv_data': (2, 2, 7, 7),
                    'deformable_conv_offset': (2, 18, 3, 3),
                    'type_dict': {'deformable_conv_data': np.float64, 'deformable_conv_offset': np.float64}},
                   {'ctx': mx.gpu(0),
                    'deformable_conv_data': (2, 2, 7, 7),
                    'deformable_conv_offset': (2, 18, 3, 3),
                    'type_dict': {'deformable_conv_data': np.float32, 'deformable_conv_offset': np.float32}},
                   ]
       sym = mx.sym.contrib.DeformableConvolution(no_bias=True, num_filter=3, kernel=(3,3), stride=(2,2), name='deformable_conv')
   
       import random
       mx.random.seed(i)
       np.random.seed(i)
       random.seed(i)
       check_consistency(sym, ctx_list)
   ```
   
   ```
   FAIL: Failure: AssertionError (
   Items are not equal:
   Error 808.527832 exceeds tolerance rtol=0.001000, atol=0.001000.  Location of maximum error:(1, 13, 0, 2), a=-1.771303, b=-0.532353
    a: array([[[[-13.73718166,   1.72301412,  -5.12820005],
            [  0.        ,  18.53644371,   2.99046469],
            [  0.        ,   0.1865356 ,   2.20830846]],...
    b: array([[[[-13.73717976,   1.72301388,  -5.12820482],
            [  0.        ,  18.53644371,   2.99046636],
            [  0.        ,   0.18653677,   2.20830774]],...)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 418, in loadTestsFromName
       addr.filename, addr.module)
     File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 47, in importFromPath
       return self.importFromDir(dir_path, fqname)
     File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 94, in importFromDir
       mod = load_module(part_fqname, fh, filename, desc)
     File "/home/ubuntu/v1.2/tests/python/gpu/test_operator_gpu.py", line 1562, in <module>
       test_deformable_convolution_options()
     File "/home/ubuntu/v1.2/tests/python/gpu/../unittest/common.py", line 157, in test_new
       orig_test(*args, **kwargs)
     File "/home/ubuntu/v1.2/tests/python/gpu/test_operator_gpu.py", line 1522, in test_deformable_convolution_options
       check_consistency(sym, ctx_list)
     File "/home/ubuntu/v1.2/python/mxnet/test_utils.py", line 1354, in check_consistency
       raise e
   AssertionError:
   Items are not equal:
   Error 808.527832 exceeds tolerance rtol=0.001000, atol=0.001000.  Location of maximum error:(1, 13, 0, 2), a=-1.771303, b=-0.532353
    a: array([[[[-13.73718166,   1.72301412,  -5.12820005],
            [  0.        ,  18.53644371,   2.99046469],
            [  0.        ,   0.1865356 ,   2.20830846]],...
    b: array([[[[-13.73717976,   1.72301388,  -5.12820482],
            [  0.        ,  18.53644371,   2.99046636],
            [  0.        ,   0.18653677,   2.20830774]],...
   -------------------- >> begin captured logging << --------------------
   ```
   It looks like the the gradient for deformable_conv_offset doesn't pass consistency check. 
   
   I can reproduce it with the following cmd consistently:
   ```
   nosetests-2.7 tests/python/gpu/test_operator_gpu.py:test_deformable_convolution_options
   ```
   @YuwenXiong have you seen this before? Any idea why this happens? With some other seeds, I sometimes see a is positive while b is negative. I'm wonder if any code in the cuda kernel is disregarding dtype of the inputs. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services