[GitHub] gkyh0805 opened a new issue #10519: Can too many connection for the symbol make unintended computation?

[GitBox <gi...@apache.org>]

gkyh0805 opened a new issue #10519: Can too many connection for the symbol make unintended computation?
URL: https://github.com/apache/incubator-mxnet/issues/10519
 
 
   
   when too many connection for the single symbol occur, can the some grads for relu_new_1 be skipped due to the MXNET_EXEC_INPLACE_GRAD_SUM_CAP?
   
   I saw "graph_executor.cc" and there is a comment
   [
           // Note: For a symbol like the following:
           // data = mx.sym.Variable('data')
           // sym = data + data + data + data + data + data + data
           // the node entries v passed in here are of the same node of
           // op _identity_with_attr_like_rhs. We should skip adding a node
           // to its own control_deps.
   ]
   
   I cannot understand the command perfectly, however, relu_new_1 in the following code is duplicated a lots and may have a dependency each other.
   
   Then, can the gradient be different according to MXNET_EXEC_INPLACE_GRAD_SUM_CAP?
   (the small connection accept all duplicate gradient and the many connection choose only one gradient among them?)
   
   In my case, the usage of the identity symbol for reducing the direct connection show the quite different training loss even though the seed is fixed.
   
   How does this code execute in mxnet through the graph executor?
   
   ```python
   conv_new_1 = mx.sym.Convolution(data=relu1, kernel=(1, 1), num_filter=1024, name="conv_new_1", lr_mult=3.0)
   relu_new_1 = mx.sym.Activation(data=conv_new_1, act_type='relu', name='relu_new_1')
   
   sym1 = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, name="sym1")
   sym2 = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, name="sym2")
   ...
   symn = mx.sym.Convolution(data=relu_new_1, kernel=(1, 1), num_filter=1024, name="symn")
   
   syma1 = relu_new_1 + sym1
   syma2 = relu_new_1 + sym2
   ...
   syman = relu_new_1 + symn
   
   final_out = syma1 + syma2 + ... + syman
   ```
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services