[GitHub] [incubator-mxnet] sxjscience commented on issue #17989: [Gradient Addto] Very serious bug of grad_req='add'

[GitBox <gi...@apache.org>]

sxjscience commented on issue #17989: [Gradient Addto] Very serious bug of grad_req='add'
URL: https://github.com/apache/incubator-mxnet/issues/17989#issuecomment-610206398
 
 
   Also, a deeper dive into the problem shows that the issue appears when one layer is reused for >=5 times:
   ```bash
   wget https://gist.githubusercontent.com/sxjscience/0bd336c921396b3c66331354e1866886/raw/80a428980fd91110455e847c1a02aef4ae2cba7f/grad_req_addto_bug.py -O grad_req_addto_bug.py
   for nrepeat in 1 2 3 4 5 6 7 8 9 10
   do
       echo "nrepeat=${nrepeat}"
       echo "with addto"
       python grad_req_addto_bug.py --addto --nrepeat ${nrepeat}
       echo "without addto"
       python grad_req_addto_bug.py --nrepeat ${nrepeat}
   done
   ```
   Result:
   ```
   nrepeat=1
   with addto
   foo0_dense0_weight 86.363464
   foo0_dense0_bias 19.548544
   without addto
   foo0_dense0_weight 86.363464
   foo0_dense0_bias 19.548544
   nrepeat=2
   with addto
   foo0_dense0_weight 21.480562
   foo0_dense0_bias 19.870453
   without addto
   foo0_dense0_weight 21.480562
   foo0_dense0_bias 19.870453
   nrepeat=3
   with addto
   foo0_dense0_weight 4.64952
   foo0_dense0_bias 19.938385
   without addto
   foo0_dense0_weight 4.64952
   foo0_dense0_bias 19.938385
   nrepeat=4
   with addto
   foo0_dense0_weight 0.94337225
   foo0_dense0_bias 19.947392
   without addto
   foo0_dense0_weight 0.94337225
   foo0_dense0_bias 19.947392
   nrepeat=5
   with addto
   foo0_dense0_weight 1.7802463
   foo0_dense0_bias 315.05945
   without addto
   foo0_dense0_weight 0.19310227
   foo0_dense0_bias 19.947094
   nrepeat=6
   with addto
   foo0_dense0_weight 0.6738244
   foo0_dense0_bias 630.11865
   without addto
   foo0_dense0_weight 0.041728128
   foo0_dense0_bias 19.946844
   nrepeat=7
   with addto
   foo0_dense0_weight 0.26325437
   foo0_dense0_bias 1260.2372
   without addto
   foo0_dense0_weight 0.009131842
   foo0_dense0_bias 19.946758
   nrepeat=8
   with addto
   foo0_dense0_weight 0.0020059107
   foo0_dense0_bias 19.946749
   without addto
   foo0_dense0_weight 0.0020059107
   foo0_dense0_bias 19.946749
   nrepeat=9
   with addto
   foo0_dense0_weight 0.00045126013
   foo0_dense0_bias 19.946749
   without addto
   foo0_dense0_weight 0.00045126013
   foo0_dense0_bias 19.946749
   nrepeat=10
   with addto
   foo0_dense0_weight 0.00010413639
   foo0_dense0_bias 19.946749
   without addto
   foo0_dense0_weight 0.00010413639
   foo0_dense0_bias 19.946749
   ```
   
   This shows that it's only wrong when `nrepeat` is 5,6,7

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services