[GitHub] rahul003 opened a new pull request #8207: Fix segfaults in and tests for distributed kvstore

[gi...@git.apache.org]

rahul003 opened a new pull request #8207: Fix segfaults in and tests for distributed kvstore
URL: https://github.com/apache/incubator-mxnet/pull/8207
 
 
   ### Description ###
   Fixes segfaults in kvstore_dist for row sparse, and fixes tests in dist_sync_kvstore
   @eric-haibin-lin and I worked on this.
   
   ### Changes ###
   1. Move copying of indices to happen before ZPull, because after ZPull we release the locks and the later function don't see actual variable
   2. nullptr was not handled for offsets variable
   3. fixes tests which were broken when WaitToWrite was removed in https://github.com/apache/incubator-mxnet/pull/8116 . Also added comments describing how they are no guarantees for (immediate) consecutive pushes
   4. Removed the usage of reference in BroadcastRowSparse to keep it consistent with style of Broadcast
   
   ### Comments ###
   When pushes happen one after the other, without a pull in between, there is no ordering guarantee we provide now that the `WaitToWrite` was removed in #8116  for performance reasons. This is because the assignment of `send_buf = merged` may happen simultaneously in two pushes. The second push may change this send_buf to its `merged` before the 1st push could do finish the push to kvstore. This won't affect us in actual training because the flow is like this then: we push gradients to kvstore, pull them back, update gradient, then push back. We don't do consecutive pushes in that case. The incorporation of pull in between serializes these pushes, pull has to happen after 1st push, and 2nd push has to happen after pull. This guarantee is because we use the same comm_buf for push and pull. 
   
   Please comment if we missed something, or if you have any suggestions, or if any clarifications are needed. 
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services