[GitHub] eric-haibin-lin commented on issue #9921: [DISCUSSION] module.contrib.SparseModule API

[GitBox <gi...@apache.org>]

eric-haibin-lin commented on issue #9921: [DISCUSSION] module.contrib.SparseModule API
URL: https://github.com/apache/incubator-mxnet/issues/9921#issuecomment-369513724
 
 
   Hi @anirudh2290 
   
   Thanks for the comments!!
   
   1. My intention for the following code is that pull/row_sparse_pull should happen when we see a new batch of data. Currently pull works for dense NDArray and row_sparse_pull works for row_sparse NDArray. A network might have both dense and sparse layers and we need to rsp_pull/pull both. 
   ```
   for batch in batches:
       kvstore.row_sparse_pull(sparse_weight, useful_rows_for_this_batch) 
       kvstore.pull(dense_weights)
       mod.forward(batch)
       mod.backward(batch) 
       mod.update()
           # kvstore.push(grad) 
   ```
   
   2. update_on_kvstore=False doesn't use kvstore to store the actual parameters, but under the hood it's used to aggregate and broadcast gradients. 
   
   3. In that case, you can have a function which doesn't use input data batch at all, such as 
   ```
   def f(batches, ...):
       return {'w': mx.nd.array([1,2,3])} 
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services