You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/06/17 00:41:02 UTC
[GitHub] casscw opened a new issue #10835: Lazy sparse param initialization
for distributed training
casscw opened a new issue #10835: Lazy sparse param initialization for distributed training
URL: https://github.com/apache/incubator-mxnet/issues/10835
## Description
In the model wide&deep，**a categorical feature 'sex' has three values '0,1,2'.** Then feed it into SparseEmbedding layer, just like the code below.
```
embed_weight = mx.symbol.Variable('single_2_embed_weight', stype='row_sparse')
mx.symbol.contrib.SparseEmbedding(data=embed, weight=embed_weight , input_dim=3, output_dim=100000)
```
## Environment info (Required)
Linux, mxnet-1.0.1,GPU,python-2.7
## Detail
By checking the model's params， **the 'single_2_embed_weight' has values in each row which is not sparse**
```
# data is the model's parmas [checkpoint-0000.params]
>>> data = mx.nd.load('model/checkpoint-0000.params')
>>> data['arg:single_2_embed_weight']
<RowSparseNDArray 100000x32 @cpu(0)>
>>> data['arg:single_2_embed_weight'].indices
[ 0 1 2 ... 99997 99998 99999]
<NDArray 100000 @cpu(0)>
>>> data['arg:single_2_embed_weight'].data
[[-0.00308852 0.00507648 -0.00339155 ... 0.00893246 0.00952273
0.00661108]
[ 0.00706272 0.00775839 -0.00312851 ... -0.00537404 0.0012673
-0.00746952]
[ 0.00495503 0.00214614 0.00021955 ... 0.00096997 0.00675734
-0.00131243]
...
[-0.00625316 -0.00615439 0.00471845 ... -0.00657647 -0.00150909
-0.00539171]
[-0.00908917 -0.00464959 -0.00893743 ... -0.00607294 -0.00969465
-0.00399319]
[-0.00455761 0.00734029 -0.00513125 ... 0.00258441 0.0042576
-0.00019451]]
<NDArray 100000x32 @cpu(0)>
```
**why the categorical input only has three values【or very few values】， it's embedding weight is dense【has values in each row which is not sparse】instead of 【few rows having values】?**
## Demo
https://gist.github.com/casscw/2e7a436704ead8804261f8b13e84f1a1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services