[MXNet Forum] [Gluon] How to apply F.softmax to Gluon Parameters

[Antonie Lin via MXNet Forum <mx...@discoursemail.com.INVALID>]

 I am trying to implement [task-specific weighting of multiple embeddings as in Elmo](https://arxiv.org/pdf/1802.05365.pdf).  
Currently, I initialized weights for multiple embeddings using `self.param.get`.  However, it throws me the error.
`AssertionError: Argument data must be Symbol instances, but got Parameter elmoembedding0_weights (shape=(3,), dtype=<class 'numpy.float32'>)`.

I can call `x.data()` for non hybridized or `x.var()` for hybridized version for the parameters.  Is there a way to simply apply Softmax to parameters and work with both versions?  Thanks!
  
My code looks something like this
```
import mxnet as mx
import mxnet.gluon as gluon

class ElmoEmbedding(gluon.HybridBlock):
    def __init__(self):
        super(ElmoEmbedding, self).__init__()

        with self.name_scope():
            self.weights = self.params.get('weights',
                                           shape=(3,),
                                           init=mx.init.Constant(1.0))
            self.scales = self.params.get('scales',
                                      shape=(1,0),
                                      init=mx.init.Constant(1.0))

    def hybrid_forward(self, F, x, *args, **kwargs):
        normalized_weights = F.softmax(self.weights)
        weighted_x = F.dot(normalized_weights, x)
        output = F.broadcast_mul(self.scales, weighted_x)
        return output

net = ElmoEmbedding()
net.hybridize()

# create input
x = mx.ndarray.random.randn(3,100)
output = net(x)
print("output", output.shape)
```





---
[Visit Topic](https://discuss.mxnet.apache.org/t/how-to-apply-f-softmax-to-gluon-parameters/6930/1) or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/97036f3ac6bc46f8d999282ce625d054f6fcc6ddb3d52b75e7e615d16c3d287d).

[MXNet Forum] [Gluon] How to apply F.softmax to Gluon Parameters

[Amir Ramezani via MXNet Forum <mx...@discoursemail.com.INVALID>]

hi,
initialize your network before calling net.hybridize() and see what happens





---
[Visit Topic](https://discuss.mxnet.apache.org/t/how-to-apply-f-softmax-to-gluon-parameters/6930/2) or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/538b94daa129343f10fe0f063caec1190eeb91b7efeafd8732f1d4ab15fe0c55).

[MXNet Forum] [Gluon] How to apply F.softmax to Gluon Parameters

[Antonie Lin via MXNet Forum <mx...@discoursemail.com.INVALID>]

`hybrid_forward` passes the parameters as part of the function arguments, so you would access them by defining them or using `kwargs`.  Then we can use them as usual for both hybridized and non-hybridized versions.

```
class ElmoEmbedding(gluon.HybridBlock):
    def __init__(self):
        ...
    def hybrid_forward(self, F, x, *args, **kwargs):
        weights = kwargs['weights']
        scales = kwargs['scales']
        normalized_weights = F.softmax(weights)
        ...     
```





---
[Visit Topic](https://discuss.mxnet.apache.org/t/how-to-apply-f-softmax-to-gluon-parameters/6930/3) or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/53ea94e87ad8805344f741777f41d4c406ca72079ba1973e7dbbd93a433b7ce7).