[GitHub] chinakook opened a new issue #10927: Feature Request: LSTM/GRU hard_sigmoid support

[GitBox <gi...@apache.org>]

chinakook opened a new issue #10927: Feature Request: LSTM/GRU hard_sigmoid support
URL: https://github.com/apache/incubator-mxnet/issues/10927
 
 
   code from rnn_cell.py
   ```python
           i2h = symbol.FullyConnected(data=inputs, weight=self._iW, bias=self._iB,
                                       num_hidden=self._num_hidden*4,
                                       name='%si2h'%name)
           h2h = symbol.FullyConnected(data=states[0], weight=self._hW, bias=self._hB,
                                       num_hidden=self._num_hidden*4,
                                       name='%sh2h'%name)
           gates = i2h + h2h
           slice_gates = symbol.SliceChannel(gates, num_outputs=4,
                                             name="%sslice"%name)
           in_gate = symbol.Activation(slice_gates[0], act_type="sigmoid",
                                       name='%si'%name)
           forget_gate = symbol.Activation(slice_gates[1], act_type="sigmoid",
                                           name='%sf'%name)
           in_transform = symbol.Activation(slice_gates[2], act_type="tanh",
                                            name='%sc'%name)
           out_gate = symbol.Activation(slice_gates[3], act_type="sigmoid",
                                        name='%so'%name)
           next_c = symbol._internal._plus(forget_gate * states[1], in_gate * in_transform,
                                           name='%sstate'%name)
           next_h = symbol._internal._mul(out_gate, symbol.Activation(next_c, act_type="tanh"),
                                          name='%sout'%name)
   ```
   I think we can offer a option to set all the recurrent activations(in_gate/forget_gate/out_gate) to 'sigmoid' or to 'hard_sigmoid'(**mx.symbol.hard_sigmoid**). "hard_sigmoid" is originated from Theano and is the default LSTM/GRU recurrent activation of Keras.
   As I've tested, 'hard_sigmoid' activation can converge faster than 'sigmoid' and get better result than 'sigmoid' in same training epochs. It's also more power efficient in inferencing.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services