[jira] [Created] (SINGA-498) Add Mish Activation Function

["Diganta Misra (Jira)" <ji...@apache.org>]

Diganta Misra created SINGA-498:
-----------------------------------

             Summary: Add Mish Activation Function
                 Key: SINGA-498
                 URL: https://issues.apache.org/jira/browse/SINGA-498
             Project: Singa
          Issue Type: New Feature
          Components: Core
            Reporter: Diganta Misra
         Attachments: se50_1.png

Mish is a new novel activation function proposed in this [paper](https://arxiv.org/abs/1908.08681).
It has shown promising results so far and has been adopted in several packages including:

- [TensorFlow-Addons](https://github.com/tensorflow/addons/tree/master/tensorflow_addons/activations)
- [SpaCy (Tok2Vec Layer)](https://github.com/explosion/spaCy)
- [Thinc - SpaCy's official NLP based ML library](https://github.com/explosion/thinc/releases/tag/v7.3.0)
- [Echo AI](https://github.com/digantamisra98/Echo)
- [Eclipse's deeplearning4j](https://github.com/eclipse/deeplearning4j/issues/8417)
- [CNTKX - Extension of Microsoft's CNTK](https://github.com/delzac/cntkx)
- [FastAI-Dev](https://github.com/fastai/fastai_dev/blob/0f613ba3205990c83de9dba0c8798a9eec5452ce/dev/local/layers.py#L441)
- [Darknet](https://github.com/AlexeyAB/darknet/commit/bf8ea4183dc265ac17f7c9d939dc815269f0a213)
- [Yolov3](https://github.com/ultralytics/yolov3/commit/444a9f7099d4ff1aef12783704e3df9a8c3aa4b3)
- [BeeDNN - Library in C++](https://github.com/edeforas/BeeDNN)
- [Gen-EfficientNet-PyTorch](https://github.com/rwightman/gen-efficientnet-pytorch)
- [dnet](https://github.com/umangjpatel/dnet/blob/master/nn/activations.py)

All benchmarks, analysis and links to official package implementations can be found in this [repository](https://github.com/digantamisra98/Mish)

It would be nice to have Mish as an option within the activation function group.

The attached image is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)