You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by wa...@apache.org on 2020/09/28 09:34:10 UTC
[singa-doc] 05/07: merge updates from upstream
This is an automated email from the ASF dual-hosted git repository.
wangwei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/singa-doc.git
commit 910946083c7f379261537fdeeeec447ec80110bc
Merge: 96bb9ef 0914fee
Author: wang wei <wa...@gmail.com>
AuthorDate: Thu Sep 24 10:41:42 2020 +0800
merge updates from upstream
docs-site/docs/contribute-code.md | 2 +-
docs-site/docs/dist-train.md | 99 +++++++++++++++++-----------
docs-site/docs/examples.md | 12 ++++
docs-site/docs/graph.md | 84 +++++++++++++-----------
docs-site/docs/onnx.md | 133 ++++++++++++++++++++------------------
docs-site/docs/team-list.md | 40 ++++++------
docs-site/docs/tensor.md | 128 ++++++++++++++++++++++++------------
7 files changed, 294 insertions(+), 204 deletions(-)
diff --cc docs-site/docs/dist-train.md
index c456440,b6cc0da..f242ffb
--- a/docs-site/docs/dist-train.md
+++ b/docs-site/docs/dist-train.md
@@@ -321,6 -342,14 +342,14 @@@ SINGA provides multiple optimization st
reduce the communication cost. Refer to the API for `DistOpt` for the
configuration of each strategy.
+ When we use `model.Model` to build a model, we need to put the options for
+ distributed training in the `train_one_batch` method. Please refer to the
+ example code on top of this page. We could just copy the code for the options
+ and use it in other models.
+
-With the defined options, we can put the arguments `dist_option` and `spars` when we
-start the training with `model(tx, ty, dist_option, spars)`
++With the defined options, we can put the arguments `dist_option` and `spars`
++when we start the training with `model(tx, ty, dist_option, spars)`
+
### No Optimizations
```python
diff --cc docs-site/docs/examples.md
index 421a9bb,c143563..53255f6
--- a/docs-site/docs/examples.md
+++ b/docs-site/docs/examples.md
@@@ -50,6 -50,18 +50,18 @@@ example is given below
| ---------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Bert-Squad | [SQuAD v1.1](https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/) | [Colab](https://colab.research.google.com/drive/1kud-lUPjS_u-TkDAzihBTw0Vqr0FjCE-) |
+ ## Text Classification
+
-| Model | Dataset | Links |
-| ----------- | ----------- | ---------- |
-| Simple LSTM | IMDB | [python]() |
++| Model | Dataset | Links |
++| ----------- | ------- | ---------- |
++| Simple LSTM | IMDB | [python]() |
+
+ ## Text Ranking
-| Model | Dataset | Links |
-| ----------- | ----------- | ---------- |
-| BiLSTM | InsuranceQA | [python]() |
+
++| Model | Dataset | Links |
++| ------ | ----------- | ---------- |
++| BiLSTM | InsuranceQA | [python]() |
+
## Misc.
- Restricted Boltzmann Machine over the MNIST dataset, [source](),
diff --cc docs-site/docs/graph.md
index a37d0e8,abce638..93347c4
--- a/docs-site/docs/graph.md
+++ b/docs-site/docs/graph.md
@@@ -126,14 -115,20 +124,22 @@@ clas
```python
class MLP(model.Model):
++<<<<<<< HEAD
++=======
+
+ def __init__(self, data_size=10, perceptron_size=100, num_classes=10):
+ super(MLP, self).__init__()
+ self.linear1 = layer.Linear(perceptron_size)
+ ...
++>>>>>>> upstream/master
def forward(self, inputs):
- x = autograd.matmul(inputs, self.w0)
+ y = self.linear1(inputs)
...
```
-The `Linear` layer is composed of the `mutmul` operator.
--`autograd` implements the `matmul` operator by calling the function `Mult`
--exposed from CPP via SWIG.
++The `Linear` layer is composed of the `mutmul` operator. `autograd` implements
++the `matmul` operator by calling the function `Mult` exposed from CPP via SWIG.
```python
# implementation of matmul()
diff --cc docs-site/docs/team-list.md
index 0746c88,680cb5b..41bd786
--- a/docs-site/docs/team-list.md
+++ b/docs-site/docs/team-list.md
@@@ -43,17 -46,14 +46,14 @@@ Zhejiang University, NetEase, Osaka Uni
## Contributors
--| Name | Email | Organization |
--| ------------------ | ---------------------------- | --------------------------------------------- |
--| Haibo Chen | hzchenhaibo@corp.netease.com | NetEase |
-| Shicheng Chen | chengsc@comp.nus.edu.sg | National University of Singapore |
--| Xin Ji | vincent.j.xin@gmail.com | Visenze, Singapore |
--| Anthony K. H. Tung | atung@comp.nus.edu.sg | National University of Singapore |
--| Ji Wang | wangji@mzhtechnologies.com | Hangzhou MZH Technologies |
--| Yuan Wang | wangyuan@corp.netease.com | NetEase |
--| Wenfeng Wu | dcswuw@gmail.com | Freelancer, China |
- | Chang Yao | yaochang2009@gmail.com | Hangzhou MZH Technologies |
- | Shicheng Chen | chengsc@comp.nus.edu.sg | National University of Singapore |
- | Joddiy Zhang | joddiyzhang@gmail.com | National University of Singapore |
- | Shicong Lin | dcslin@nus.edu.sg | National University of Singapore |
--| Kaiyuan Yang | yangky@comp.nus.edu.sg | National University of Singapore |
- | Rulin Xing | xjdkcsq3@gmail.com | Huazhong University of Science and Technology |
-| Chang Yao | yaochang2009@gmail.com | Hangzhou MZH Technologies |
++| Name | Email | Organization |
++| ------------------ | ---------------------------- | -------------------------------- |
++| Haibo Chen | hzchenhaibo@corp.netease.com | NetEase |
++| Shicheng Chen | chengsc@comp.nus.edu.sg | National University of Singapore |
++| Xin Ji | vincent.j.xin@gmail.com | Visenze, Singapore |
++| Anthony K. H. Tung | atung@comp.nus.edu.sg | National University of Singapore |
++| Ji Wang | wangji@mzhtechnologies.com | Hangzhou MZH Technologies |
++| Yuan Wang | wangyuan@corp.netease.com | NetEase |
++| Wenfeng Wu | dcswuw@gmail.com | Freelancer, China |
++| Kaiyuan Yang | yangky@comp.nus.edu.sg | National University of Singapore |
++| Chang Yao | yaochang2009@gmail.com | Hangzhou MZH Technologies |
diff --cc docs-site/docs/tensor.md
index 1889a0e,6e983b5..aad5366
--- a/docs-site/docs/tensor.md
+++ b/docs-site/docs/tensor.md
@@@ -45,6 -45,15 +45,16 @@@ array([[1., 0., 0.]
[0. 0.]]
```
+ `tensor` transformation up to 6 dims
-``` python
++
++```python
+ >>> a = tensor.random((2,3,4,5,6,7))
+ >>> a.shape
+ (2, 3, 4, 5, 6, 7)
+ >>> a.reshape((2,3,4,5,7,6)).transpose((3,2,1,0,4,5)).shape
+ (5, 4, 3, 2, 7, 6)
+ ```
+
### Tensor Arithmetic Methods
`tensor` is evaluated in real time.
@@@ -58,6 -67,37 +68,39 @@@
[0. 0.2 0. ]]
```
+ `tensor` broadcasting arithmetic:
-``` python
++
++```python
+ >>> a
+ [[1. 2. 3.]
+ [4. 5. 6.]]
+ >>> b
+ [[1. 2. 3.]]
+ >>> a + b
+ [[2. 4. 6.]
+ [5. 7. 9.]]
+ >>> a * b
+ [[ 1. 4. 9.]
+ [ 4. 10. 18.]]
+ >>> a / b
+ [[1. 1. 1. ]
+ [4. 2.5 2. ]]
+ >>> a/=b # inplace operation
+ >>> a
+ [[1. 1. 1. ]
+ [4. 2.5 2. ]]
+ ```
+
+ `tensor` broadcasting on matrix multiplication (GEMM)
-``` python
++
++```python
+ >>> from singa import tensor
+ >>> a = tensor.random((2,2,2,3))
+ >>> b = tensor.random((2,3,4))
+ >>> tensor.mult(a,b).shape
+ (2, 2, 2, 4)
+ ```
+
### Tensor Functions
Functions in module `singa.tensor` return new `tensor` object after applying the
@@@ -86,55 -126,55 +129,54 @@@ between `device`s via `to_device()` fun
>>> x.to_device(device.get_default_device())
```
- ### Simple Neural Network Example
-
+ ### use Tensor to train MLP
```python
- from singa import device
- from singa import tensor
- from singa import opt
- from singa import autograd
- class MLP:
- def __init__(self):
- self.linear1 = autograd.Linear(3, 4)
- self.linear2 = autograd.Linear(4, 5)
- def forward(self, x):
- y=self.linear1(x)
- return self.linear2(y)
- def train(model, x, t, dev, epochs=10):
- for i in range(epochs):
- y = model.forward(x)
- loss = autograd.mse_loss(y, t)
- print("loss: ", loss)
- sgd = opt.SGD()
- for p, gp in autograd.backward(loss):
- sgd.update(p, gp)
- sgd.step()
- print("training completed")
- if __name__ == "__main__":
- autograd.training = True
- model = MLP()
- dev = device.get_default_device()
- x = tensor.Tensor((2, 3), dev)
- t = tensor.Tensor((2, 5), dev)
- x.gaussian(1,1)
- t.gaussian(1,1)
- train(model, x, t, dev)
+
+ """
- code snipet from examples/mlp/module.py
++ code snipet from examples/mlp/module.py
+ """
+
+ label = get_label()
+ data = get_data()
+
+ dev = device.create_cuda_gpu_on(0)
+ sgd = opt.SGD(0.05)
+
+ # define tensor for input data and label
+ tx = tensor.Tensor((400, 2), dev, tensor.float32)
+ ty = tensor.Tensor((400,), dev, tensor.int32)
+ model = MLP(data_size=2, perceptron_size=3, num_classes=2)
+
+ # attached model to graph
+ model.set_optimizer(sgd)
+ model.compile([tx], is_train=True, use_graph=True, sequential=False)
+ model.train()
+
+ for i in range(1001):
+ tx.copy_from_numpy(data)
+ ty.copy_from_numpy(label)
+ out, loss = model(tx, ty, 'fp32', spars=None)
+
+ if i % 100 == 0:
+ print("training loss = ", tensor.to_numpy(loss)[0])
```
Output:
- ```
- loss: [4.917431]
- loss: [2.5147934]
- loss: [2.0670078]
- loss: [1.9179827]
- loss: [1.8192691]
- loss: [1.7269677]
- loss: [1.6308627]
- loss: [1.52674]
- loss: [1.4122975]
- loss: [1.2866782]
- training completed
-``` bash
-$ python3 examples/mlp/module.py
++```bash
++$ python3 examples/mlp/module.py
+ training loss = 0.6158037
+ training loss = 0.52852553
+ training loss = 0.4571422
+ training loss = 0.37274635
+ training loss = 0.30146334
+ training loss = 0.24906921
+ training loss = 0.21128304
+ training loss = 0.18390492
+ training loss = 0.16362564
+ training loss = 0.148164
+ training loss = 0.13589878
```
## Tensor Implementation