You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by wa...@apache.org on 2015/09/19 05:52:33 UTC
svn commit: r1703939 - in /incubator/singa/site/trunk/content:
markdown/docs/ resources/images/
Author: wangwei
Date: Sat Sep 19 03:52:32 2015
New Revision: 1703939
URL: http://svn.apache.org/viewvc?rev=1703939&view=rev
Log:
update RBM example and CNN example pages.
Added:
incubator/singa/site/trunk/content/resources/images/example-autoencoder.png (with props)
incubator/singa/site/trunk/content/resources/images/example-cnn.png (with props)
incubator/singa/site/trunk/content/resources/images/example-mlp.png (with props)
incubator/singa/site/trunk/content/resources/images/example-rbm1.png (with props)
incubator/singa/site/trunk/content/resources/images/example-rbm2.png (with props)
incubator/singa/site/trunk/content/resources/images/example-rbm3.png (with props)
incubator/singa/site/trunk/content/resources/images/example-rbm4.png (with props)
Removed:
incubator/singa/site/trunk/content/resources/images/RBM0.PNG
incubator/singa/site/trunk/content/resources/images/RBM0_new.PNG
incubator/singa/site/trunk/content/resources/images/RBM1.PNG
incubator/singa/site/trunk/content/resources/images/RBM1_new.PNG
incubator/singa/site/trunk/content/resources/images/RBM2_new.PNG
incubator/singa/site/trunk/content/resources/images/RBM3_new.PNG
incubator/singa/site/trunk/content/resources/images/cnn-example.png
incubator/singa/site/trunk/content/resources/images/mlp-example.png
Modified:
incubator/singa/site/trunk/content/markdown/docs/cnn.md
incubator/singa/site/trunk/content/markdown/docs/debug.md
incubator/singa/site/trunk/content/markdown/docs/layer.md
incubator/singa/site/trunk/content/markdown/docs/mlp.md
incubator/singa/site/trunk/content/markdown/docs/rbm.md
Modified: incubator/singa/site/trunk/content/markdown/docs/cnn.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/docs/cnn.md?rev=1703939&r1=1703938&r2=1703939&view=diff
==============================================================================
--- incubator/singa/site/trunk/content/markdown/docs/cnn.md (original)
+++ incubator/singa/site/trunk/content/markdown/docs/cnn.md Sat Sep 19 03:52:32 2015
@@ -10,8 +10,8 @@ use a deep CNN model to do image classif
## Running instructions
-Please refer to the [installation](http://singa.incubator.apache.org/docs/installation) page for
-instructions on building SINGA, and the [quick start](http://singa.incubator.apache.org/docs/quick-start)
+Please refer to the [installation](installation.html) page for
+instructions on building SINGA, and the [quick start](quick-start.html)
for instructions on starting zookeeper.
We have provided scripts for preparing the training and test dataset in *examples/cifar10/*.
@@ -22,11 +22,11 @@ We have provided scripts for preparing t
$ make create
-After the datasets are prepared, we start the training by
+We can start the training by
./bin/singa-run.sh -conf examples/cifar10/job.conf
-After it is started, you should see output like
+You should see output like
Record job information to /tmp/singa-log/job-info/job-2-20150817-055601
Executing : ./singa -conf /xxx/incubator-singa/examples/cifar10/job.conf -singa_conf /xxx/incubator-singa/conf/singa.conf -singa_job 2
@@ -44,7 +44,7 @@ After it is started, you should see outp
E0817 06:58:05.578366 33849 trainer.cc:373] Train step-210, loss : 2.300143, accuracy : 0.154167
E0817 06:58:12.518497 33849 trainer.cc:373] Train step-240, loss : 2.295912, accuracy : 0.185417
-After the training of some steps (depends on the setting) or the job is
+After training some steps (depends on the setting) or the job is
finished, SINGA will [checkpoint](http://singa.incubator.apache.org/docs/checkpoint) the model parameters.
## Details
@@ -55,205 +55,168 @@ algorithm (BP or CD), SGD update algorit
number of training/test steps, etc.
### Data preparation
-Before using SINGA, you need to write a program to pre-process the dataset you
-use to a format that SINGA can read. Please refer to the
-[Data Preparation](http://singa.incubator.apache.org/docs/data#example---cifar-dataset) to get details about preparing this CIFAR10 dataset.
+
+Before using SINGA, you need to write a program to convert the dataset
+into a format that SINGA can read. Please refer to the
+[Data Preparation](data.html#example---cifar-dataset) to get details about
+preparing this CIFAR10 dataset.
### Neural net
Figure 1 shows the net structure of the CNN model we used in this example, which is
-set following [this page](https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg.)
+set following [Alex](https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg.)
The dashed circle represents one feature transformation stage, which generally
has four layers as shown in the figure. Sometimes the rectifier layer and normalization layer
-is omitted or swapped in one stage. For this example, there are 3 such stages.
+are omitted or swapped in one stage. For this example, there are 3 such stages.
-Next we follow the guide in [neural net page](http://singa.incubator.apache.org/docs/neural-net)
-and [layer page](http://singa.incubator.apache.org/docs/layer) to write the neural net configuration.
+Next we follow the guide in [neural net page](neural-net.html)
+and [layer page](layer.html) to write the neural net configuration.
<div style = "text-align: center">
-<img src = "http://singa.incubator.apache.org/images/cnn-example.png" style = "width: 200px"> <br/>
+<img src = "../images/example-cnn.png" style = "width: 200px"> <br/>
<strong>Figure 1 - Net structure of the CNN example.</strong></img>
</div>
-* We configure a [data layer](http://singa.incubator.apache.org/docs/layer#data-layers) to read
+* We configure a [data layer](layer.html#data-layers) to read
the training/testing `Records` from `DataShard`.
layer{
- name: "data"
- type: kShardData
- sharddata_conf {
- path: "examples/cifar10/cifar10_train_shard"
- batchsize: 16
- random_skip: 5000
- }
- exclude: kTest # exclude this layer for the testing net
+ name: "data"
+ type: kShardData
+ sharddata_conf {
+ path: "examples/cifar10/cifar10_train_shard"
+ batchsize: 16
+ random_skip: 5000
}
+ exclude: kTest # exclude this layer for the testing net
+ }
layer{
- name: "data"
- type: kShardData
- sharddata_conf {
- path: "examples/cifar10/cifar10_test_shard"
- batchsize: 100
- }
- exclude: kTrain # exclude this layer for the training net
+ name: "data"
+ type: kShardData
+ sharddata_conf {
+ path: "examples/cifar10/cifar10_test_shard"
+ batchsize: 100
}
+ exclude: kTrain # exclude this layer for the training net
+ }
-* We configure two [parser layers](http://singa.incubator.apache.org/docs/layer#parser-layers)
+* We configure two [parser layers](layer.html#parser-layers)
to extract the image feature and label from `Records`s loaded by the *data* layer.
layer{
- name:"rgb"
- type: kRGBImage
- srclayers: "data"
- rgbimage_conf {
- meanfile: "examples/cifar10/image_mean.bin" # normalize image feature
- }
- }
+ name:"rgb"
+ type: kRGBImage
+ srclayers: "data"
+ rgbimage_conf {...}
+ }
layer{
- name: "label"
- type: kLabel
- srclayers: "data"
- }
-
+ name: "label"
+ type: kLabel
+ srclayers: "data"
+ }
* We configure layers for the feature transformation as follows
(all layers are built-in layers in SINGA; hyper-parameters of these layers are set according to
[Alex's setting](https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg)).
layer {
- name: "conv1"
- type: kConvolution
- srclayers: "rgb"
- convolution_conf {
- num_filters: 32
- kernel: 5
- stride: 1
- pad:2
- }
- param {
- name: "w1"
- init {
- type:kGaussian
- std:0.0001
- }
- }
- param {
- name: "b1"
- lr_scale:2.0
- init {
- type: kConstant
- value:0
- }
- }
- }
-
- layer {
- name: "pool1"
- type: kPooling
- srclayers: "conv1"
- pooling_conf {
- pool: MAX
- kernel: 3
- stride: 2
- }
- }
- layer {
- name: "relu1"
- type: kReLU
- srclayers:"pool1"
- }
- layer {
- name: "norm1"
- type: kLRN
- lrn_conf {
- local_size: 3
- alpha: 5e-05
- beta: 0.75
- }
- srclayers:"relu1"
- }
+ name: "conv1"
+ type: kConvolution
+ srclayers: "rgb"
+ convolution_conf {... }
+ ...
+ }
+ layer {
+ name: "pool1"
+ type: kPooling
+ srclayers: "conv1"
+ pooling_conf {... }
+ }
+ layer {
+ name: "relu1"
+ type: kReLU
+ srclayers:"pool1"
+ }
+ layer {
+ name: "norm1"
+ type: kLRN
+ lrn_conf {... }
+ srclayers:"relu1"
+ }
The configurations for another 2 stages are omitted here.
-* There is a [inner product layer](http://singa.incubator.apache.org/docs/layer#innerproductlayer)
+* There is an [inner product layer](layer.html#innerproductlayer)
after the 3 transformation stages, which is
configured with 10 output units, i.e., the number of total labels. The weight
-matrix param is configured with a large weight decay scale to reduce the over-fitting.
+matrix Param is configured with a large weight decay scale to reduce the over-fitting.
layer {
- name: "ip1"
- type: kInnerProduct
- srclayers:"pool3"
- innerproduct_conf {
- num_output: 10
- }
- param {
- name: "w4"
- wd_scale:250
- init {
- type:kGaussian
- std:0.01
- }
- }
- param {
- name: "b4"
- lr_scale:2.0
- wd_scale:0
- init {
- type: kConstant
- value:0
- }
- }
+ name: "ip1"
+ type: kInnerProduct
+ srclayers:"pool3"
+ innerproduct_conf {
+ num_output: 10
+ }
+ param {
+ name: "w4"
+ wd_scale:250
+ ...
+ }
+ param {
+ name: "b4"
+ ...
}
+ }
-* The last layer is a [Softmax loss layer](http://singa.incubator.apache.org/docs/layer#softmaxloss)
-
- layer{
- name: "loss"
- type: kSoftmaxLoss
- softmaxloss_conf{
- topk:1
- }
- srclayers:"ip1"
- srclayers: "label"
- }
+* The last layer is a [Softmax loss layer](layer.html#softmaxloss)
+ layer{
+ name: "loss"
+ type: kSoftmaxLoss
+ softmaxloss_conf{ topk:1 }
+ srclayers:"ip1"
+ srclayers: "label"
+ }
### Updater
-The [normal SGD updater](http://singa.incubator.apache.org/docs/updater#updater) is selected.
-The learning rate is changed like stairs, and is configured using the
-[kFixedStep](http://singa.incubator.apache.org/docs/updater#kfixedstep) type.
-
- updater{
- type: kSGD
- weight_decay:0.004
- learning_rate {
- type: kFixedStep
- fixedstep_conf:{
- step:0 # lr for step 0-60000 is 0.001
- step:60000 # lr for step 60000-65000 is 0.0001
- step:65000 # lr for step 650000- is 0.00001
- step_lr:0.001
- step_lr:0.0001
- step_lr:0.00001
+
+The [normal SGD updater](updater.html#updater) is selected.
+The learning rate is changed like going down stairs, and is configured using the
+[kFixedStep](updater.html#kfixedstep) type.
+
+ updater{
+ type: kSGD
+ weight_decay:0.004
+ learning_rate {
+ type: kFixedStep
+ fixedstep_conf:{
+ step:0 # lr for step 0-60000 is 0.001
+ step:60000 # lr for step 60000-65000 is 0.0001
+ step:65000 # lr for step 650000- is 0.00001
+ step_lr:0.001
+ step_lr:0.0001
+ step_lr:0.00001
+ }
+ }
}
- }
- }
### TrainOneBatch algorithm
+
The CNN model is a feed forward model, thus should be configured to use the
-[Back-propagation algorithm]({{ BASE_PATH}}/docs/train-one-batch#back-propagation).
+[Back-propagation algorithm](train-one-batch.html#back-propagation).
- alg: kBP
+ train_one_batch {
+ alg: kBP
+ }
### Cluster setting
+
The following configuration set a single worker and server for training.
-[Training frameworks](http://singa.incubator.apache.org/docs/frameworks) page introduces configurations of a couple of distributed
+[Training frameworks](frameworks.html) page introduces configurations of a couple of distributed
training frameworks.
cluster {
nworker_groups: 1
nserver_groups: 1
}
-
-
Modified: incubator/singa/site/trunk/content/markdown/docs/debug.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/docs/debug.md?rev=1703939&r1=1703938&r2=1703939&view=diff
==============================================================================
--- incubator/singa/site/trunk/content/markdown/docs/debug.md (original)
+++ incubator/singa/site/trunk/content/markdown/docs/debug.md Sat Sep 19 03:52:32 2015
@@ -1,5 +1,4 @@
# How to Debug
-
___
Since SINGA is developed on Linux using C++, GDB is the preferred debugging
@@ -8,6 +7,10 @@ tool. To use GDB, the code must be compi
./configure --enable-debug
make
+## Debugging for single process job
+
+If your job launches only one process, then use the default *conf/singa.conf*
+for debugging. The process will be launched locally.
To debug, first start zookeeper if it is not started yet, and launch GDB
@@ -21,3 +24,5 @@ Then set the command line arguments
set args -conf JOBCONF
Now you can set your breakpoints and start running.
+
+## Debugging for jobs with multiple processes
Modified: incubator/singa/site/trunk/content/markdown/docs/layer.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/docs/layer.md?rev=1703939&r1=1703938&r2=1703939&view=diff
==============================================================================
--- incubator/singa/site/trunk/content/markdown/docs/layer.md (original)
+++ incubator/singa/site/trunk/content/markdown/docs/layer.md Sat Sep 19 03:52:32 2015
@@ -255,9 +255,9 @@ Use the enum AVE and MAX to choose the m
[ReLuLayer](../api/classsinga_1_1ReLULayer.html) has rectified linear neurons, which conducts the following
transformation, `f(x) = Max(0, x)`. It has no specific configuration fields.
-##### TanhLayer
+##### STanhLayer
-[TanhLayer](../api/classsinga_1_1TanhLayer.html) uses the tanh as activation function, i.e., `f(x)=tanh(x)`.
+[STanhLayer](../api/classsinga_1_1TanhLayer.html) uses the scaled tanh as activation function, i.e., `f(x)=1.7159047* tanh(0.6666667 * x)`.
It has no specific configuration fields.
##### SigmoidLayer
Modified: incubator/singa/site/trunk/content/markdown/docs/mlp.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/docs/mlp.md?rev=1703939&r1=1703938&r2=1703939&view=diff
==============================================================================
--- incubator/singa/site/trunk/content/markdown/docs/mlp.md (original)
+++ incubator/singa/site/trunk/content/markdown/docs/mlp.md Sat Sep 19 03:52:32 2015
@@ -1,14 +1,17 @@
# MLP Example
-Multilayer perceptron (MLP) is a feed-forward artificial neural network model.
+---
+
+Multilayer perceptron (MLP) is a subclass of feed-forward neural networks.
A MLP typically consists of multiple directly connected layers, with each layer fully
connected to the next one. In this example, we will use SINGA to train a
[simple MLP model proposed by Ciresan](http://arxiv.org/abs/1003.0358)
for classifying handwritten digits from the [MNIST dataset](http://yann.lecun.com/exdb/mnist/).
## Running instructions
-Please refer to the [installation](http://singa.incubator.apache.org/docs/installation) page for
-instructions on building SINGA, and the [quick start](http://singa.incubator.apache.org/docs/quick-start)
+
+Please refer to the [installation](installation.html) page for
+instructions on building SINGA, and the [quick start](quick-start.html)
for instructions on starting zookeeper.
We have provided scripts for preparing the training and test dataset in *examples/cifar10/*.
@@ -46,7 +49,7 @@ After it is started, you should see outp
E0817 07:19:57.327133 34073 trainer.cc:373] Test step-120, loss : 1.216652, accuracy : 0.769900
After the training of some steps (depends on the setting) or the job is
-finished, SINGA will [checkpoint](http://singa.incubator.apache.org/docs/checkpoint) the model parameters.
+finished, SINGA will [checkpoint](checkpoint.html) the model parameters.
## Details
@@ -57,16 +60,17 @@ algorithm (BP or CD), SGD update algorit
number of training/test steps, etc.
### Data preparation
+
Before using SINGA, you need to write a program to pre-process the dataset you
use to a format that SINGA can read. Please refer to the
-[Data Preparation](http://singa.incubator.apache.org/docs/data#example---mnist-dataset) to get details about preparing
+[Data Preparation](data.html#example---mnist-dataset) to get details about preparing
this MNIST dataset.
### Neural net
<div style = "text-align: center">
-<img src = "http://singa.incubator.apache.org/images/mlp-example.png" style = "width: 230px">
+<img src = "../images/example-mlp.png" style = "width: 230px">
<br/><strong>Figure 1 - Net structure of the MLP example. </strong></img>
</div>
@@ -74,13 +78,13 @@ this MNIST dataset.
Figure 1 shows the structure of the simple MLP model, which is constructed following
[Ciresan's paper](http://arxiv.org/abs/1003.0358). The dashed circle contains
two layers which represent one feature transformation stage. There are 6 such
-stages in total. They sizes of the [InnerProductLayer](http://singa.incubator.apache.org/docs/layer#innerproductlayer)s in these circles decrease from
+stages in total. They sizes of the [InnerProductLayer](layer.html#innerproductlayer)s in these circles decrease from
2500->2000->1500->1000->500->10.
-Next we follow the guide in [neural net page](http://singa.incubator.apache.org/docs/neural-net)
-and [layer page](http://singa.incubator.apache.org/docs/layer) to write the neural net configuration.
+Next we follow the guide in [neural net page](neural-net.html)
+and [layer page](layer.html) to write the neural net configuration.
-* We configure a [data layer](http://singa.incubator.apache.org/docs/layer#data-layers) to read
+* We configure a [data layer](layer.html#data-layers) to read
the training/testing `Records` from `DataShard`.
layer {
@@ -103,9 +107,9 @@ the training/testing `Records` from `Dat
exclude: kTrain
}
-* We configure two [parser layers](http://singa.incubator.apache.org/docs/layer#parser-layers)
+* We configure two [parser layers](layer.html#parser-layers)
to extract the image feature and label from `Records`s loaded by the *data* layer.
-The [MnistLayer](http://singa.incubator.apache.org/docs/layer#mnistlayer) will normalize the pixel
+The [MnistLayer](layer.html#mnistlayer) will normalize the pixel
values into [-1,1].
layer{
@@ -124,7 +128,7 @@ values into [-1,1].
srclayers: "data"
}
-* All [InnerProductLayer](http://singa.incubator.apache.org/docs/layer#innerproductlayer)s are configured similarly as,
+* All [InnerProductLayer](layer.html#innerproductlayer)s are configured similarly as,
layer{
name: "fc1"
@@ -135,53 +139,39 @@ values into [-1,1].
}
param{
name: "w1"
- init {
- type: kUniform
- low:-0.05
- high:0.05
- }
+ ...
}
param{
name: "b1"
- init {
- type : kUniform
- low: -0.05
- high:0.05
- }
+ ..
}
}
with the `num_output` decreasing from 2500 to 10.
-* All [TanhLayer](http://singa.incubator.apache.org/docs/layer#tanhlayer) are configured similarly as,
+* A [STanhLayer](layer.html#stanhlayer) is connected to every InnerProductLayer
+except the last one. It transforms the feature via scaled tanh function.
layer{
name: "tanh1"
- type: kTanh
- tanh_conf {
- outer_scale: 1.7159047
- inner_scale: 0.6666667
- }
+ type: kSTanh
srclayers:"fc1"
}
- every neuron from the source layer is transformed as `outer_scale*tanh(inner_scale* x)`.
-
-* The final [Softmax loss layer](http://singa.incubator.apache.org/docs/layer#softmaxloss) connects
-to LabelLayer and the last TanhLayer.
+* The final [Softmax loss layer](layer.html#softmaxloss) connects
+to LabelLayer and the last STanhLayer.
layer{
name: "loss"
type:kSoftmaxLoss
- softmaxloss_conf{
- topk:1
- }
+ softmaxloss_conf{ topk:1 }
srclayers:"fc6"
srclayers:"label"
}
### Updater
-The [normal SGD updater](http://singa.incubator.apache.org/docs/updater#updater) is selected.
+
+The [normal SGD updater](updater.html#updater) is selected.
The learning rate shrinks by 0.997 every 60 steps (i.e., one epoch).
updater{
@@ -202,12 +192,14 @@ The MLP model is a feed-forward model, h
[Back-propagation algorithm]({{ BASE_PATH}}/docs/train-one-batch#back-propagation)
is selected.
- alg: kBP
-
+ train_one_batch {
+ alg: kBP
+ }
### Cluster setting
+
The following configuration set a single worker and server for training.
-[Training frameworks](http://singa.incubator.apache.org/docs/frameworks) page introduces configurations of a couple of distributed
+[Training frameworks](frameworks.html) page introduces configurations of a couple of distributed
training frameworks.
cluster {
Modified: incubator/singa/site/trunk/content/markdown/docs/rbm.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/docs/rbm.md?rev=1703939&r1=1703938&r2=1703939&view=diff
==============================================================================
--- incubator/singa/site/trunk/content/markdown/docs/rbm.md (original)
+++ incubator/singa/site/trunk/content/markdown/docs/rbm.md Sat Sep 19 03:52:32 2015
@@ -27,10 +27,10 @@ The training is separated into two phase
The pre-training phase trains 4 RBMs in sequence,
# at SINGA_ROOT/
- $ ./bin/singa-run.sh -conf examples/rbm/rbm0.conf
$ ./bin/singa-run.sh -conf examples/rbm/rbm1.conf
$ ./bin/singa-run.sh -conf examples/rbm/rbm2.conf
$ ./bin/singa-run.sh -conf examples/rbm/rbm3.conf
+ $ ./bin/singa-run.sh -conf examples/rbm/rbm4.conf
The fine-tuning phase trains the auto-encoder by,
@@ -39,14 +39,14 @@ The fine-tuning phase trains the auto-en
## Training details
-### RBM0
+### RBM1
-<img src="../images/RBM0_new.PNG" align="center" width="200px"/>
-<span><strong>Figure 1 - RBM0.</strong></span>
+<img src="../images/example-rbm1.png" align="center" width="200px"/>
+<span><strong>Figure 1 - RBM1.</strong></span>
-The neural net structure for training RBM0 is shown in Figure 1.
-The data layer and parser layer provides features for training RBM0.
-The visible layer (connected with parser layer) of RBM0 accepts the image feature
+The neural net structure for training RBM1 is shown in Figure 1.
+The data layer and parser layer provides features for training RBM1.
+The visible layer (connected with parser layer) of RBM1 accepts the image feature
(784 dimension). The hidden layer is set to have 1000 neurons (units).
These two layers are configured as,
@@ -59,7 +59,7 @@ These two layers are configured as,
hdim: 1000
}
param{
- name: "w0"
+ name: "w1"
init{
type: kGaussian
mean: 0.0
@@ -67,7 +67,7 @@ These two layers are configured as,
}
}
param{
- name: "b01"
+ name: "b11"
init{
type: kConstant
value: 0.0
@@ -83,11 +83,11 @@ These two layers are configured as,
hdim: 1000
}
param{
- name: "w0_"
- share_from: "w0"
+ name: "w1_"
+ share_from: "w1"
}
param{
- name: "b02"
+ name: "b12"
init{
type: kConstant
value: 0.0
@@ -98,9 +98,9 @@ These two layers are configured as,
For RBM, the weight matrix is shared by the visible and hidden layers. For instance,
-`w0` is shared by `vis` and `hid` layers shown in Figure 1. In SINGA, we can configure
+`w1` is shared by `vis` and `hid` layers shown in Figure 1. In SINGA, we can configure
the `share_from` field to enable [parameter sharing](param.html)
-as shown above for the param `w0` and `w0_`.
+as shown above for the param `w1` and `w1_`.
[Contrastive Divergence](train-one-batch.html#contrastive-divergence)
is configured as the algorithm for [TrainOneBatch](train-one-batch.html).
@@ -123,19 +123,19 @@ configure the `workspace` field to speci
For example, if we configure it as,
cluster {
- workspace: "examples/rbm/rbm0/"
+ workspace: "examples/rbm/rbm1/"
}
-Then SINGA will [checkpoint the parameters](checkpoint.html) into *examples/rbm/rbm0/*.
+Then SINGA will [checkpoint the parameters](checkpoint.html) into *examples/rbm/rbm1/*.
### RBM1
-<img src="../images/RBM1_new.PNG" align="center" width="200px"/>
-<span><strong>Figure 2 - RBM1.</strong></span>
+<img src="../images/example-rbm2.png" align="center" width="200px"/>
+<span><strong>Figure 2 - RBM2.</strong></span>
-Figure 2 shows the net structure of training RBM1.
-The visible units of RBM1 accept the output from the Sigmoid1 layer. The Inner1 layer
-is a `InnerProductLayer` whose parameters are set to the `w0` and `b1` learned
-from RBM0.
+Figure 2 shows the net structure of training RBM2.
+The visible units of RBM2 accept the output from the Sigmoid1 layer. The Inner1 layer
+is a `InnerProductLayer` whose parameters are set to the `w1` and `b12` learned
+from RBM1.
The neural net configuration is (with layers for data layer and parser layer omitted).
layer{
@@ -145,8 +145,8 @@ The neural net configuration is (with la
innerproduct_conf{
num_output: 1000
}
- param{ name: "w0" }
- param{ name: "b02"}
+ param{ name: "w1" }
+ param{ name: "b12"}
}
layer{
@@ -164,11 +164,11 @@ The neural net configuration is (with la
hdim: 500
}
param{
- name: "w1"
+ name: "w2"
...
}
param{
- name: "b11"
+ name: "b21"
...
}
}
@@ -181,50 +181,50 @@ The neural net configuration is (with la
hdim: 500
}
param{
- name: "w1_"
- share_from: "w1"
+ name: "w2_"
+ share_from: "w2"
}
param{
- name: "b12"
+ name: "b22"
...
}
}
To load w0 and b02 from RBM0's checkpoint file, we configure the `checkpoint_path` as,
- checkpoint_path: "examples/rbm/rbm0/checkpoint/step6000-worker0.bin"
+ checkpoint_path: "examples/rbm/rbm1/checkpoint/step6000-worker0.bin"
cluster{
- workspace: "examples/rbm/rbm1"
+ workspace: "examples/rbm/rbm2"
}
-The workspace is changed for checkpointing `w1`, `b11` and `b12` into
-*examples/rbm/rbm1/*.
+The workspace is changed for checkpointing `w2`, `b21` and `b22` into
+*examples/rbm/rbm2/*.
-### RBM2
+### RBM3
-<img src="../images/RBM2_new.PNG" align="center" width="200px"/>
-<span><strong>Figure 3 - RBM2.</strong></span>
+<img src="../images/example-rbm3.png" align="center" width="200px"/>
+<span><strong>Figure 3 - RBM3.</strong></span>
-Figure 3 shows the net structure of training RBM2. In this model, a layer with
-250 units is added as the hidden layer of RBM2. The visible units of RBM2
+Figure 3 shows the net structure of training RBM3. In this model, a layer with
+250 units is added as the hidden layer of RBM3. The visible units of RBM3
accepts output from Sigmoid2 layer. Parameters of Inner1 and Innner2 are set to
-`w0,b02,w1,b12` which can be load from the checkpoint file of RBM1,
-i.e., "examples/rbm/rbm1/".
+`w1,b12,w2,b22` which can be load from the checkpoint file of RBM2,
+i.e., "examples/rbm/rbm2/".
-### RBM3
+### RBM4
-<img src="../images/RBM3_new.PNG" align="center" width="200px"/>
-<span><strong>Figure 4 - RBM3.</strong></span>
+<img src="../images/example-rbm4.png" align="center" width="200px"/>
+<span><strong>Figure 4 - RBM4.</strong></span>
-Figure 4 shows the net structure of training RBM3. It is similar to Figure 3,
+Figure 4 shows the net structure of training RBM4. It is similar to Figure 3,
but according to [Hinton's science paper](http://www.cs.toronto.edu/~hinton/science.pdf), the hidden units of the
-top RBM (RBM3) have stochastic real-valued states drawn from a unit variance
+top RBM (RBM4) have stochastic real-valued states drawn from a unit variance
Gaussian whose mean is determined by the input from the RBM's logistic visible
units. So we add a `gaussian` field in the RBMHid layer to control the
sampling distribution (Gaussian or Bernoulli). In addition, this
RBM has a much smaller learning rate (0.001). The neural net configuration for
-the RBM3 and the updating protocol is (with layers for data layer and parser
+the RBM4 and the updating protocol is (with layers for data layer and parser
layer omitted),
# Updater Configuration
@@ -247,11 +247,11 @@ layer omitted),
hdim: 30
}
param{
- name: "w3"
+ name: "w4"
...
}
param{
- name: "b31"
+ name: "b41"
...
}
}
@@ -265,11 +265,11 @@ layer omitted),
gaussian: true
}
param{
- name: "w3_"
- share_from: "w3"
+ name: "w4_"
+ share_from: "w4"
}
param{
- name: "b32"
+ name: "b42"
...
}
}
@@ -278,8 +278,8 @@ layer omitted),
In the fine-tuning stage, the 4 RBMs are "unfolded" to form encoder and decoder
networks that are initialized using the parameters from the previous 4 RBMs.
-<img src="../images/autoencoder_new.PNG" align="center" width="500px"/>
-<span><strong>Figure 5 - Auto-Encoder.</strong></span>
+<img src="../images/example-autoencoder.png" align="center" width="500px"/>
+<span><strong>Figure 5 - Auto-Encoders.</strong></span>
Figure 5 shows the neural net structure for training the auto-encoder.
@@ -307,8 +307,8 @@ configuration is (with some of the middl
layer{ name:"mnist" }
layer{
name: "Inner1"
- param{ name: "w0" }
- param{ name: "b02" }
+ param{ name: "w1" }
+ param{ name: "b12" }
}
layer{ name: "Sigmoid1" }
...
@@ -322,7 +322,7 @@ configuration is (with some of the middl
name: "w8"
share_from: "w1"
}
- param{ name: "b01" }
+ param{ name: "b11" }
}
layer{ name: "Sigmoid8" }
@@ -337,10 +337,10 @@ configuration is (with some of the middl
To load pre-trained parameters from the 4 RBMs' checkpoint file we configure `checkpoint_path` as
### Checkpoint Configuration
- checkpoint_path: "examples/rbm/checkpoint/rbm0/checkpoint/step6000-worker0.bin"
checkpoint_path: "examples/rbm/checkpoint/rbm1/checkpoint/step6000-worker0.bin"
checkpoint_path: "examples/rbm/checkpoint/rbm2/checkpoint/step6000-worker0.bin"
checkpoint_path: "examples/rbm/checkpoint/rbm3/checkpoint/step6000-worker0.bin"
+ checkpoint_path: "examples/rbm/checkpoint/rbm4/checkpoint/step6000-worker0.bin"
## Visualization Results
@@ -359,7 +359,7 @@ To load pre-trained parameters from the
<span><strong>Figure 7 - Top layer features.</strong></span>
</div>
-Figure 6 visualizes sample columns of the weight matrix of RBM0, We can see the
+Figure 6 visualizes sample columns of the weight matrix of RBM1, We can see the
Gabor-like filters are learned. Figure 7 depicts the features extracted from
the top-layer of the auto-encoder, wherein one point represents one image.
Different colors represent different digits. We can see that most images are
Added: incubator/singa/site/trunk/content/resources/images/example-autoencoder.png
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/resources/images/example-autoencoder.png?rev=1703939&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/singa/site/trunk/content/resources/images/example-autoencoder.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/singa/site/trunk/content/resources/images/example-cnn.png
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/resources/images/example-cnn.png?rev=1703939&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/singa/site/trunk/content/resources/images/example-cnn.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/singa/site/trunk/content/resources/images/example-mlp.png
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/resources/images/example-mlp.png?rev=1703939&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/singa/site/trunk/content/resources/images/example-mlp.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/singa/site/trunk/content/resources/images/example-rbm1.png
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/resources/images/example-rbm1.png?rev=1703939&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/singa/site/trunk/content/resources/images/example-rbm1.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/singa/site/trunk/content/resources/images/example-rbm2.png
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/resources/images/example-rbm2.png?rev=1703939&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/singa/site/trunk/content/resources/images/example-rbm2.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/singa/site/trunk/content/resources/images/example-rbm3.png
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/resources/images/example-rbm3.png?rev=1703939&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/singa/site/trunk/content/resources/images/example-rbm3.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/singa/site/trunk/content/resources/images/example-rbm4.png
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/resources/images/example-rbm4.png?rev=1703939&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/singa/site/trunk/content/resources/images/example-rbm4.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream