You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by wa...@apache.org on 2016/01/15 03:19:27 UTC
[1/2] incubator-singa git commit: SINGA-125 Improve Python Helper
Repository: incubator-singa
Updated Branches:
refs/heads/master e385d2a81 -> 1cfdac6c3
SINGA-125 Improve Python Helper
- Update README.md
- Update layer.py and model.py
. deal with non-square values for kernel, stride, pad
. users can specify Accuracy layer by 'show_acc=True'
- Update cifar10 examples
. set momentum to 0.9 to speed up convergence
Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/4662dc3e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/4662dc3e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/4662dc3e
Branch: refs/heads/master
Commit: 4662dc3e363f30b352d06e962c98c5028ad2d688
Parents: e385d2a
Author: chonho <le...@comp.nus.edu.sg>
Authored: Wed Jan 6 10:39:56 2016 +0800
Committer: chonho <le...@comp.nus.edu.sg>
Committed: Thu Jan 14 11:17:59 2016 +0800
----------------------------------------------------------------------
tool/python/README.md | 242 +++++++++++++------------
tool/python/examples/cifar10_cnn.py | 2 +-
tool/python/examples/cifar10_cnn_cudnn.py | 2 +-
tool/python/singa/layer.py | 52 ++++--
tool/python/singa/model.py | 19 +-
5 files changed, 189 insertions(+), 128 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/4662dc3e/tool/python/README.md
----------------------------------------------------------------------
diff --git a/tool/python/README.md b/tool/python/README.md
index 02e7fd1..e383cfb 100644
--- a/tool/python/README.md
+++ b/tool/python/README.md
@@ -1,6 +1,8 @@
-## SINGA-81 Add Python Helper, which enables users to construct a model (JobProto) and run Singa in Python
+# Python Helper
- SINGAROOT/tool/python
+Users can construct a model and run SINGA using Python. Specifically, the Python helper enables users to generate JobProto for the model and run Driver::Train or Driver::Test using Python. The Python Helper tool can be found in `SINGA_ROOT/tool/python` consisting of the following directories.
+
+ SINGAROOT/tool/python
|-- pb2 (has job_pb2.py)
|-- singa
|-- model.py
@@ -11,79 +13,83 @@
|-- utility.py
|-- message.py
|-- examples
- |-- cifar10_cnn.py, mnist_mlp.py, , mnist_rbm1.py, mnist_ae.py, etc.
+ |-- cifar10_cnn.py, mnist_mlp.py, mnist_rbm1.py, mnist_ae.py, etc.
|-- datasets
|-- cifar10.py
|-- mnist.py
-### How to Run
+##1. Basic User Guide
+
+In order to use the Python Helper features, users need to add the following option when building SINGA as follows.
+```
+./configure --enable-python --with-python=PYTHON_DIR
+```
+where `PYTHON_DIR` has `Python.h`
+
+### (a) How to Run
```
bin/singa-run.sh -exec user_main.py
```
-The python code, e.g., user_main.py, would create the JobProto object and pass it to Driver::Train.
+The python code, e.g., `user_main.py`, would create the JobProto object and pass it to Driver::Train or Driver:Test.
-For example,
+For running CIFAR10 example,
```
cd SINGA_ROOT
bin/singa-run.sh -exec tool/python/examples/cifar10_cnn.py
```
-
-Note that, in order to use the Python Helper feature, users need to add the following option
+For running MNIST example,
```
-./configure --enable-python --with-python=PYTHON_DIR
+cd SINGA_ROOT
+bin/singa-run.sh -exec tool/python/examples/mnist_mlp.py
```
-where PYTHON_DIR has Python.h
-### Layer class (inherited)
+### (b) Class Description
+
+#### Layer class
+
+The following classes configure field values for a particular layer and generate its LayerProto.
+
+* `Data` for a data layer.
+* `Dense` for an innerproduct layer.
+* `Activation` for an activation layer.
+* `Convolution2D` for a convolution layer.
+* `MaxPooling2D` for a max pooling layer.
+* `AvgPooling2D` for an average pooling layer.
+* `LRN2D` for a normalization (or local response normalization) layer.
+* `Dropout` for a dropout layer.
-* Data
-* Dense
-* Activation
-* Convolution2D
-* MaxPooling2D
-* AvgPooling2D
-* LRN2D
-* Dropout
-* RBM
-* Autoencoder
+In addition, the following classes generate multiple layers for particular models.
-### Model class
+* `RBM` for constructing layers of RBM.
+* `Autoencoder` for constructing layers of Autoencoder
-* Model class has `jobconf` (JobProto) and `layers` (layer list)
+
+#### Model class
+
+Model class has `jobconf` (JobProto) and `layers` (a layer list).
Methods in Model class
-* add
- * add Layer into Model
- * 2 subclasses: Sequential model and Energy model
+* `add` to add Layer into the model
+ * 2 subclasses: `Sequential` model and `Energy` model
-* compile
- * set Updater (i.e., optimizer) and Cluster (i.e., topology) components
+* `compile` to configure an optimizer and topology for training.
+ * set `Updater` (i.e., optimizer) and `Cluster` (i.e., topology) components
-* fit
+* `fit` to configure field values for training.
* set Training data and parameter values for the training
* (optional) set Validatiaon data and parameter values
- * set Train_one_batch component
- * specify `with_test` field if a user wants to run singa with test data simultaneously.
- * [TODO] recieve train/validation results, e.g., accuracy, loss, ppl, etc.
-
-* evaluate
- * set Testing data and parameter values for the testing
- * specify `checkpoint_path` field if a user want to run singa only for testing.
- * [TODO] recieve test results, e.g., accuracy, loss, ppl, etc.
+ * set `Train_one_batch` component
+ * set `with_test` argument `True` if users want to run SINGA with test data simultaneously.
+ * return train/validation results, e.g., accuracy, loss, ppl, etc.
-#### Results
+* `evaluate` to configure field values for test.
+ * set Testing data and parameter values for the test
+ * specify `checkpoint_path` field if users want to run SINGA only for test.
+ * return test results, e.g., accuracy, loss, ppl, etc.
-fit() and evaluate() return train/test results, a dictionary containing
-
-* [key]: step number
-* [value]: a list of dictionay
- * 'acc' for accuracy
- * 'loss' for loss
- * 'ppl' for ppl
- * 'se' for squred error
-#### To run Singa on GPU
+### (c) To Run Singa on GPU
Users need to set a list of gpu ids to `device` field in fit() or evaluate().
@@ -94,60 +100,53 @@ m.fit(X_train, nb_epoch=100, with_test=True, device=gpu_id)
```
-### Parameter class
+### (d) How to set/update parameter values
-Users need to set parameter and initial values. For example,
+Users may need to set/update parameter field values.
-* Parameter (fields in Param proto)
- * lr = (float) // learning rate multiplier, used to scale the learning rate when updating parameters.
- * wd = (float) // weight decay multiplier, used to scale the weight decay when updating parameters.
+* Parameter fields for both Weight and Bias (i.e., fields of ParamProto)
+ * `lr` = (float) : learning rate multiplier, used to scale the learning rate when updating parameters.
+ * `wd` = (float) : weight decay multiplier, used to scale the weight decay when updating parameters.
-* Parameter initialization (fields in ParamGen proto)
- * init = (string) // one of the types, 'uniform', 'constant', 'gaussian'
- * high = (float) // for 'uniform'
- * low = (float) // for 'uniform'
- * value = (float) // for 'constant'
- * mean = (float) // for 'gaussian'
- * std = (float) // for 'gaussian'
+* Parameter initialization (fields of ParamGenProto)
+ * `init` = (string) : one of the types, 'uniform', 'constant', 'gaussian'
+ * `scale` = (float) : for 'uniform', it is used to set `low`=-scale and `high`=+scale
+ * `high` = (float) : for 'uniform'
+ * `low` = (float) : for 'uniform'
+ * `value` = (float) : for 'constant'
+ * `mean` = (float) : for 'gaussian'
+ * `std` = (float) : for 'gaussian'
-* Weight (`w_param`) is 'gaussian' with mean=0, std=0.01 at default
+* Weight (`w_param`) is set as 'gaussian' with `mean`=0 and `std`=0.01 at default.
-* Bias (`b_param`) is 'constant' with value=0 at default
+* Bias (`b_param`) is set as 'constant' with `value`=0 at default.
-* How to update the parameter fields
- * for updating Weight, put `w_` in front of field name
- * for updating Bias, put `b_` in front of field name
+* In order to set/update the parameter fields of either Weight or Bias
+ * for Weight, put `w_` in front of field name
+ * for Bias, put `b_` in front of field name
-Several ways to set Parameter values
-```
-parw = Parameter(lr=2, wd=10, init='gaussian', std=0.1)
-parb = Parameter(lr=1, wd=0, init='constant', value=0)
-m.add(Convolution2D(10, w_param=parw, b_param=parb, ...)
-```
-```
-m.add(Dense(10, w_mean=1, w_std=0.1, w_lr=2, w_wd=10, ...)
-```
-```
-parw = Parameter(init='constant', mean=0)
-m.add(Dense(10, w_param=parw, w_lr=1, w_wd=1, b_value=1, ...)
-```
+ For example,
+ ```
+ m.add(Dense(10, w_mean=1, w_std=0.1, w_lr=2, w_wd=10, ...)
+ ```
+### (e) Results
-#### Other classes
+fit() and evaluate() return training/test results, i.e., a dictionary containing
-* Store
-* Algorithm
-* Updater
-* SGD
-* AdaGrad
-* Cluster
+* [key]: step number
+* [value]: a list of dictionay
+ * 'acc' for accuracy
+ * 'loss' for loss
+ * 'ppl' for ppl
+ * 'se' for squred error
-## MLP Example
-An example (to generate job.conf for mnist)
+## 2. Examples
+### MLP example (to generate job.conf for MNIST)
```
X_train, X_test, workspace = mnist.load_data()
@@ -167,10 +166,7 @@ m.fit(X_train, nb_epoch=1000, with_test=True)
result = m.evaluate(X_test, batch_size=100, test_steps=10, test_freq=60)
```
-## CNN Example
-
-An example (to generate job.conf for cifar10)
-
+### CNN example (to generate job.conf for cifar10)
```
X_train, X_test, workspace = cifar10.load_data()
@@ -199,8 +195,7 @@ m.fit(X_train, nb_epoch=1000, with_test=True)
result = m.evaluate(X_test, 1000, test_steps=30, test_freq=300)
```
-
-## RBM Example
+### RBM Example
```
rbmid = 3
X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid)
@@ -215,7 +210,7 @@ m.compile(optimizer=sgd, cluster=topo)
m.fit(X_train, alg='cd', nb_epoch=6000)
```
-## AutoEncoder Example
+### AutoEncoder Example
```
rbmid = 4
X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid+1)
@@ -230,7 +225,47 @@ m.compile(loss='mean_squared_error', optimizer=agd, cluster=topo)
m.fit(X_train, alg='bp', nb_epoch=12200)
```
-### TIPS
+
+## 3. Advanced User Guide
+
+### Parameter class
+
+Users can explicitly set/update parameter. There are several ways to set Parameter values
+```
+parw = Parameter(lr=2, wd=10, init='gaussian', std=0.1)
+parb = Parameter(lr=1, wd=0, init='constant', value=0)
+m.add(Convolution2D(10, w_param=parw, b_param=parb, ...)
+```
+```
+m.add(Dense(10, w_mean=1, w_std=0.1, w_lr=2, w_wd=10, ...)
+```
+```
+parw = Parameter(init='constant', mean=0)
+m.add(Dense(10, w_param=parw, w_lr=1, w_wd=1, b_value=1, ...)
+```
+
+### Data layer
+
+There are alternative ways to add Data layer. In addition, users can write your own `load_data` method of `cifar10.py` and `mnist.py` in `examples/dataset`.
+```
+X_train, X_test = mnist.load_data() // parameter values are set in load_data()
+m.fit(X_train, ...) // Data layer for training is added
+m.evaluate(X_test, ...) // Data layer for testing is added
+```
+```
+X_train, X_test = mnist.load_data() // parameter values are set in load_data()
+m.add(X_train) // explicitly add Data layer
+m.add(X_test) // explicitly add Data layer
+```
+```
+store = Store(path='train.bin', batch_size=64, ...) // parameter values are set explicitly
+m.add(Data(load='recordinput', phase='train', conf=store)) // Data layer is added
+store = Store(path='test.bin', batch_size=100, ...) // parameter values are set explicitly
+m.add(Data(load='recordinput', phase='test', conf=store)) // Data layer is added
+```
+
+
+### Other TIPS
Hidden layers for MLP can be written as
```
@@ -281,26 +316,9 @@ m.add(Dense(10, w_param=parw, w_wd=250, b_param=parb, b_lr=2, b_wd=0, activation
```
-Alternative ways to add Data layer
-```
-X_train, X_test = mnist.load_data() // parameter values are set in load_data()
-m.fit(X_train, ...) // Data layer for training is added
-m.evaluate(X_test, ...) // Data layer for testing is added
-```
-```
-X_train, X_test = mnist.load_data() // parameter values are set in load_data()
-m.add(X_train) // explicitly add Data layer
-m.add(X_test) // explicitly add Data layer
-```
-```
-store = Store(path='train.bin', batch_size=64, ...) // parameter values are set explicitly
-m.add(Data(load='recordinput', phase='train', conf=store)) // Data layer is added
-store = Store(path='test.bin', batch_size=100, ...) // parameter values are set explicitly
-m.add(Data(load='recordinput', phase='test', conf=store)) // Data layer is added
-```
-### Cases to run singa
+### Different Cases to Run SINGA
(1) Run singa for training
```
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/4662dc3e/tool/python/examples/cifar10_cnn.py
----------------------------------------------------------------------
diff --git a/tool/python/examples/cifar10_cnn.py b/tool/python/examples/cifar10_cnn.py
index f03b611..8d4e778 100755
--- a/tool/python/examples/cifar10_cnn.py
+++ b/tool/python/examples/cifar10_cnn.py
@@ -47,7 +47,7 @@ m.add(AvgPooling2D(pool_size=(3,3), stride=2))
m.add(Dense(10, w_wd=250, b_lr=2, b_wd=0, activation='softmax'))
-sgd = SGD(decay=0.004, lr_type='manual', step=(0,60000,65000), step_lr=(0.001,0.0001,0.00001))
+sgd = SGD(decay=0.004, momentum=0.9, lr_type='manual', step=(0,60000,65000), step_lr=(0.001,0.0001,0.00001))
topo = Cluster(workspace)
m.compile(loss='categorical_crossentropy', optimizer=sgd, cluster=topo)
m.fit(X_train, nb_epoch=1000, with_test=True)
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/4662dc3e/tool/python/examples/cifar10_cnn_cudnn.py
----------------------------------------------------------------------
diff --git a/tool/python/examples/cifar10_cnn_cudnn.py b/tool/python/examples/cifar10_cnn_cudnn.py
index e87b5c4..e243834 100755
--- a/tool/python/examples/cifar10_cnn_cudnn.py
+++ b/tool/python/examples/cifar10_cnn_cudnn.py
@@ -47,7 +47,7 @@ m.add(AvgPooling2D(pool_size=(3,3), stride=2))
m.add(Dense(10, w_wd=250, b_lr=2, b_wd=0, activation='softmax'))
-sgd = SGD(decay=0.004, lr_type='manual', step=(0,60000,65000), step_lr=(0.001,0.0001,0.00001))
+sgd = SGD(decay=0.004, momentum=0.9, lr_type='manual', step=(0,60000,65000), step_lr=(0.001,0.0001,0.00001))
topo = Cluster(workspace)
m.compile(loss='categorical_crossentropy', optimizer=sgd, cluster=topo)
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/4662dc3e/tool/python/singa/layer.py
----------------------------------------------------------------------
diff --git a/tool/python/singa/layer.py b/tool/python/singa/layer.py
index b391d26..491e98b 100644
--- a/tool/python/singa/layer.py
+++ b/tool/python/singa/layer.py
@@ -95,12 +95,12 @@ class Convolution2D(Layer):
activation=None, **kwargs):
'''
required
- nb_filter = (int) // the number of filters
- kernel = (int) // the size of filter
+ nb_filter = (int) // the number of filters
+ kernel = (int/tuple) // the size of filter
optional
- stride = (int) // the size of stride
- pad = (int) // the size of padding
- init = (string) // 'unirom', 'gaussian', 'constant'
+ stride = (int/tuple) // the size of stride
+ pad = (int/tuple) // the size of padding
+ init = (string) // 'uniform', 'gaussian', 'constant'
w_param = (Parameter) // Parameter object for weight
b_param = (Parameter) // Parameter object for bias
**kwargs (KEY=VALUE)
@@ -112,13 +112,29 @@ class Convolution2D(Layer):
b_wd = (float) // weight decay multiplier for bias
'''
- assert nb_filter > 0 and kernel > 0, 'should be set as positive int'
+ assert nb_filter > 0, 'nb_filter should be set as positive int'
super(Convolution2D, self).__init__(name=generate_name('conv', 1),
type=kCConvolution)
- fields = {'num_filters' : nb_filter,
- 'kernel' : kernel,
- 'stride' : stride,
- 'pad' : pad}
+ fields = {}
+ # for kernel
+ if type(kernel) == int:
+ fields['kernel'] = kernel
+ else:
+ fields['kernel_x'] = kernel[0]
+ fields['kernel_y'] = kernel[1]
+ # for stride
+ if type(stride) == int:
+ fields['stride'] = stride
+ else:
+ fields['stride_x'] = stride[0]
+ fields['stride_y'] = stride[1]
+ # for pad
+ if type(pad) == int:
+ fields['pad'] = pad
+ else:
+ fields['pad_x'] = pad[0]
+ fields['pad_y'] = pad[1]
+
setval(self.layer.convolution_conf, **fields)
# parameter w
@@ -158,7 +174,7 @@ class MaxPooling2D(Layer):
if type(pool_size) == int:
pool_size = (pool_size, pool_size)
assert type(pool_size) == tuple and pool_size[0] == pool_size[1], \
- 'pool size should be square in Singa'
+ 'currently pool size should be square in Singa'
super(MaxPooling2D, self).__init__(name=generate_name('pool'),
type=kCPooling, **kwargs)
fields = {'pool' : PoolingProto().MAX,
@@ -184,7 +200,7 @@ class AvgPooling2D(Layer):
if type(pool_size) == int:
pool_size = (pool_size, pool_size)
assert type(pool_size) == tuple and pool_size[0] == pool_size[1], \
- 'pool size should be square in Singa'
+ 'currently pool size should be square in Singa'
super(AvgPooling2D, self).__init__(name=generate_name('pool'),
type=kCPooling, **kwargs)
self.layer.pooling_conf.pool = PoolingProto().AVG
@@ -242,6 +258,16 @@ class Dropout(Layer):
type=self.layer_type)
self.layer.dropout_conf.dropout_ratio = ratio
+class Accuracy(Layer):
+
+ def __init__(self):
+ '''
+ '''
+
+ self.name = 'accuracy'
+ self.layer_type = enumLayerType(self.name)
+ super(Accuracy, self).__init__(name=generate_name(self.name),
+ type=self.layer_type)
class RGB(Layer):
@@ -268,7 +294,7 @@ class Dense(Layer):
output_dim = (int)
optional
activation = (string)
- init = (string) // 'unirom', 'gaussian', 'constant'
+ init = (string) // 'uniform', 'gaussian', 'constant'
w_param = (Parameter) // Parameter object for weight
b_param = (Parameter) // Parameter object for bias
**kwargs
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/4662dc3e/tool/python/singa/model.py
----------------------------------------------------------------------
diff --git a/tool/python/singa/model.py b/tool/python/singa/model.py
index 6ad9422..f652f86 100644
--- a/tool/python/singa/model.py
+++ b/tool/python/singa/model.py
@@ -55,6 +55,7 @@ class Model(object):
self.result = None
self.last_checkpoint_path = None
self.cudnn = False
+ self.accuracy = False
def add(self, layer):
'''
@@ -151,6 +152,17 @@ class Model(object):
else:
getattr(lastly, 'srclayers').append(self.layers[0].layer.name)
+ if self.accuracy == True:
+ smly = net.layer.add()
+ smly.CopyFrom(Layer(name='softmax', type=kSoftmax).layer)
+ setval(smly, include=kTest)
+ getattr(smly, 'srclayers').append(self.layers[-1].layer.name)
+ aly = net.layer.add()
+ aly.CopyFrom(Accuracy().layer)
+ setval(aly, include=kTest)
+ getattr(aly, 'srclayers').append('softmax')
+ getattr(aly, 'srclayers').append(self.layers[0].layer.name)
+
# use of cudnn
if self.cudnn == True:
self.set_cudnn_layer_type(net)
@@ -230,7 +242,8 @@ class Model(object):
pass
def evaluate(self, data=None, alg='bp',
- checkpoint_path=None, execpath='', device=None, **fields):
+ checkpoint_path=None, execpath='',
+ device=None, show_acc=False, **fields):
'''
required
data = (Data) // Data class object for testing data
@@ -239,6 +252,7 @@ class Model(object):
checkpoint_path = (list) // checkpoint path
execpaths = (string) // path to user's own executable
device = (int/list) // a list of gpu ids
+ show_acc = (bool) // compute and the accuacy
**fields (KEY=VALUE)
batch_size = (int) // batch size for testing data
test_freq = (int) // frequency of testing
@@ -276,6 +290,9 @@ class Model(object):
setval(self.jobconf, gpu=device)
self.cudnn = True
+ # set True if showing the accuracy
+ self.accuracy = show_acc
+
self.build() # construct Nneuralnet Component
#--- generate job.conf file for debug purpose
[2/2] incubator-singa git commit: SINGA-125 Improve Python Helper
Posted by wa...@apache.org.
SINGA-125 Improve Python Helper
- Update license header of __init__.py
Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/1cfdac6c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/1cfdac6c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/1cfdac6c
Branch: refs/heads/master
Commit: 1cfdac6c304018fc295a0fe8e0eca740cb62817f
Parents: 4662dc3
Author: chonho <le...@comp.nus.edu.sg>
Authored: Thu Jan 14 11:22:41 2016 +0800
Committer: chonho <le...@comp.nus.edu.sg>
Committed: Thu Jan 14 11:25:27 2016 +0800
----------------------------------------------------------------------
tool/python/README.md | 396 +++++++++++++++++--------------
tool/python/singa/__init__.py | 40 ++--
tool/python/singa/utils/__init__.py | 40 ++--
3 files changed, 254 insertions(+), 222 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1cfdac6c/tool/python/README.md
----------------------------------------------------------------------
diff --git a/tool/python/README.md b/tool/python/README.md
index e383cfb..8c90cfe 100644
--- a/tool/python/README.md
+++ b/tool/python/README.md
@@ -1,163 +1,77 @@
-# Python Helper
+# Python Binding
-Users can construct a model and run SINGA using Python. Specifically, the Python helper enables users to generate JobProto for the model and run Driver::Train or Driver::Test using Python. The Python Helper tool can be found in `SINGA_ROOT/tool/python` consisting of the following directories.
+---
- SINGAROOT/tool/python
- |-- pb2 (has job_pb2.py)
- |-- singa
- |-- model.py
- |-- layer.py
- |-- parameter.py
- |-- initialization.py
- |-- utils
- |-- utility.py
- |-- message.py
- |-- examples
- |-- cifar10_cnn.py, mnist_mlp.py, mnist_rbm1.py, mnist_ae.py, etc.
- |-- datasets
- |-- cifar10.py
- |-- mnist.py
-
-##1. Basic User Guide
-
-In order to use the Python Helper features, users need to add the following option when building SINGA as follows.
-```
-./configure --enable-python --with-python=PYTHON_DIR
-```
-where `PYTHON_DIR` has `Python.h`
-
-### (a) How to Run
-```
-bin/singa-run.sh -exec user_main.py
-```
-The python code, e.g., `user_main.py`, would create the JobProto object and pass it to Driver::Train or Driver:Test.
-
-For running CIFAR10 example,
-```
-cd SINGA_ROOT
-bin/singa-run.sh -exec tool/python/examples/cifar10_cnn.py
-```
-For running MNIST example,
-```
-cd SINGA_ROOT
-bin/singa-run.sh -exec tool/python/examples/mnist_mlp.py
-```
-
-### (b) Class Description
-
-#### Layer class
-
-The following classes configure field values for a particular layer and generate its LayerProto.
-
-* `Data` for a data layer.
-* `Dense` for an innerproduct layer.
-* `Activation` for an activation layer.
-* `Convolution2D` for a convolution layer.
-* `MaxPooling2D` for a max pooling layer.
-* `AvgPooling2D` for an average pooling layer.
-* `LRN2D` for a normalization (or local response normalization) layer.
-* `Dropout` for a dropout layer.
-
-In addition, the following classes generate multiple layers for particular models.
-
-* `RBM` for constructing layers of RBM.
-* `Autoencoder` for constructing layers of Autoencoder
+Python binding provides APIs for configuring a training job following
+[keras](http://keras.io/), including the configuration of neural net, training
+algorithm, etc. It replaces the configuration file (e.g., *job.conf*) in
+protobuf format, which is typically long and error-prone to prepare. In later
+version, we will add python functions to interact with the layer and neural net
+objects, which would enable users to train and debug their models
+interactively.
+Here is the layout of python related code,
-#### Model class
-
-Model class has `jobconf` (JobProto) and `layers` (a layer list).
+ SINGAROOT/tool/python
+ |-- pb2 (has job_pb2.py)
+ |-- singa
+ |-- model.py
+ |-- layer.py
+ |-- parameter.py
+ |-- initialization.py
+ |-- utils
+ |-- utility.py
+ |-- message.py
+ |-- examples
+ |-- cifar10_cnn.py, mnist_mlp.py, , mnist_rbm1.py, mnist_ae.py, etc.
+ |-- datasets
+ |-- cifar10.py
+ |-- mnist.py
-Methods in Model class
+## Compiling and running instructions
-* `add` to add Layer into the model
- * 2 subclasses: `Sequential` model and `Energy` model
+In order to use the Python APIs, users need to add the following arguments when compiling
+SINGA,
-* `compile` to configure an optimizer and topology for training.
- * set `Updater` (i.e., optimizer) and `Cluster` (i.e., topology) components
+ ./configure --enable-python --with-python=PYTHON_DIR
+ make
-* `fit` to configure field values for training.
- * set Training data and parameter values for the training
- * (optional) set Validatiaon data and parameter values
- * set `Train_one_batch` component
- * set `with_test` argument `True` if users want to run SINGA with test data simultaneously.
- * return train/validation results, e.g., accuracy, loss, ppl, etc.
+where PYTHON_DIR has Python.h
-* `evaluate` to configure field values for test.
- * set Testing data and parameter values for the test
- * specify `checkpoint_path` field if users want to run SINGA only for test.
- * return test results, e.g., accuracy, loss, ppl, etc.
+The training program is launched by
-### (c) To Run Singa on GPU
+ bin/singa-run.sh -exec <user_main.py>
-Users need to set a list of gpu ids to `device` field in fit() or evaluate().
+where user_main.py creates the JobProto object and passes it to Driver::Train to
+start the training.
For example,
-```
-gpu_id = [0]
-m.fit(X_train, nb_epoch=100, with_test=True, device=gpu_id)
-```
-
-### (d) How to set/update parameter values
+ cd SINGAROOT
+ bin/singa-run.sh -exec tool/python/examples/cifar10_cnn.py
-Users may need to set/update parameter field values.
-* Parameter fields for both Weight and Bias (i.e., fields of ParamProto)
- * `lr` = (float) : learning rate multiplier, used to scale the learning rate when updating parameters.
- * `wd` = (float) : weight decay multiplier, used to scale the weight decay when updating parameters.
-* Parameter initialization (fields of ParamGenProto)
- * `init` = (string) : one of the types, 'uniform', 'constant', 'gaussian'
- * `scale` = (float) : for 'uniform', it is used to set `low`=-scale and `high`=+scale
- * `high` = (float) : for 'uniform'
- * `low` = (float) : for 'uniform'
- * `value` = (float) : for 'constant'
- * `mean` = (float) : for 'gaussian'
- * `std` = (float) : for 'gaussian'
-
-* Weight (`w_param`) is set as 'gaussian' with `mean`=0 and `std`=0.01 at default.
-
-* Bias (`b_param`) is set as 'constant' with `value`=0 at default.
-
-* In order to set/update the parameter fields of either Weight or Bias
- * for Weight, put `w_` in front of field name
- * for Bias, put `b_` in front of field name
-
- For example,
- ```
- m.add(Dense(10, w_mean=1, w_std=0.1, w_lr=2, w_wd=10, ...)
- ```
-
-
-### (e) Results
-
-fit() and evaluate() return training/test results, i.e., a dictionary containing
-
-* [key]: step number
-* [value]: a list of dictionay
- * 'acc' for accuracy
- * 'loss' for loss
- * 'ppl' for ppl
- * 'se' for squred error
+## Examples
+### MLP Example
-## 2. Examples
+This example uses python APIs to configure and train a MLP model over the MNIST
+dataset. The configuration content is the same as that written in *SINGAROOT/examples/mnist/job.conf*.
-### MLP example (to generate job.conf for MNIST)
```
X_train, X_test, workspace = mnist.load_data()
-m = Sequential('mlp', sys.argv)
+m = Sequential('mlp', sys.argv)
m.add(Dense(2500, init='uniform', activation='tanh'))
m.add(Dense(2000, init='uniform', activation='tanh'))
m.add(Dense(1500, init='uniform', activation='tanh'))
m.add(Dense(1000, init='uniform', activation='tanh'))
m.add(Dense(500, init='uniform', activation='tanh'))
-m.add(Dense(10, init='uniform', activation='softmax'))
+m.add(Dense(10, init='uniform', activation='softmax'))
sgd = SGD(lr=0.001, lr_type='step')
topo = Cluster(workspace)
@@ -166,7 +80,12 @@ m.fit(X_train, nb_epoch=1000, with_test=True)
result = m.evaluate(X_test, batch_size=100, test_steps=10, test_freq=60)
```
-### CNN example (to generate job.conf for cifar10)
+### CNN Example
+
+This example uses python APIs to configure and train a CNN model over the Cifar10
+dataset. The configuration content is the same as that written in *SINGAROOT/examples/cifar10/job.conf*.
+
+
```
X_train, X_test, workspace = cifar10.load_data()
@@ -195,25 +114,35 @@ m.fit(X_train, nb_epoch=1000, with_test=True)
result = m.evaluate(X_test, 1000, test_steps=30, test_freq=300)
```
+
### RBM Example
+
+This example uses python APIs to configure and train a RBM model over the MNIST
+dataset. The configuration content is the same as that written in *SINGAROOT/examples/rbm*.conf*.
+
```
-rbmid = 3
-X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid)
+rbmid = 3
+X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid)
m = Energy('rbm'+str(rbmid), sys.argv)
out_dim = [1000, 500, 250]
-m.add(RBM(out_dim, w_std=0.1, b_wd=0))
-
-sgd = SGD(lr=0.1, decay=0.0002, momentum=0.8)
-topo = Cluster(workspace)
-m.compile(optimizer=sgd, cluster=topo)
-m.fit(X_train, alg='cd', nb_epoch=6000)
+m.add(RBM(out_dim, w_std=0.1, b_wd=0))
+
+sgd = SGD(lr=0.1, decay=0.0002, momentum=0.8)
+topo = Cluster(workspace)
+m.compile(optimizer=sgd, cluster=topo)
+m.fit(X_train, alg='cd', nb_epoch=6000)
```
### AutoEncoder Example
+This example uses python APIs to configure and train an autoencoder model over
+the MNIST dataset. The configuration content is the same as that written in
+*SINGAROOT/examples/autoencoder.conf*.
+
+
```
rbmid = 4
-X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid+1)
+X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid+1)
m = Sequential('autoencoder', sys.argv)
hid_dim = [1000, 500, 250, 30]
@@ -225,49 +154,22 @@ m.compile(loss='mean_squared_error', optimizer=agd, cluster=topo)
m.fit(X_train, alg='bp', nb_epoch=12200)
```
+### To run SINGA on GPU
-## 3. Advanced User Guide
-
-### Parameter class
-
-Users can explicitly set/update parameter. There are several ways to set Parameter values
-```
-parw = Parameter(lr=2, wd=10, init='gaussian', std=0.1)
-parb = Parameter(lr=1, wd=0, init='constant', value=0)
-m.add(Convolution2D(10, w_param=parw, b_param=parb, ...)
-```
-```
-m.add(Dense(10, w_mean=1, w_std=0.1, w_lr=2, w_wd=10, ...)
-```
-```
-parw = Parameter(init='constant', mean=0)
-m.add(Dense(10, w_param=parw, w_lr=1, w_wd=1, b_value=1, ...)
-```
+Users need to set a list of gpu ids to `device` field in fit() or evaluate().
+The number of GPUs must be the same to the number of workers configured for
+cluster topology.
-### Data layer
-There are alternative ways to add Data layer. In addition, users can write your own `load_data` method of `cifar10.py` and `mnist.py` in `examples/dataset`.
-```
-X_train, X_test = mnist.load_data() // parameter values are set in load_data()
-m.fit(X_train, ...) // Data layer for training is added
-m.evaluate(X_test, ...) // Data layer for testing is added
-```
-```
-X_train, X_test = mnist.load_data() // parameter values are set in load_data()
-m.add(X_train) // explicitly add Data layer
-m.add(X_test) // explicitly add Data layer
-```
```
-store = Store(path='train.bin', batch_size=64, ...) // parameter values are set explicitly
-m.add(Data(load='recordinput', phase='train', conf=store)) // Data layer is added
-store = Store(path='test.bin', batch_size=100, ...) // parameter values are set explicitly
-m.add(Data(load='recordinput', phase='test', conf=store)) // Data layer is added
+gpu_id = [0]
+m.fit(X_train, nb_epoch=100, with_test=True, device=gpu_id)
```
+### TIPS
-### Other TIPS
+Hidden layers for MLP can be configured as
-Hidden layers for MLP can be written as
```
for n in [2500, 2000, 1500, 1000, 500]:
m.add(Dense(n, init='uniform', activation='tanh'))
@@ -275,14 +177,14 @@ m.add(Dense(10, init='uniform', activation='softmax'))
```
Activation layer can be specified separately
+
```
m.add(Dense(2500, init='uniform'))
m.add(Activation('tanh'))
```
-Users can explicity specify weight and bias, and their values
+Users can explicitly specify hyper-parameters of weight and bias
-for example of MLP
```
par = Parameter(init='uniform', scale=0.05)
m.add(Dense(2500, w_param=par, b_param=par, activation='tanh'))
@@ -293,7 +195,7 @@ m.add(Dense(500, w_param=par, b_param=par, activation='tanh'))
m.add(Dense(10, w_param=par, b_param=par, activation='softmax'))
```
-for example of Cifar10
+
```
parw = Parameter(init='gauss', std=0.0001)
parb = Parameter(init='const', value=0)
@@ -316,28 +218,158 @@ m.add(Dense(10, w_param=parw, w_wd=250, b_param=parb, b_lr=2, b_wd=0, activation
```
+Data can be added in this way,
+
+```
+X_train, X_test = mnist.load_data() // parameter values are set in load_data()
+m.fit(X_train, ...) // Data layer for training is added
+m.evaluate(X_test, ...) // Data layer for testing is added
+```
+or this way,
+
+```
+X_train, X_test = mnist.load_data() // parameter values are set in load_data()
+m.add(X_train) // explicitly add Data layer
+m.add(X_test) // explicitly add Data layer
+```
+
+
+```
+store = Store(path='train.bin', batch_size=64, ...) // parameter values are set explicitly
+m.add(Data(load='recordinput', phase='train', conf=store)) // Data layer is added
+store = Store(path='test.bin', batch_size=100, ...) // parameter values are set explicitly
+m.add(Data(load='recordinput', phase='test', conf=store)) // Data layer is added
+```
+
+### Cases to run SINGA
-### Different Cases to Run SINGA
+(1) Run SINGA for training
-(1) Run singa for training
```
m.fit(X_train, nb_epoch=1000)
```
-(2) Run singa for training and validation
+(2) Run SINGA for training and validation
+
```
m.fit(X_train, validate_data=X_valid, nb_epoch=1000)
```
-(3) Run singa for test while training
+(3) Run SINGA for test while training
+
```
m.fit(X_train, nb_epoch=1000, with_test=True)
result = m.evaluate(X_test, batch_size=100, test_steps=100)
```
-(4) Run singa for test only
+(4) Run SINGA for test only
Assume a checkpoint exists after training
+
```
result = m.evaluate(X_test, batch_size=100, checkpoint_path=workspace+'/checkpoint/step100-worker0')
```
+
+
+## Implementation Details
+
+### Layer class (inherited)
+
+* Data
+* Dense
+* Activation
+* Convolution2D
+* MaxPooling2D
+* AvgPooling2D
+* LRN2D
+* Dropout
+* RBM
+* Autoencoder
+
+### Model class
+
+Model class has `jobconf` (JobProto) and `layers` (layer list)
+
+Methods in Model class
+
+* add
+ * add Layer into Model
+ * 2 subclasses: Sequential model and Energy model
+
+* compile
+ * set Updater (i.e., optimizer) and Cluster (i.e., topology) components
+
+* fit
+ * set Training data and parameter values for the training
+ * (optional) set Validatiaon data and parameter values
+ * set Train_one_batch component
+ * specify `with_test` field if a user wants to run SINGA with test data simultaneously.
+ * [TODO] recieve train/validation results, e.g., accuracy, loss, ppl, etc.
+
+* evaluate
+ * set Testing data and parameter values for the testing
+ * specify `checkpoint_path` field if a user want to run SINGA only for testing.
+ * [TODO] recieve test results, e.g., accuracy, loss, ppl, etc.
+
+### Results
+
+fit() and evaluate() return train/test results, a dictionary containing
+
+* [key]: step number
+* [value]: a list of dictionay
+ * 'acc' for accuracy
+ * 'loss' for loss
+ * 'ppl' for ppl
+ * 'se' for squred error
+
+
+### Parameter class
+
+Users need to set parameter and initial values. For example,
+
+* Parameter (fields in Param proto)
+ * lr = (float) // learning rate multiplier, used to scale the learning rate when updating parameters.
+ * wd = (float) // weight decay multiplier, used to scale the weight decay when updating parameters.
+
+* Parameter initialization (fields in ParamGen proto)
+ * init = (string) // one of the types, 'uniform', 'constant', 'gaussian'
+ * high = (float) // for 'uniform'
+ * low = (float) // for 'uniform'
+ * value = (float) // for 'constant'
+ * mean = (float) // for 'gaussian'
+ * std = (float) // for 'gaussian'
+
+* Weight (`w_param`) is 'gaussian' with mean=0, std=0.01 at default
+
+* Bias (`b_param`) is 'constant' with value=0 at default
+
+* How to update the parameter fields
+ * for updating Weight, put `w_` in front of field name
+ * for updating Bias, put `b_` in front of field name
+
+Several ways to set Parameter values
+
+```
+parw = Parameter(lr=2, wd=10, init='gaussian', std=0.1)
+parb = Parameter(lr=1, wd=0, init='constant', value=0)
+m.add(Convolution2D(10, w_param=parw, b_param=parb, ...)
+```
+
+```
+m.add(Dense(10, w_mean=1, w_std=0.1, w_lr=2, w_wd=10, ...)
+```
+
+```
+parw = Parameter(init='constant', mean=0)
+m.add(Dense(10, w_param=parw, w_lr=1, w_wd=1, b_value=1, ...)
+```
+
+### Other classes
+
+* Store
+* Algorithm
+* Updater
+* SGD
+* AdaGrad
+* Cluster
+
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1cfdac6c/tool/python/singa/__init__.py
----------------------------------------------------------------------
diff --git a/tool/python/singa/__init__.py b/tool/python/singa/__init__.py
index 55edef3..a796a7a 100644
--- a/tool/python/singa/__init__.py
+++ b/tool/python/singa/__init__.py
@@ -1,22 +1,22 @@
-/************************************************************
-*
-* Licensed to the Apache Software Foundation (ASF) under one
-* or more contributor license agreements. See the NOTICE file
-* distributed with this work for additional information
-* regarding copyright ownership. The ASF licenses this file
-* to you under the Apache License, Version 2.0 (the
-* "License"); you may not use this file except in compliance
-* with the License. You may obtain a copy of the License at
-*
-* http://www.apache.org/licenses/LICENSE-2.0
-*
-* Unless required by applicable law or agreed to in writing,
-* software distributed under the License is distributed on an
-* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-* KIND, either express or implied. See the License for the
-* specific language governing permissions and limitations
-* under the License.
-*
-*************************************************************/
+#/************************************************************
+#*
+#* Licensed to the Apache Software Foundation (ASF) under one
+#* or more contributor license agreements. See the NOTICE file
+#* distributed with this work for additional information
+#* regarding copyright ownership. The ASF licenses this file
+#* to you under the Apache License, Version 2.0 (the
+#* "License"); you may not use this file except in compliance
+#* with the License. You may obtain a copy of the License at
+#*
+#* http://www.apache.org/licenses/LICENSE-2.0
+#*
+#* Unless required by applicable law or agreed to in writing,
+#* software distributed under the License is distributed on an
+#* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+#* KIND, either express or implied. See the License for the
+#* specific language governing permissions and limitations
+#* under the License.
+#*
+#*************************************************************/
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/1cfdac6c/tool/python/singa/utils/__init__.py
----------------------------------------------------------------------
diff --git a/tool/python/singa/utils/__init__.py b/tool/python/singa/utils/__init__.py
index 55edef3..a796a7a 100644
--- a/tool/python/singa/utils/__init__.py
+++ b/tool/python/singa/utils/__init__.py
@@ -1,22 +1,22 @@
-/************************************************************
-*
-* Licensed to the Apache Software Foundation (ASF) under one
-* or more contributor license agreements. See the NOTICE file
-* distributed with this work for additional information
-* regarding copyright ownership. The ASF licenses this file
-* to you under the Apache License, Version 2.0 (the
-* "License"); you may not use this file except in compliance
-* with the License. You may obtain a copy of the License at
-*
-* http://www.apache.org/licenses/LICENSE-2.0
-*
-* Unless required by applicable law or agreed to in writing,
-* software distributed under the License is distributed on an
-* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-* KIND, either express or implied. See the License for the
-* specific language governing permissions and limitations
-* under the License.
-*
-*************************************************************/
+#/************************************************************
+#*
+#* Licensed to the Apache Software Foundation (ASF) under one
+#* or more contributor license agreements. See the NOTICE file
+#* distributed with this work for additional information
+#* regarding copyright ownership. The ASF licenses this file
+#* to you under the Apache License, Version 2.0 (the
+#* "License"); you may not use this file except in compliance
+#* with the License. You may obtain a copy of the License at
+#*
+#* http://www.apache.org/licenses/LICENSE-2.0
+#*
+#* Unless required by applicable law or agreed to in writing,
+#* software distributed under the License is distributed on an
+#* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+#* KIND, either express or implied. See the License for the
+#* specific language governing permissions and limitations
+#* under the License.
+#*
+#*************************************************************/