You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by wa...@apache.org on 2016/01/08 10:41:55 UTC
svn commit: r1723668 - in /incubator/singa/site/trunk/content/markdown:
docs/index.md docs/python.md downloads.md
Author: wangwei
Date: Fri Jan 8 09:41:54 2016
New Revision: 1723668
URL: http://svn.apache.org/viewvc?rev=1723668&view=rev
Log:
add docs for python binding; chonho wrote this doc; wangwei updated it.
Added:
incubator/singa/site/trunk/content/markdown/docs/python.md
Modified:
incubator/singa/site/trunk/content/markdown/docs/index.md
incubator/singa/site/trunk/content/markdown/downloads.md
Modified: incubator/singa/site/trunk/content/markdown/docs/index.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/docs/index.md?rev=1723668&r1=1723667&r2=1723668&view=diff
==============================================================================
--- incubator/singa/site/trunk/content/markdown/docs/index.md (original)
+++ incubator/singa/site/trunk/content/markdown/docs/index.md Fri Jan 8 09:41:54 2016
@@ -12,6 +12,7 @@
* [Distributed Training](distributed-training.html)
* [Data Preparation](data.html)
* [Checkpoint and Resume](checkpoint.html)
+* [Python Binding](python.html)
* [Performance test and Feature extraction](test.html)
* [Training on GPU](gpu.html)
* [Examples](examples.html)
Added: incubator/singa/site/trunk/content/markdown/docs/python.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/docs/python.md?rev=1723668&view=auto
==============================================================================
--- incubator/singa/site/trunk/content/markdown/docs/python.md (added)
+++ incubator/singa/site/trunk/content/markdown/docs/python.md Fri Jan 8 09:41:54 2016
@@ -0,0 +1,374 @@
+# Python Binding
+
+---
+
+Python binding provides APIs for configuring a training job following
+[keras](http://keras.io/), including the configuration of neural net, training
+algorithm, etc. It replaces the configuration file (e.g., *job.conf*) in
+protobuf format, which is typically long and error-prone to prepare. In later
+version, we will add python functions to interact with the layer and neural net
+objects, which would enable users to train and debug their models
+interactively.
+
+Here is the layout of python related code,
+
+ SINGAROOT/tool/python
+ |-- pb2 (has job_pb2.py)
+ |-- singa
+ |-- model.py
+ |-- layer.py
+ |-- parameter.py
+ |-- initialization.py
+ |-- utils
+ |-- utility.py
+ |-- message.py
+ |-- examples
+ |-- cifar10_cnn.py, mnist_mlp.py, , mnist_rbm1.py, mnist_ae.py, etc.
+ |-- datasets
+ |-- cifar10.py
+ |-- mnist.py
+
+## Compiling and running instructions
+
+In order to use the Python APIs, users need to add the following option when compiling
+SINGA,
+
+ ./configure --enable-python --with-python=PYTHON_DIR
+ make
+
+where PYTHON_DIR has Python.h
+
+
+The training program is launched by
+
+ bin/singa-run.sh -exec <user_main.py>
+
+where user_main.py creates the JobProto object and pass it to Driver::Train to
+start the training.
+
+For example,
+
+ cd SINGA_ROOT
+ bin/singa-run.sh -exec tool/python/examples/cifar10_cnn.py
+
+
+
+## Examples
+
+
+### MLP Example
+
+This example uses python APIs to configure and train a MLP model over the MNIST
+dataset. The configuration content is the same as that written in *SINGAROOT/examples/mnist/job.conf*.
+
+```
+X_train, X_test, workspace = mnist.load_data()
+
+m = Sequential('mlp', sys.argv)
+
+m.add(Dense(2500, init='uniform', activation='tanh'))
+m.add(Dense(2000, init='uniform', activation='tanh'))
+m.add(Dense(1500, init='uniform', activation='tanh'))
+m.add(Dense(1000, init='uniform', activation='tanh'))
+m.add(Dense(500, init='uniform', activation='tanh'))
+m.add(Dense(10, init='uniform', activation='softmax'))
+
+sgd = SGD(lr=0.001, lr_type='step')
+topo = Cluster(workspace)
+m.compile(loss='categorical_crossentropy', optimizer=sgd, cluster=topo)
+m.fit(X_train, nb_epoch=1000, with_test=True)
+result = m.evaluate(X_test, batch_size=100, test_steps=10, test_freq=60)
+```
+
+### CNN Example
+
+This example uses python APIs to configure and train a CNN model over the Cifar10
+dataset. The configuration content is the same as that written in *SINGAROOT/examples/cifar10/job.conf*.
+
+
+```
+X_train, X_test, workspace = cifar10.load_data()
+
+m = Sequential('cnn', sys.argv)
+
+m.add(Convolution2D(32, 5, 1, 2, w_std=0.0001, b_lr=2))
+m.add(MaxPooling2D(pool_size=(3,3), stride=2))
+m.add(Activation('relu'))
+m.add(LRN2D(3, alpha=0.00005, beta=0.75))
+
+m.add(Convolution2D(32, 5, 1, 2, b_lr=2))
+m.add(Activation('relu'))
+m.add(AvgPooling2D(pool_size=(3,3), stride=2))
+m.add(LRN2D(3, alpha=0.00005, beta=0.75))
+
+m.add(Convolution2D(64, 5, 1, 2))
+m.add(Activation('relu'))
+m.add(AvgPooling2D(pool_size=(3,3), stride=2))
+
+m.add(Dense(10, w_wd=250, b_lr=2, b_wd=0, activation='softmax'))
+
+sgd = SGD(decay=0.004, lr_type='manual', step=(0,60000,65000), step_lr=(0.001,0.0001,0.00001))
+topo = Cluster(workspace)
+m.compile(updater=sgd, cluster=topo)
+m.fit(X_train, nb_epoch=1000, with_test=True)
+result = m.evaluate(X_test, 1000, test_steps=30, test_freq=300)
+```
+
+
+### RBM Example
+
+This example uses python APIs to configure and train a RBM model over the MNIST
+dataset. The configuration content is the same as that written in *SINGAROOT/examples/rbm*.conf*.
+
+```
+rbmid = 3
+X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid)
+m = Energy('rbm'+str(rbmid), sys.argv)
+
+out_dim = [1000, 500, 250]
+m.add(RBM(out_dim, w_std=0.1, b_wd=0))
+
+sgd = SGD(lr=0.1, decay=0.0002, momentum=0.8)
+topo = Cluster(workspace)
+m.compile(optimizer=sgd, cluster=topo)
+m.fit(X_train, alg='cd', nb_epoch=6000)
+```
+
+### AutoEncoder Example
+This example uses python APIs to configure and train an autoencoder model over
+the MNIST dataset. The configuration content is the same as that written in
+*SINGAROOT/examples/autoencoder.conf*.
+
+
+```
+rbmid = 4
+X_train, X_test, workspace = mnist.load_data(nb_rbm=rbmid+1)
+m = Sequential('autoencoder', sys.argv)
+
+hid_dim = [1000, 500, 250, 30]
+m.add(Autoencoder(hid_dim, out_dim=784, activation='sigmoid', param_share=True))
+
+agd = AdaGrad(lr=0.01)
+topo = Cluster(workspace)
+m.compile(loss='mean_squared_error', optimizer=agd, cluster=topo)
+m.fit(X_train, alg='bp', nb_epoch=12200)
+```
+
+### To run SINGA on GPU
+
+Users need to set a list of gpu ids to `device` field in fit() or evaluate().
+The number of GPUs must be the same to the number of workers configured for
+cluster topology.
+
+
+```
+gpu_id = [0]
+m.fit(X_train, nb_epoch=100, with_test=True, device=gpu_id)
+```
+
+### TIPS
+
+Hidden layers for MLP can be configured as
+
+```
+for n in [2500, 2000, 1500, 1000, 500]:
+ m.add(Dense(n, init='uniform', activation='tanh'))
+m.add(Dense(10, init='uniform', activation='softmax'))
+```
+
+Activation layer can be specified separately
+
+```
+m.add(Dense(2500, init='uniform'))
+m.add(Activation('tanh'))
+```
+
+Users can explicitly specify hyper-parameters of weight and bias
+
+```
+par = Parameter(init='uniform', scale=0.05)
+m.add(Dense(2500, w_param=par, b_param=par, activation='tanh'))
+m.add(Dense(2000, w_param=par, b_param=par, activation='tanh'))
+m.add(Dense(1500, w_param=par, b_param=par, activation='tanh'))
+m.add(Dense(1000, w_param=par, b_param=par, activation='tanh'))
+m.add(Dense(500, w_param=par, b_param=par, activation='tanh'))
+m.add(Dense(10, w_param=par, b_param=par, activation='softmax'))
+```
+
+
+```
+parw = Parameter(init='gauss', std=0.0001)
+parb = Parameter(init='const', value=0)
+m.add(Convolution(32, 5, 1, 2, w_param=parw, b_param=parb, b_lr=2))
+m.add(MaxPooling2D(pool_size(3,3), stride=2))
+m.add(Activation('relu'))
+m.add(LRN2D(3, alpha=0.00005, beta=0.75))
+
+parw.update(std=0.01)
+m.add(Convolution(32, 5, 1, 2, w_param=parw, b_param=parb))
+m.add(Activation('relu'))
+m.add(AvgPooling2D(pool_size(3,3), stride=2))
+m.add(LRN2D(3, alpha=0.00005, beta=0.75))
+
+m.add(Convolution(64, 5, 1, 2, w_param=parw, b_param=parb, b_lr=1))
+m.add(Activation('relu'))
+m.add(AvgPooling2D(pool_size(3,3), stride=2))
+
+m.add(Dense(10, w_param=parw, w_wd=250, b_param=parb, b_lr=2, b_wd=0, activation='softmax'))
+```
+
+
+Data can be added in this way,
+
+```
+X_train, X_test = mnist.load_data() // parameter values are set in load_data()
+m.fit(X_train, ...) // Data layer for training is added
+m.evaluate(X_test, ...) // Data layer for testing is added
+```
+or this way,
+
+```
+X_train, X_test = mnist.load_data() // parameter values are set in load_data()
+m.add(X_train) // explicitly add Data layer
+m.add(X_test) // explicitly add Data layer
+```
+
+
+```
+store = Store(path='train.bin', batch_size=64, ...) // parameter values are set explicitly
+m.add(Data(load='recordinput', phase='train', conf=store)) // Data layer is added
+store = Store(path='test.bin', batch_size=100, ...) // parameter values are set explicitly
+m.add(Data(load='recordinput', phase='test', conf=store)) // Data layer is added
+```
+
+
+### Cases to run SINGA
+
+(1) Run SINGA for training
+
+```
+m.fit(X_train, nb_epoch=1000)
+```
+
+(2) Run SINGA for training and validation
+
+```
+m.fit(X_train, validate_data=X_valid, nb_epoch=1000)
+```
+
+(3) Run SINGA for test while training
+
+```
+m.fit(X_train, nb_epoch=1000, with_test=True)
+result = m.evaluate(X_test, batch_size=100, test_steps=100)
+```
+
+(4) Run SINGA for test only
+Assume a checkpoint exists after training
+
+```
+result = m.evaluate(X_test, batch_size=100, checkpoint_path=workspace+'/checkpoint/step100-worker0')
+```
+
+
+## Implementation Details
+
+### Layer class (inherited)
+
+* Data
+* Dense
+* Activation
+* Convolution2D
+* MaxPooling2D
+* AvgPooling2D
+* LRN2D
+* Dropout
+* RBM
+* Autoencoder
+
+### Model class
+
+Model class has `jobconf` (JobProto) and `layers` (layer list)
+
+Methods in Model class
+
+* add
+ * add Layer into Model
+ * 2 subclasses: Sequential model and Energy model
+
+* compile
+ * set Updater (i.e., optimizer) and Cluster (i.e., topology) components
+
+* fit
+ * set Training data and parameter values for the training
+ * (optional) set Validatiaon data and parameter values
+ * set Train_one_batch component
+ * specify `with_test` field if a user wants to run SINGA with test data simultaneously.
+ * [TODO] recieve train/validation results, e.g., accuracy, loss, ppl, etc.
+
+* evaluate
+ * set Testing data and parameter values for the testing
+ * specify `checkpoint_path` field if a user want to run SINGA only for testing.
+ * [TODO] recieve test results, e.g., accuracy, loss, ppl, etc.
+
+### Results
+
+fit() and evaluate() return train/test results, a dictionary containing
+
+* [key]: step number
+* [value]: a list of dictionay
+ * 'acc' for accuracy
+ * 'loss' for loss
+ * 'ppl' for ppl
+ * 'se' for squred error
+
+
+### Parameter class
+
+Users need to set parameter and initial values. For example,
+
+* Parameter (fields in Param proto)
+ * lr = (float) // learning rate multiplier, used to scale the learning rate when updating parameters.
+ * wd = (float) // weight decay multiplier, used to scale the weight decay when updating parameters.
+
+* Parameter initialization (fields in ParamGen proto)
+ * init = (string) // one of the types, 'uniform', 'constant', 'gaussian'
+ * high = (float) // for 'uniform'
+ * low = (float) // for 'uniform'
+ * value = (float) // for 'constant'
+ * mean = (float) // for 'gaussian'
+ * std = (float) // for 'gaussian'
+
+* Weight (`w_param`) is 'gaussian' with mean=0, std=0.01 at default
+
+* Bias (`b_param`) is 'constant' with value=0 at default
+
+* How to update the parameter fields
+ * for updating Weight, put `w_` in front of field name
+ * for updating Bias, put `b_` in front of field name
+
+Several ways to set Parameter values
+
+```
+parw = Parameter(lr=2, wd=10, init='gaussian', std=0.1)
+parb = Parameter(lr=1, wd=0, init='constant', value=0)
+m.add(Convolution2D(10, w_param=parw, b_param=parb, ...)
+```
+
+```
+m.add(Dense(10, w_mean=1, w_std=0.1, w_lr=2, w_wd=10, ...)
+```
+
+```
+parw = Parameter(init='constant', mean=0)
+m.add(Dense(10, w_param=parw, w_lr=1, w_wd=1, b_value=1, ...)
+```
+
+### Other classes
+
+* Store
+* Algorithm
+* Updater
+* SGD
+* AdaGrad
+* Cluster
Modified: incubator/singa/site/trunk/content/markdown/downloads.md
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/content/markdown/downloads.md?rev=1723668&r1=1723667&r2=1723668&view=diff
==============================================================================
--- incubator/singa/site/trunk/content/markdown/downloads.md (original)
+++ incubator/singa/site/trunk/content/markdown/downloads.md Fri Jan 8 09:41:54 2016
@@ -9,11 +9,11 @@
[\[KEYS\]](https://dist.apache.org/repos/dist/dev/incubator/singa/0.2.0/KEYS)
* [Release Notes 0.2.0](releases/RELEASE_NOTES_0.2.0.html)
* New features and major updates,
- * [Training on GPU](../docs/gpu.html) enables training of complex models on a single node with multiple GPU cards.
- * [Hybrid neural net partitioning](../docs/frameworks.html) supports data and model parallelism at the same time.
- * [Python wrapper](../docs/python.html) makes it easy to configure the job, including neural net and SGD algorithm.
- * [RNN model and BPTT algorithm](../docs/rnn.html) are implemented to support applications based on RNN models, e.g., GRU.
- * [Cloud software integration](../docs/distributed-training.md) includes Mesos, Docker and HDFS.
+ * [Training on GPU](docs/gpu.html) enables training of complex models on a single node with multiple GPU cards.
+ * [Hybrid neural net partitioning](docs/frameworks.html) supports data and model parallelism at the same time.
+ * [Python wrapper](docs/python.html) makes it easy to configure the job, including neural net and SGD algorithm.
+ * [RNN model and BPTT algorithm](docs/rnn.html) are implemented to support applications based on RNN models, e.g., GRU.
+ * [Cloud software integration](docs/distributed-training.html) includes Mesos, Docker and HDFS.
* v0.1.0 (8 October 2015):