You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by bu...@apache.org on 2015/09/26 14:10:19 UTC
svn commit: r966886 - in /websites/staging/singa/trunk/content: ./
docs/data.html docs/distributed-training.html docs/examples.html
docs/frameworks.html docs/model-config.html docs/programming-guide.html
docs/train-one-batch.html
Author: buildbot
Date: Sat Sep 26 12:10:18 2015
New Revision: 966886
Log:
Staging update by buildbot for singa
Modified:
websites/staging/singa/trunk/content/ (props changed)
websites/staging/singa/trunk/content/docs/data.html
websites/staging/singa/trunk/content/docs/distributed-training.html
websites/staging/singa/trunk/content/docs/examples.html
websites/staging/singa/trunk/content/docs/frameworks.html
websites/staging/singa/trunk/content/docs/model-config.html
websites/staging/singa/trunk/content/docs/programming-guide.html
websites/staging/singa/trunk/content/docs/train-one-batch.html
Propchange: websites/staging/singa/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Sat Sep 26 12:10:18 2015
@@ -1 +1 @@
-1705429
+1705434
Modified: websites/staging/singa/trunk/content/docs/data.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/data.html (original)
+++ websites/staging/singa/trunk/content/docs/data.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Data Preparation</h1>
+<hr />
<p>To submit a training job, users need to convert raw data (e.g., images, text documents) into SINGA recognizable <a href="../api/classsinga_1_1Record.html">Record</a>s. SINGA uses <a href="layer#data-layers">data layers</a> to load these records into memory and uses <a href="layer#parser-layers">parser layers</a> to parse features (e.g., image pixels and labels) from these <tt>Record</tt>s. <tt>Record</tt>s could be stored in a file, a database, or HDFS, as long as there is a corresponding <a href="../api/classsinga_1_1DataLayer.html">DataLayer</a>.</p>
<div class="section">
<h2><a name="DataShard"></a>DataShard</h2>
Modified: websites/staging/singa/trunk/content/docs/distributed-training.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/distributed-training.html (original)
+++ websites/staging/singa/trunk/content/docs/distributed-training.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Distributed Training</h1>
+<hr />
<p>SINGA is designed for distributed training of large deep learning models with huge amount of training data.</p>
<p>Here we introduce distrbuted SINGA in following aspects:</p>
Modified: websites/staging/singa/trunk/content/docs/examples.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/examples.html (original)
+++ websites/staging/singa/trunk/content/docs/examples.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Example Models</h1>
+<hr />
<p>Different models are provided as examples to help users get familiar with SINGA. <a href="neural-net.html">Neural Network</a> gives details on the models that are supported by SINGA.</p>
<div class="section">
<div class="section">
Modified: websites/staging/singa/trunk/content/docs/frameworks.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/frameworks.html (original)
+++ websites/staging/singa/trunk/content/docs/frameworks.html Sat Sep 26 12:10:18 2015
@@ -489,9 +489,9 @@
<div id="bodyColumn" class="span10" >
<h1>Distributed Training Framework</h1>
+<hr />
<div class="section">
-<div class="section">
-<h3><a name="Cluster_Topology_Configuration"></a>Cluster Topology Configuration</h3>
+<h2><a name="Cluster_Topology_Configuration"></a>Cluster Topology Configuration</h2>
<p>Here we describe how to configure SINGA’s cluster topology to support different distributed training frameworks. The cluster topology is configured in the <tt>cluster</tt> field in <tt>JobProto</tt>. The <tt>cluster</tt> is of type <tt>ClusterProto</tt>:</p>
<div class="source">
@@ -520,12 +520,12 @@
<li><tt>server_worker_separate</tt>: separate servers and workers in different processes.</li>
</ul></div>
<div class="section">
-<h3><a name="Different_Training_Frameworks"></a>Different Training Frameworks</h3>
+<h2><a name="Different_Training_Frameworks"></a>Different Training Frameworks</h2>
<p>In SINGA, worker groups run asynchronously and workers within one group run synchronously. Users can leverage this general design to run both <b>synchronous</b> and <b>asynchronous</b> training frameworks. Here we illustrate how to configure popular distributed training frameworks in SINGA.</p>
<p><img src="../images/frameworks.png" style="width: 800px" alt="" />
<p><b> Fig.1 - Training frameworks in SINGA</b></p>
<div class="section">
-<h4><a name="Sandblaster"></a>Sandblaster</h4>
+<h3><a name="Sandblaster"></a>Sandblaster</h3>
<p>This is a <b>synchronous</b> framework used by Google Brain. Fig.2(a) shows the Sandblaster framework implemented in SINGA. Its configuration is as follows:</p>
<div class="source">
@@ -539,7 +539,7 @@
</pre></div></div>
<p>A single server group is launched to handle all requests from workers. A worker computes on its partition of the model, and only communicates with servers handling related parameters.</p></div>
<div class="section">
-<h4><a name="AllReduce"></a>AllReduce</h4>
+<h3><a name="AllReduce"></a>AllReduce</h3>
<p>This is a <b>synchronous</b> framework used by Baidu’s DeepImage. Fig.2(b) shows the AllReduce framework implemented in SINGA. Its configuration is as follows:</p>
<div class="source">
@@ -553,7 +553,7 @@
</pre></div></div>
<p>We bind each worker with a server on the same node, so that each node is responsible for maintaining a partition of parameters and collecting updates from all other nodes.</p></div>
<div class="section">
-<h4><a name="Downpour"></a>Downpour</h4>
+<h3><a name="Downpour"></a>Downpour</h3>
<p>This is a <b>asynchronous</b> framework used by Google Brain. Fig.2(c) shows the Downpour framework implemented in SINGA. Its configuration is as follows:</p>
<div class="source">
@@ -567,7 +567,7 @@
</pre></div></div>
<p>Similar to the synchronous Sandblaster, all workers send requests to a global server group. We divide workers into several worker groups, each running independently and working on parameters from the last <i>update</i> response.</p></div>
<div class="section">
-<h4><a name="Distributed_Hogwild"></a>Distributed Hogwild</h4>
+<h3><a name="Distributed_Hogwild"></a>Distributed Hogwild</h3>
<p>This is a <b>asynchronous</b> framework used by Caffe. Fig.2(d) shows the Distributed Hogwild framework implemented in SINGA. Its configuration is as follows:</p>
<div class="source">
@@ -579,7 +579,7 @@
server_worker_separate: false
}
</pre></div></div>
-<p>Each node contains a complete server group and a complete worker group. Parameter updates are done locally, so that communication cost during each training step is minimized. However, the server group must periodically synchronize with neighboring groups to improve the training convergence.</p></div></div></div>
+<p>Each node contains a complete server group and a complete worker group. Parameter updates are done locally, so that communication cost during each training step is minimized. However, the server group must periodically synchronize with neighboring groups to improve the training convergence.</p></div></div>
</div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/model-config.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/model-config.html (original)
+++ websites/staging/singa/trunk/content/docs/model-config.html Sat Sep 26 12:10:18 2015
@@ -489,12 +489,12 @@
<div id="bodyColumn" class="span10" >
<h1>Model Configuration</h1>
+<hr />
<p>SINGA uses the stochastic gradient descent (SGD) algorithm to train parameters of deep learning models. For each SGD iteration, there is a <a href="architecture.html">Worker</a> computing gradients of parameters from the NeuralNet and a <a href="">Updater</a> updating parameter values based on gradients. Hence the model configuration mainly consists these three parts. We will introduce the NeuralNet, Worker and Updater in the following paragraphs and describe the configurations for them. All model configuration is specified in the model.conf file in the user provided workspace folder. E.g., the <a class="externalLink" href="https://github.com/apache/incubator-singa/tree/master/examples/cifar10">cifar10 example folder</a> has a model.conf file.</p>
<div class="section">
+<h2><a name="NeuralNet"></a>NeuralNet</h2>
<div class="section">
-<h3><a name="NeuralNet"></a>NeuralNet</h3>
-<div class="section">
-<h4><a name="Uniform_model_neuralnet_representation"></a>Uniform model (neuralnet) representation</h4>
+<h3><a name="Uniform_model_neuralnet_representation"></a>Uniform model (neuralnet) representation</h3>
<p><img src="../images/model-categorization.png" style="width: 400px" alt="" /> Fig. 1: Deep learning model categorization</img></p>
<p>Many deep learning models have being proposed. Fig. 1 is a categorization of popular deep learning models based on the layer connections. The <a class="externalLink" href="https://github.com/apache/incubator-singa/blob/master/include/neuralnet/neuralnet.h">NeuralNet</a> abstraction of SINGA consists of multiple directly connected layers. This abstraction is able to represent models from all the three categorizations.</p>
@@ -540,7 +540,7 @@ LayerProto layer_proto_; // layer confi
<div class="source"><pre class="prettyprint">Param weight;
</pre></div></div></div>
<div class="section">
-<h4><a name="Configure_the_structure_of_a_NeuralNet_instance"></a>Configure the structure of a NeuralNet instance</h4>
+<h3><a name="Configure_the_structure_of_a_NeuralNet_instance"></a>Configure the structure of a NeuralNet instance</h3>
<p>To train a deep learning model, the first step is to write the configurations for the model structure, i.e., the layers and connections for the NeuralNet. Like <a class="externalLink" href="http://caffe.berkeleyvision.org/">Caffe</a>, we use the <a class="externalLink" href="https://developers.google.com/protocol-buffers/">Google Protocol Buffer</a> to define the configuration protocol. The <a class="externalLink" href="https://github.com/apache/incubator-singa/blob/master/src/proto/model.proto">NetProto</a> specifies the configuration fields for a NeuralNet instance,</p>
<p>message NetProto { repeated LayerProto layer = 1; … }</p>
<p>The configuration is then</p>
@@ -604,7 +604,7 @@ layer {
<p>Fig. 4 shows the model structure corresponding to the neural network configuration for the <a class="externalLink" href="https://github.com/apache/incubator-singa/blob/master/examples/cifar10/model.conf">deep convolutional model</a>.</p>
<p><img src="../images/dcnn-cifar10.png" style="width: 200px" alt="" /> Fig. 4: Neural network structure for the example DCNN model (cifar10)</img></p></div></div>
<div class="section">
-<h3><a name="Worker"></a>Worker</h3>
+<h2><a name="Worker"></a>Worker</h2>
<p>At the beginning, the Work will initialize the values of Param instances of each layer either randomly (according to user configured distribution) or loading from a <a href="">checkpoint file</a>. For each training iteration, the worker visits layers of the neural network to compute gradients of Param instances of each layer. Corresponding to the three categories of models, there are three different algorithm to compute the gradients of a neural network.</p>
<ol style="list-style-type: decimal">
@@ -668,7 +668,7 @@ layer {
</pre></div></div>
<p>All <a href="">Layer implementations</a> must implement the above two functions.</p></div>
<div class="section">
-<h3><a name="Updater"></a>Updater</h3>
+<h2><a name="Updater"></a>Updater</h2>
<p>Once the gradients of parameters are computed, the Updater will update parameter values. There are many SGD variants for updating parameters, like <a class="externalLink" href="http://arxiv.org/pdf/1212.5701v1.pdf">AdaDelta</a>, <a class="externalLink" href="http://www.magicbroom.info/Papers/DuchiHaSi10.pdf">AdaGrad</a>, <a class="externalLink" href="http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf">RMSProp</a>, <a class="externalLink" href="http://scholar.google.com/citations?view_op=view_citation&hl=en&user=DJ8Ep8YAAAAJ&citation_for_view=DJ8Ep8YAAAAJ:hkOj_22Ku90C">Nesterov</a> and SGD with momentum. The core functions of the Updater is</p>
<div class="source">
@@ -725,7 +725,7 @@ float GetLearningRate(int step);
}
</pre></div></div></div>
<div class="section">
-<h3><a name="Other_model_configuration_fields"></a>Other model configuration fields</h3>
+<h2><a name="Other_model_configuration_fields"></a>Other model configuration fields</h2>
<p>Some other important configuration fields for training a deep learning model is listed:</p>
<div class="source">
@@ -741,7 +741,7 @@ optional int32 checkpoint_frequency = 34
// checkpoint path
optional bool resume = 36 [default = false];
</pre></div></div>
-<p>The pages of <a href="checkpoint.html">checkpoint and restore</a> has more details on related fields.</p></div></div>
+<p>The pages of <a href="checkpoint.html">checkpoint and restore</a> has more details on related fields.</p></div>
</div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/programming-guide.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/programming-guide.html (original)
+++ websites/staging/singa/trunk/content/docs/programming-guide.html Sat Sep 26 12:10:18 2015
@@ -491,6 +491,7 @@
<div id="bodyColumn" class="span10" >
<h1>Programming Guide</h1>
+<hr />
<p>To submit a training job, users must provide the configuration of the four components shown in Figure 1:</p>
<ul>
Modified: websites/staging/singa/trunk/content/docs/train-one-batch.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/train-one-batch.html (original)
+++ websites/staging/singa/trunk/content/docs/train-one-batch.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
<div id="bodyColumn" class="span10" >
<h1>Train-One-Batch</h1>
+<hr />
<p>For each SGD iteration, every worker calls the <tt>TrainOneBatch</tt> function to compute gradients of parameters associated with local layers (i.e., layers dispatched to it). SINGA has implemented two algorithms for the <tt>TrainOneBatch</tt> function. Users select the corresponding algorithm for their model in the configuration.</p>
<div class="section">
<h2><a name="Basic_user_guide"></a>Basic user guide</h2>