You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by bu...@apache.org on 2015/05/12 15:05:25 UTC
svn commit: r951037 - in /websites/staging/singa/trunk/content: ./ docs/
Author: buildbot
Date: Tue May 12 13:05:25 2015
New Revision: 951037
Log:
Staging update by buildbot for singa
Added:
websites/staging/singa/trunk/content/docs/code-structure.html
websites/staging/singa/trunk/content/docs/neuralnet-partition.html
websites/staging/singa/trunk/content/docs/programming-model.html
websites/staging/singa/trunk/content/introduction.html
Modified:
websites/staging/singa/trunk/content/ (props changed)
websites/staging/singa/trunk/content/community.html
websites/staging/singa/trunk/content/docs.html
websites/staging/singa/trunk/content/docs/architecture.html
websites/staging/singa/trunk/content/docs/communication.html
websites/staging/singa/trunk/content/docs/installation.html
websites/staging/singa/trunk/content/index.html
websites/staging/singa/trunk/content/quick-start.html
Propchange: websites/staging/singa/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Tue May 12 13:05:25 2015
@@ -1 +1 @@
-1678870
+1678931
Modified: websites/staging/singa/trunk/content/community.html
==============================================================================
--- websites/staging/singa/trunk/content/community.html (original)
+++ websites/staging/singa/trunk/content/community.html Tue May 12 13:05:25 2015
@@ -53,7 +53,10 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
<ul class="dropdown-menu">
- <li> <a href="index.html" title="Introduction">Introduction</a>
+ <li> <a href="index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="introduction.html" title="Introduction">Introduction</a>
</li>
<li> <a href="quick-start.html" title="Quick Start">Quick Start</a>
@@ -72,6 +75,15 @@
<li> <a href="docs/communication.html" title="Communication">Communication</a>
</li>
+
+ <li> <a href="docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
</ul>
</li>
<li class="dropdown">
@@ -135,7 +147,14 @@
<li>
- <a href="index.html" title="Introduction">
+ <a href="index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="introduction.html" title="Introduction">
<span class="none"></span>
Introduction</a>
</li>
@@ -168,6 +187,27 @@
<span class="none"></span>
Communication</a>
</li>
+
+ <li>
+
+ <a href="docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
<li class="nav-header">External Links</li>
<li>
@@ -188,7 +228,7 @@
<div class="clear"></div>
<div class="clear"></div>
<a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
- <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" width="150" />
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
</a>
</div>
</div>
@@ -197,8 +237,9 @@
<div id="bodyColumn" class="span10" >
- <h1>Community</h1>
-<hr />
+ <div class="section">
+<h2><a name="Community"></a>Community</h2>
+<hr /></div>
</div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs.html
==============================================================================
--- websites/staging/singa/trunk/content/docs.html (original)
+++ websites/staging/singa/trunk/content/docs.html Tue May 12 13:05:25 2015
@@ -53,7 +53,10 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
<ul class="dropdown-menu">
- <li> <a href="index.html" title="Introduction">Introduction</a>
+ <li> <a href="index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="introduction.html" title="Introduction">Introduction</a>
</li>
<li> <a href="quick-start.html" title="Quick Start">Quick Start</a>
@@ -72,6 +75,15 @@
<li> <a href="docs/communication.html" title="Communication">Communication</a>
</li>
+
+ <li> <a href="docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
</ul>
</li>
<li class="dropdown">
@@ -135,7 +147,14 @@
<li>
- <a href="index.html" title="Introduction">
+ <a href="index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="introduction.html" title="Introduction">
<span class="none"></span>
Introduction</a>
</li>
@@ -168,6 +187,27 @@
<span class="none"></span>
Communication</a>
</li>
+
+ <li>
+
+ <a href="docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
<li class="nav-header">External Links</li>
<li>
@@ -188,7 +228,7 @@
<div class="clear"></div>
<div class="clear"></div>
<a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
- <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" width="150" />
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
</a>
</div>
</div>
@@ -208,6 +248,12 @@
<li><a href="docs/architecture.html">System Architecture</a></li>
<li><a href="docs/communication.html">Communication</a></li>
+
+<li><a href="docs/code-structure.html">Code Structure</a></li>
+
+<li><a href="docs/neuralnet-partition.html">Neural Network Partition</a></li>
+
+<li><a href="docs/programming-model.html">Programming Model</a></li>
</ul></div>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/architecture.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/architecture.html (original)
+++ websites/staging/singa/trunk/content/docs/architecture.html Tue May 12 13:05:25 2015
@@ -53,7 +53,10 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
<ul class="dropdown-menu">
- <li> <a href="../index.html" title="Introduction">Introduction</a>
+ <li> <a href="../index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="../introduction.html" title="Introduction">Introduction</a>
</li>
<li> <a href="../quick-start.html" title="Quick Start">Quick Start</a>
@@ -72,6 +75,15 @@
<li> <a href="../docs/communication.html" title="Communication">Communication</a>
</li>
+
+ <li> <a href="../docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="../docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
</ul>
</li>
<li class="dropdown">
@@ -135,7 +147,14 @@
<li>
- <a href="../index.html" title="Introduction">
+ <a href="../index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="../introduction.html" title="Introduction">
<span class="none"></span>
Introduction</a>
</li>
@@ -166,6 +185,27 @@
<span class="none"></span>
Communication</a>
</li>
+
+ <li>
+
+ <a href="../docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
<li class="nav-header">External Links</li>
<li>
@@ -186,7 +226,7 @@
<div class="clear"></div>
<div class="clear"></div>
<a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
- <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" width="150" />
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
</a>
</div>
</div>
Added: websites/staging/singa/trunk/content/docs/code-structure.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/code-structure.html (added)
+++ websites/staging/singa/trunk/content/docs/code-structure.html Tue May 12 13:05:25 2015
@@ -0,0 +1,330 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2015-05-12
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20150512" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Code Structure</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="../introduction.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../docs/installation.html" title="Installation">Installation</a>
+</li>
+
+ <li> <a href="../docs/architecture.html" title="System Architecture">System Architecture</a>
+</li>
+
+ <li> <a href="../docs/communication.html" title="Communication">Communication</a>
+</li>
+
+ <li> <a href="../docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="../docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../images/singa-logo.png" alt="Apache SINGA" width="150"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../images/singa-title.png" alt="Apache SINGA" width="600"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Code Structure</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="../introduction.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../docs/installation.html" title="Installation">
+ <span class="none"></span>
+ Installation</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/architecture.html" title="System Architecture">
+ <span class="none"></span>
+ System Architecture</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/communication.html" title="Communication">
+ <span class="none"></span>
+ Communication</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><span class="none"></span>Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <div class="section">
+<h2><a name="Code_Structure"></a>Code Structure</h2>
+<hr />
+<div class="section">
+<h3><a name="Worker_Side"></a>Worker Side</h3>
+<div class="section">
+<h4><a name="Main_Classes"></a>Main Classes</h4>
+<p><img src="../images/code-structure/main.jpg" style="width:70%;" align="center" alt="" /></p>
+
+<ul>
+
+<li><b>Worker</b>: start the solver to conduct training or resume from previous training snapshots.</li>
+
+<li><b>Solver</b>: construct the neural network and run training algorithms over it. Validation and testing is also done by the solver along the training.</li>
+
+<li><b>TableDelegate</b>: delegate for the parameter table physically stored in parameter servers. it runs a thread to communicate with table servers for parameter transferring.</li>
+
+<li><b>Net</b>: the neural network consists of multiple layers constructed from input configuration file.</li>
+
+<li><b>Layer</b>: the core abstraction, read data (neurons) from connecting layers, and compute the data of itself according to layer specific ComputeFeature functions. Data from the bottom layer is forwarded layer by layer to the top.</li>
+</ul></div>
+<div class="section">
+<h4><a name="Data_types"></a>Data types</h4>
+<p><img src="../images/code-structure/layer.jpg" style="width:90%;" align="center" alt="" /></p>
+
+<ul>
+
+<li><b>ComputeFeature</b>: read data (neurons) from in-coming layers, and compute the data of itself according to layer type. This function can be overrided to implement different types layers.</li>
+
+<li><b>ComputeGradient</b>: read gradients (and data) from in-coming layers and compute gradients of parameters and data w.r.t the learning objective (loss).</li>
+</ul>
+<p>We adpat the implementation for <b>PoolingLayer</b>, <b>Im2colLayer</b> and <b>LRNLayer</b> from <a class="externalLink" href="http://caffe.berkeleyvision.org/">Caffe</a>.</p>
+<p><img src="../images/code-structure/darray.jpg" style="width:55%;" align="center" alt="" /></p>
+
+<ul>
+
+<li><b>DArray</b>: provide the abstraction of distributed array on multiple nodes, supporting array/matrix operations and element-wise operations. Users can use it as a local structure.</li>
+
+<li><b>LArray</b>: the local part for the DArray. Each LArray is treated as an independent array, and support all array-related operations.</li>
+
+<li><b>MemSpace</b>: manage the memory used by DArray. Distributed memory are allocated and managed by armci. Multiple DArray can share a same MemSpace, the memory will be released when no DArray uses it anymore.</li>
+
+<li><b>Partition</b>: maintain both global shape and local partition information. used when two DArray are going to interact.</li>
+
+<li><b>Shape</b>: basic class for representing the scope of a DArray/LArray</li>
+
+<li><b>Range</b>: basic class for representing the scope of a Partition</li>
+</ul></div></div>
+<div class="section">
+<h3><a name="Parameter_Server"></a>Parameter Server</h3>
+<div class="section">
+<h4><a name="Main_classes"></a>Main classes</h4>
+<p><img src="../images/code-structure/uml.jpg" style="width:90%;" align="center" alt="" /></p>
+
+<ul>
+
+<li><b>NetworkService</b>: provide access to the network (sending and receiving messages). It maintains a queue for received messages, implemented by NetworkQueue.</li>
+
+<li><b>RequestDispatcher</b>: pick up next message (request) from the queue, and invoked a method (callback) to process them.</li>
+
+<li><b>TableServer</b>: provide access to the data table (parameters). Register callbacks for different types of requests to RequestDispatcher.</li>
+
+<li><b>GlobalTable</b>: implement the table. Data is partitioned into multiple Shard objects per table. User-defined consistency model supported by extending TableServerHandler for each table.</li>
+</ul></div>
+<div class="section">
+<h4><a name="Data_types"></a>Data types</h4>
+<p><img src="../images/code-structure/type.jpg" style="width:400px;" align="middle" alt="" /></p>
+<p>Table related messages are either of type <b>RequestBase</b> which contains different types of request, or of type <b>TableData</b> containing a key-value tuple.</p></div>
+<div class="section">
+<h4><a name="Control_flow_and_thread_model"></a>Control flow and thread model</h4>
+<p><img src="../images/code-structure/threads.jpg" alt="uml" /></p>
+<p>The figure above shows how a GET request sent from a worker is processed by the table server. The control flow for other types of requests is similar. At the server side, there are at least 3 threads running at any time: two by NetworkService for sending and receiving message, and at least one by the RequestDispatcher for dispatching requests.</p></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+ <p >Copyright © 2015
+ <a href="http://www.apache.org">The Apache Software Foundation</a>.
+ All rights reserved.
+
+ </p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Modified: websites/staging/singa/trunk/content/docs/communication.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/communication.html (original)
+++ websites/staging/singa/trunk/content/docs/communication.html Tue May 12 13:05:25 2015
@@ -53,7 +53,10 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
<ul class="dropdown-menu">
- <li> <a href="../index.html" title="Introduction">Introduction</a>
+ <li> <a href="../index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="../introduction.html" title="Introduction">Introduction</a>
</li>
<li> <a href="../quick-start.html" title="Quick Start">Quick Start</a>
@@ -72,6 +75,15 @@
<li> <a href="../docs/communication.html" title="Communication">Communication</a>
</li>
+
+ <li> <a href="../docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="../docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
</ul>
</li>
<li class="dropdown">
@@ -135,7 +147,14 @@
<li>
- <a href="../index.html" title="Introduction">
+ <a href="../index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="../introduction.html" title="Introduction">
<span class="none"></span>
Introduction</a>
</li>
@@ -166,6 +185,27 @@
<a href="#"><span class="none"></span>Communication</a>
</li>
+
+ <li>
+
+ <a href="../docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
<li class="nav-header">External Links</li>
<li>
@@ -186,7 +226,7 @@
<div class="clear"></div>
<div class="clear"></div>
<a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
- <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" width="150" />
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
</a>
</div>
</div>
Modified: websites/staging/singa/trunk/content/docs/installation.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/installation.html (original)
+++ websites/staging/singa/trunk/content/docs/installation.html Tue May 12 13:05:25 2015
@@ -53,7 +53,10 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
<ul class="dropdown-menu">
- <li> <a href="../index.html" title="Introduction">Introduction</a>
+ <li> <a href="../index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="../introduction.html" title="Introduction">Introduction</a>
</li>
<li> <a href="../quick-start.html" title="Quick Start">Quick Start</a>
@@ -72,6 +75,15 @@
<li> <a href="../docs/communication.html" title="Communication">Communication</a>
</li>
+
+ <li> <a href="../docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="../docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
</ul>
</li>
<li class="dropdown">
@@ -135,7 +147,14 @@
<li>
- <a href="../index.html" title="Introduction">
+ <a href="../index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="../introduction.html" title="Introduction">
<span class="none"></span>
Introduction</a>
</li>
@@ -166,6 +185,27 @@
<span class="none"></span>
Communication</a>
</li>
+
+ <li>
+
+ <a href="../docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
<li class="nav-header">External Links</li>
<li>
@@ -186,7 +226,7 @@
<div class="clear"></div>
<div class="clear"></div>
<a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
- <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" width="150" />
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
</a>
</div>
</div>
@@ -294,16 +334,20 @@ zeromq zeromq lib
<p>After the execution, czmq will be installed in /usr/local while zeromq is installed in /usr/local/zeromq.</p></div>
<div class="section">
<h3><a name="FAQ"></a>FAQ</h3>
-<p>Q1:While compiling Singa and installing glog on max OS X, I get fatal error “‘ext/slist’ file not found” A1:You may install glog individually and try command :</p>
+<div class="section">
+<h4><a name="While_compiling_Singa_and_installing_glog_on_max_OS_X_I_get_fatal_error_extslist_file_not_found."></a>While compiling Singa and installing glog on max OS X, I get fatal error “‘ext/slist’ file not found”.</h4>
+<p>You may install glog individually and try command:</p>
<div class="source">
<div class="source"><pre class="prettyprint">$ make CFLAGS='-stdlib=libstdc++' CXXFLAGS='stdlib=libstdc++'
-</pre></div></div>
-<p>Q2:While compiling Singa, I get error “SSE2 instruction set not enabled” A2:You can try following command:</p>
+</pre></div></div></div>
+<div class="section">
+<h4><a name="While_compiling_Singa_I_get_error_SSE2_instruction_set_not_enabled."></a>While compiling Singa, I get error “SSE2 instruction set not enabled”.</h4>
+<p>You can try following command:</p>
<div class="source">
<div class="source"><pre class="prettyprint">$ make CFLAGS='-msse2' CXXFLAGS='-msse2'
-</pre></div></div></div></div>
+</pre></div></div></div></div></div>
</div>
</div>
</div>
Added: websites/staging/singa/trunk/content/docs/neuralnet-partition.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/neuralnet-partition.html (added)
+++ websites/staging/singa/trunk/content/docs/neuralnet-partition.html Tue May 12 13:05:25 2015
@@ -0,0 +1,270 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2015-05-12
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20150512" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Neural Network Partition</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="../introduction.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../docs/installation.html" title="Installation">Installation</a>
+</li>
+
+ <li> <a href="../docs/architecture.html" title="System Architecture">System Architecture</a>
+</li>
+
+ <li> <a href="../docs/communication.html" title="Communication">Communication</a>
+</li>
+
+ <li> <a href="../docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="../docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../images/singa-logo.png" alt="Apache SINGA" width="150"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../images/singa-title.png" alt="Apache SINGA" width="600"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Neural Network Partition</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="../introduction.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../docs/installation.html" title="Installation">
+ <span class="none"></span>
+ Installation</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/architecture.html" title="System Architecture">
+ <span class="none"></span>
+ System Architecture</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/communication.html" title="Communication">
+ <span class="none"></span>
+ Communication</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><span class="none"></span>Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <div class="section">
+<h2><a name="Neural_Network_Partition"></a>Neural Network Partition</h2>
+<hr />
+<p>The purposes of partitioning neural network is to distribute the partitions onto different working units (e.g., threads or nodes, called workers in this article) and parallelize the processing. Another reason for partition is to handle large neural network which cannot be hold in a single node. For instance, to train models against images with high resolution we need large neural networks (in terms of training parameters).</p>
+<p>Since <i>Layer</i> is the first class citizen in SIGNA, we do the partition against layers. Specifically, we support partitions at two levels. First, users can configure the location (i.e., worker ID) of each layer. In this way, users assign one worker for each layer. Secondly, for one layer, we can partition its neurons or partition the instances (e.g, images). They are called layer partition and data partition respectively. We illustrate the two types of partitions using an simple convolutional neural network.</p>
+<p><img src="../images/conv-mnist.png" align="center" width="200px" alt="" /></p>
+<p>The above figure shows a convolutional neural network without any partition. It has 8 layers in total (one rectangular represents one layer). The first layer is DataLayer (data) which reads data from local disk files/databases (or HDFS). The second layer is a MnistLayer which parses the records from MNIST data to get the pixels of a batch of 28 images (each image is of size 28x28). The LabelLayer (label) parses the records to get the label of each image in the batch. The ConvolutionalLayer (conv1) transforms the input image to the shape of 8x27x27. The ReLULayer (relu1) conducts elementwise transformations. The PoolingLayer (pool1) sub-samples the images. The fc1 layer is fully connected with pool1 layer. It mulitplies each image with a weight matrix to generate a 10 dimension hidden feature which is then normalized by a SoftmaxLossLayer to get the prediction.</p>
+<p><img src="../images/conv-mnist-datap.png" align="center" width="400px" alt="" /></p>
+<p>The above figure shows the convolutional neural network after partitioning all layers except the DataLayer and ParserLayers, into 3 partitions using data partition. The read layers process 4 images of the batch, the black and blue layers process 2 images respectively. Some helper layers, i.e., SliceLayer, ConcateLayer, BridgeSrcLayer, BridgeDstLayer and SplitLayer, are added automatically by our partition algorithm. Layers of the same color resident in the same worker. There would be data transferring across different workers at the boundary layers (i.e., BridgeSrcLayer and BridgeDstLayer), e.g., between s-slice-mnist-conv1 and d-slice-mnist-conv1.</p>
+<p><img src="../images/conv-mnist-layerp.png" align="center" width="400px" alt="" /></p>
+<p>The above figure shows the convolutional neural network after partitioning all layers except the DataLayer and ParserLayers, into 2 partitions using layer partition. We can see that each layer processes all 8 images from the batch. But different partitions process different part of one image. For instance, the layer conv1-00 process only 4 channels. The other 4 channels are processed by conv1-01 which residents in another worker.</p>
+<p>Since the partition is done at the layer level, we can apply different partitions for different layers to get a hybrid partition for the whole neural network. Moreover, we can also specify the layer locations to locate different layers to different workers.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+ <p >Copyright © 2015
+ <a href="http://www.apache.org">The Apache Software Foundation</a>.
+ All rights reserved.
+
+ </p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: websites/staging/singa/trunk/content/docs/programming-model.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/programming-model.html (added)
+++ websites/staging/singa/trunk/content/docs/programming-model.html Tue May 12 13:05:25 2015
@@ -0,0 +1,359 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2015-05-12
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20150512" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Programming Model</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="../introduction.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../docs/installation.html" title="Installation">Installation</a>
+</li>
+
+ <li> <a href="../docs/architecture.html" title="System Architecture">System Architecture</a>
+</li>
+
+ <li> <a href="../docs/communication.html" title="Communication">Communication</a>
+</li>
+
+ <li> <a href="../docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="../docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../images/singa-logo.png" alt="Apache SINGA" width="150"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../images/singa-title.png" alt="Apache SINGA" width="600"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Programming Model</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="../introduction.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../docs/installation.html" title="Installation">
+ <span class="none"></span>
+ Installation</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/architecture.html" title="System Architecture">
+ <span class="none"></span>
+ System Architecture</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/communication.html" title="Communication">
+ <span class="none"></span>
+ Communication</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><span class="none"></span>Programming Model</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <div class="section">
+<h2><a name="Programming_Model"></a>Programming Model</h2>
+<p>We describe the programming model of SINGA in this article. Base data structures are introduced firstly, and then we show examples for users with different levels of deep learning background.</p>
+<div class="section">
+<h3><a name="Base_Data_Structures"></a>Base Data Structures</h3>
+<div class="section">
+<h4><a name="Layer"></a>Layer</h4>
+<p>Layer is the first class citizen in SINGA. Users construct their deep learning models by creating layer objects and combining them. SINGA takes care of running BackPropagation (or Contrastive Divergence) algorithms to calculate the gradients for parameters and calling <a href="#updater">Updaters</a> to update them.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">class Layer{
+ /**
+ * Setup layer properties.
+ * Setup the shapes for data and parameters, also setup some properties
+ * based on the layer configuration and connected src layers.
+ * @param conf user defined layer configuration of type [LayerProto](#netproto)
+ * @param srclayers layers connecting to this layer
+ */
+ Setup(conf, srclayers);
+ /**
+ * Setup the layer properties.
+ * This function is called if the model is partitioned due to distributed
+ * training. Shape of the layer is already set by the partition algorithm,
+ * and is passed in to set other properties.
+ * @param conf user defined layer configuration of type [LayerProto](#netproto)
+ * @param shape shape set by partition algorithm (for distributed training).
+ * @param srclayers layers connecting to this layer
+ */
+ SetupAfterPartition(conf, shape, srclayers);
+ /**
+ * Compute features of this layer based on connected layers.
+ * BP and CD will call this to calculate gradients
+ * @param training boolean phase indicator for training or test
+ * @param srclayers layers connecting to this layer
+ */
+ ComputeFeature(training, srclayers);
+ /**
+ * Compute gradients for parameters and connected layers.
+ * BP and CD will call this to calculate gradients
+ * @param srclayers layers connecting to this layer.
+ */
+ ComputeGradient(srclayers)=0;
+}
+</pre></div></div>
+<p>The above pseudo code shows the base Layer class. Users override these methods to implement their own layer classes. For example, we have implemented popular layers like ConvolutionLayer, InnerProductLayer. We also provide a DataLayer which is a base layer for loading (and prefetching) data from disk or HDFS. A base ParserLayer is created for parsing the raw data and convert it into records that are recognizable by SINGA.</p></div>
+<div class="section">
+<h4><a name="NetProto"></a>NetProto</h4>
+<p>Since deep learning models consist of multiple layers. The model structure includes the properties of each layer and the connections between layers. SINGA uses google protocol buffer for users to configure the model structure. The protocol buffer message for the model structure is defined as:</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">NetProto{
+ repeated LayerProto layer;
+}
+
+LayerProto{
+ string name; // user defined layer name for displaying
+ string type; // One layer class has a unique type.
+ repeated string srclayer_name; // connected layer names;
+ repeated ParamProto param; // parameter configurations
+ ...
+}
+</pre></div></div>
+<p>Users can create a plain text file and fill it with the configurations. SINGA parses it according to user provided path.</p></div>
+<div class="section">
+<h4><a name="Param"></a>Param</h4>
+<p>The Param class is shown below. Users do not need to extend the Param class for most cases. We make it a base class just for future extension. For example, if a new initialization trick is proposed in the future, we can override the <tt>Init</tt> method to implement it.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">Param{
+ /**
+ * Set properties of the parameter.
+ * @param conf user defined parameter configuration of type ParamProto
+ * @param shape shape of the parameter
+ Setup(conf, shape);
+ /**
+ * Initialize the data of the parameter.
+ /
+ Init();
+ ...// methods to handle synchronizations with parameter servers and other workers
+}
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Updater"></a>Updater</h4>
+<p>There are many SGD extensions for updating parameters, like <a class="externalLink" href="http://arxiv.org/pdf/1212.5701v1.pdf">AdaDelta</a>, <a class="externalLink" href="http://www.magicbroom.info/Papers/DuchiHaSi10.pdf">AdaGrad</a>, <a class="externalLink" href="http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf">RMSProp</a>, <a class="externalLink" href="http://scholar.google.com/citations?view_op=view_citation&hl=en&user=DJ8Ep8YAAAAJ&citation_for_view=DJ8Ep8YAAAAJ:hkOj_22Ku90C">Nesterov</a> and SGD with momentum. We provide a base Updater to deal with these algorithms. New parameter updating algorithms can be added by extending the base Updater.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">Updater{
+ /**
+ * @param proto user configuration for the updater.
+ Init(conf);
+ /**
+ * Update parameter based on its gradient
+ * @param step training step
+ * @param param the Param object
+ */
+ Update(step, param);
+}
+</pre></div></div></div></div>
+<div class="section">
+<h3><a name="Examples"></a>Examples</h3>
+<p>The <a href="..">MLP example</a> shows how to configure the model through google protocol buffer.</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+ <p >Copyright © 2015
+ <a href="http://www.apache.org">The Apache Software Foundation</a>.
+ All rights reserved.
+
+ </p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Modified: websites/staging/singa/trunk/content/index.html
==============================================================================
--- websites/staging/singa/trunk/content/index.html (original)
+++ websites/staging/singa/trunk/content/index.html Tue May 12 13:05:25 2015
@@ -53,7 +53,10 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
<ul class="dropdown-menu">
- <li> <a href="index.html" title="Introduction">Introduction</a>
+ <li> <a href="index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="introduction.html" title="Introduction">Introduction</a>
</li>
<li> <a href="quick-start.html" title="Quick Start">Quick Start</a>
@@ -72,6 +75,15 @@
<li> <a href="docs/communication.html" title="Communication">Communication</a>
</li>
+
+ <li> <a href="docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
</ul>
</li>
<li class="dropdown">
@@ -135,11 +147,18 @@
<li class="active">
- <a href="#"><span class="none"></span>Introduction</a>
+ <a href="#"><span class="none"></span>Welcome</a>
</li>
<li>
+ <a href="introduction.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
<a href="quick-start.html" title="Quick Start">
<span class="none"></span>
Quick Start</a>
@@ -166,6 +185,27 @@
<span class="none"></span>
Communication</a>
</li>
+
+ <li>
+
+ <a href="docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
<li class="nav-header">External Links</li>
<li>
@@ -186,7 +226,7 @@
<div class="clear"></div>
<div class="clear"></div>
<a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
- <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" width="150" />
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
</a>
</div>
</div>
@@ -217,7 +257,7 @@
<li>Please subscribe to our development mailing list <a class="externalLink" href="mailto:dev@singa.incubator.apache.org">dev@singa.incubator.apache.org</a>.</li>
-<li>If you find any issues using SIGNA, please report it to the <a class="externalLink" href="https://issues.apache.org/jira/browse/singa">Issue Tracker</a>.</li>
+<li>If you find any issues using SINGA, please report it to the <a class="externalLink" href="https://issues.apache.org/jira/browse/singa">Issue Tracker</a>.</li>
</ul>
<p>More details on contributing to SINGA is describe <a href="community.html">here</a>.</p></div>
<div class="section">
Added: websites/staging/singa/trunk/content/introduction.html
==============================================================================
--- websites/staging/singa/trunk/content/introduction.html (added)
+++ websites/staging/singa/trunk/content/introduction.html Tue May 12 13:05:25 2015
@@ -0,0 +1,312 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2015-05-12
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20150512" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Introduction</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="introduction.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="docs/installation.html" title="Installation">Installation</a>
+</li>
+
+ <li> <a href="docs/architecture.html" title="System Architecture">System Architecture</a>
+</li>
+
+ <li> <a href="docs/communication.html" title="Communication">Communication</a>
+</li>
+
+ <li> <a href="docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="images/singa-logo.png" alt="Apache SINGA" width="150"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="images/singa-title.png" alt="Apache SINGA" width="600"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Introduction</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><span class="none"></span>Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="docs/installation.html" title="Installation">
+ <span class="none"></span>
+ Installation</a>
+ </li>
+
+ <li>
+
+ <a href="docs/architecture.html" title="System Architecture">
+ <span class="none"></span>
+ System Architecture</a>
+ </li>
+
+ <li>
+
+ <a href="docs/communication.html" title="Communication">
+ <span class="none"></span>
+ Communication</a>
+ </li>
+
+ <li>
+
+ <a href="docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<hr />
+<p>SINGA is a distributed deep learning platform, for training large-scale deep learning models. Our design is driven by two key observations. First, the structures and training algorithms of deep learning models can be expressed using simple abstractions, e.g., the layer. SINGA allows users to write their own training algorithms by exposing intuitive programming abstractions and hiding complex details pertaining distributed execution of the training. Specifically, our programming model consists of data objects (layer and network) that define the model, and of computation functions over the data objects. Our second observation is that there are multiple approaches to partitioning the model and the training data onto multiple machines to achieve model parallelism, data parallelism or both. Each approach incurs different communication and synchronization overhead which directly affects the system’s scalability. We analyze the fundamental trade-offs of existing parallelism appro
aches, and propose an optimization algorithm that generates the parallelism scheme with minimal overhead.</p>
+<div class="section">
+<h3><a name="Goals_and_Principles"></a>Goals and Principles</h3>
+<div class="section">
+<h4><a name="Goals"></a>Goals</h4>
+
+<ul>
+
+<li>Scalability: A distributed platform that can scale to a large model and training dataset, e.g., 1 Billion parameters and 10M images.</li>
+
+<li>Usability: To provide abstraction and easy to use interface so that users can implement their deep learning model/algorithm without much awareness of the underlying distributed platform.</li>
+
+<li>Extensibility: We try to make SINGA extensible for implementing different consistency models, training algorithms and deep learning models.</li>
+</ul></div>
+<div class="section">
+<h4><a name="Principles"></a>Principles</h4>
+<p>To achieve the scalability goal, we parallelize the computation across a cluster of nodes by the following partitioning approaches:</p>
+
+<ul>
+
+<li>Model Partition—one model replica spreads across multiple machines to handle large models, which have too many parameters to be kept in the memory of a single machine. Overhead: synchronize layer data across machines within one model replica Partition.</li>
+
+<li>Data Partition—one model replica trains against a partition of the whole training dataset. This approach can handle large training dataset. Overhead: synchronize parameters among model replicas.</li>
+
+<li>Hybrid Partition—exploit a cost model to find optimal model and data partitions which would reduce both overheads.</li>
+</ul>
+<p>To achieve the usability goal, we propose our programming model with the following two major considerations:</p>
+
+<ul>
+
+<li>Extract common data structures and operations for deep learning training algorithms, i.e., Back Propagation and Contrastive Divergence. Users implement their models by inheriting these data structures and overriding the operations.</li>
+
+<li>Manage model partition and data partition automatically through distributed array. Users write code against the distributed array, without much awareness of the array partition (which part is stored on which machine).</li>
+</ul>
+<p>Considering extensibility, we make our core data structures (e.g., Layer) and operations general enough for programmers to override.</p></div></div>
+<div class="section">
+<h3><a name="System_Overview"></a>System Overview</h3>
+<p><img src="images/software_stack.jpg" alt="SINGA software stack" /></p>
+<p>Three goals are considered in designing SINGA, namely ease of use, scalability and extensibility. We will introduce them together with the software stack as shown in the above figure. Algorithms for deep learning models are complex to code and hard to train. To make it ease of use, we provide a simple concept ‘Layer’ to construct deep complex models. Built-in Layer implementations include common layers, e.g., convolution layer and fully connected layer. Users can configure their models by combining these built-in layers through web interface or configuration files. Once the model and training data is configured, we start SINGA to conduct the training using the standard training algorithm (Back-Propagation,BP or Contrastive Divergence, CD) on a cluster of nodes and visualize the training performance to users (e.g., through web interface). Advanced users can also implement their own layers by overloading the base Layer class through Python, Matlab, etc wrappers. Distr
ibutedArray is proposed for easy array operations that are heavily used for realizing layer logics. SINGA manages the distributed arrays (stored across multiple nodes) automatically and efficiently based on MPI. Training scalability is achieved by partitioning the training data and model onto multiple computing nodes and parallelizing the computation. A logically centralized parameter server maintains the model parameters in a ParameterTable. Computing nodes work according to the consistency policy and send information to the parameter server which updates the parameters based on SGD (stochastic gradient descent) algorithms. Besides the Layer class, other components like SGD algorithms and consistency module are also extensible. <!-- -
+The above figure shows the basic components of SINGA. It starts training a deep
+learning model by parsing a model configuration, which specifies the layer and
+network structure at the every worker node. After that, it initializes the table servers and starts
+workers to run their tasks. Each table server maintains a partition (i.e., a set
+of rows) of a distributed parameter table where model parameters are stored.
+Worker groups consisting one or more worker nodes run in parallel to compute the
+gradients of parameters. In one iteration, every group fetches fresh parameters
+from the table servers, runs BP or CD algorithm to compute gradients against a
+mini-batch from the local data shard (a partition of the training dataset), and
+then sends gradients to the table servers. The data shard is created by loading
+training data from HDFS off-line. The master monitors the training progress and
+stops the workers and table servers once the model has converged to a given loss. --></p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+ <p >Copyright © 2015
+ <a href="http://www.apache.org">The Apache Software Foundation</a>.
+ All rights reserved.
+
+ </p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Modified: websites/staging/singa/trunk/content/quick-start.html
==============================================================================
--- websites/staging/singa/trunk/content/quick-start.html (original)
+++ websites/staging/singa/trunk/content/quick-start.html Tue May 12 13:05:25 2015
@@ -53,7 +53,10 @@
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
<ul class="dropdown-menu">
- <li> <a href="index.html" title="Introduction">Introduction</a>
+ <li> <a href="index.html" title="Welcome">Welcome</a>
+</li>
+
+ <li> <a href="introduction.html" title="Introduction">Introduction</a>
</li>
<li> <a href="quick-start.html" title="Quick Start">Quick Start</a>
@@ -72,6 +75,15 @@
<li> <a href="docs/communication.html" title="Communication">Communication</a>
</li>
+
+ <li> <a href="docs/code-structure.html" title="Code Structure">Code Structure</a>
+</li>
+
+ <li> <a href="docs/neuralnet-partition.html" title="Neural Network Partition">Neural Network Partition</a>
+</li>
+
+ <li> <a href="docs/programming-model.html" title="Programming Model">Programming Model</a>
+</li>
</ul>
</li>
<li class="dropdown">
@@ -135,7 +147,14 @@
<li>
- <a href="index.html" title="Introduction">
+ <a href="index.html" title="Welcome">
+ <span class="none"></span>
+ Welcome</a>
+ </li>
+
+ <li>
+
+ <a href="introduction.html" title="Introduction">
<span class="none"></span>
Introduction</a>
</li>
@@ -166,6 +185,27 @@
<span class="none"></span>
Communication</a>
</li>
+
+ <li>
+
+ <a href="docs/code-structure.html" title="Code Structure">
+ <span class="none"></span>
+ Code Structure</a>
+ </li>
+
+ <li>
+
+ <a href="docs/neuralnet-partition.html" title="Neural Network Partition">
+ <span class="none"></span>
+ Neural Network Partition</a>
+ </li>
+
+ <li>
+
+ <a href="docs/programming-model.html" title="Programming Model">
+ <span class="none"></span>
+ Programming Model</a>
+ </li>
<li class="nav-header">External Links</li>
<li>
@@ -186,7 +226,7 @@
<div class="clear"></div>
<div class="clear"></div>
<a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
- <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" width="150" />
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
</a>
</div>
</div>
@@ -213,7 +253,7 @@ git clone https://github.com/apache/incu
<div class="source"><pre class="prettyprint">./configure
make
</pre></div></div>
-<p>If there are dependent libraries missing, please refer to [installation]({{ BASE_PATH }}{% post_url /docs/2015-01-20-installation %}) page for guidance on installing them. After successful compilation, the libsinga.so and singa executable will be built into the build folder.</p></div>
+<p>If there are dependent libraries missing, please refer to <a href="docs/installation.html">installation</a> page for guidance on installing them. After successful compilation, the libsinga.so and singa executable will be built into the build folder.</p></div>
<div class="section">
<h3><a name="Run_in_standalone_mode"></a>Run in standalone mode</h3>
<p>Running SINGA in standalone mode is on the contrary of running it on Mesos or YARN. For standalone mode, users have to manage the resources manually. For instance, they have to prepare a host file containing all running nodes. There is no management on CPU and memory resources, hence SINGA consumes as much CPU and memory resources as it needs.</p>
@@ -243,7 +283,7 @@ nworkers_per_group: 1
nserver_groups: 1
nservers_per_group: 1
</pre></div></div>
-<p>One worker group trains against one partition of the training dataset. If <i>nworker_groups</i> is set to 1, then there is no data partitioning. One worker runs over a partition of the model. If <i>nworkers_per_group</i> is set to 1, then there is no model partitioning. More details on the cluster configuration are described in the <a href="">System Architecture</a> page.</p>
+<p>One worker group trains against one partition of the training dataset. If <i>nworker_groups</i> is set to 1, then there is no data partitioning. One worker runs over a partition of the model. If <i>nworkers_per_group</i> is set to 1, then there is no model partitioning. More details on the cluster configuration are described in the <a href="docs/architecture.html">System Architecture</a> page.</p>
<p>Start the training by running:</p>
<div class="source">