You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by bu...@apache.org on 2016/04/20 07:12:08 UTC
svn commit: r986140 [7/34] - in /websites/staging/singa/trunk/content: ./
community/ develop/ docs/ docs/jp/ docs/kr/ docs/zh/ releases/ v0.1.0/
v0.2.0/ v0.2.0/jp/ v0.2.0/kr/ v0.2.0/zh/ v0.3.0/ v0.3.0/jp/ v0.3.0/kr/
v0.3.0/zh/
Added: websites/staging/singa/trunk/content/v0.3.0/frameworks.html
==============================================================================
--- websites/staging/singa/trunk/content/v0.3.0/frameworks.html (added)
+++ websites/staging/singa/trunk/content/v0.3.0/frameworks.html Wed Apr 20 05:12:03 2016
@@ -0,0 +1,431 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-04-20
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160420" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Distributed Training Framework</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+
+<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
+
+
+<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
+
+
+<meta content="Apache SINGA" name="author"/>
+
+
+<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/>
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../docs/overview.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+
+ <li> <a href="../downloads.html" title="Downloads">Downloads</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../docs/index.html" title="v0.3.0">v0.3.0</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../docs/index.html" title="English">English</a>
+</li>
+ <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a>
+</li>
+ <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a>
+</li>
+ <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a>
+</li>
+ </ul>
+ </li>
+
+ <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a>
+</li>
+
+ <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a>
+</li>
+
+ <li class="dropdown-submenu">
+ <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../develop/contribute-code.html" title="Code">Code</a>
+</li>
+ <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+
+ <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+
+ <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+
+ <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+
+ <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" title="NUS Site">NUS Site</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../images/singa-logo.png" alt="Apache SINGA"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../images/singa-title.png" alt="Apache SINGA"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Distributed Training Framework</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../docs/overview.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+
+ <li>
+
+ <a href="../downloads.html" title="Downloads">
+ <span class="none"></span>
+ Downloads</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../docs/index.html" title="v0.3.0">
+ <span class="icon-chevron-right"></span>
+ v0.3.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.2.0/index.html" title="v0.2.0">
+ <span class="none"></span>
+ v0.2.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.1.0/index.html" title="v0.1.0">
+ <span class="none"></span>
+ v0.1.0</a>
+ </li>
+ <li class="nav-header">Development</li>
+
+ <li>
+
+ <a href="../develop/schedule.html" title="Schedule">
+ <span class="none"></span>
+ Schedule</a>
+ </li>
+
+ <li>
+
+ <a href="../develop/how-contribute.html" title="How to Contribute">
+ <span class="icon-chevron-right"></span>
+ How to Contribute</a>
+ </li>
+ <li class="nav-header">Community</li>
+
+ <li>
+
+ <a href="../community/source-repository.html" title="Source Repository">
+ <span class="none"></span>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="../community/mail-lists.html" title="Mailing Lists">
+ <span class="none"></span>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="../community/issue-tracking.html" title="Issue Tracking">
+ <span class="none"></span>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="../community/team-list.html" title="SINGA Team">
+ <span class="none"></span>
+ SINGA Team</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" class="externalLink" title="NUS Site">
+ <span class="none"></span>
+ NUS Site</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <h1>Distributed Training Framework</h1>
+<hr />
+<div class="section">
+<h2><a name="Cluster_Topology_Configuration"></a>Cluster Topology Configuration</h2>
+<p>Here we describe how to configure SINGA’s cluster topology to support different distributed training frameworks. The cluster topology is configured in the <tt>cluster</tt> field in <tt>JobProto</tt>. The <tt>cluster</tt> is of type <tt>ClusterProto</tt>:</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">message ClusterProto {
+ optional int32 nworker_groups = 1;
+ optional int32 nserver_groups = 2;
+ optional int32 nworkers_per_group = 3 [default = 1];
+ optional int32 nservers_per_group = 4 [default = 1];
+ optional int32 nworkers_per_procs = 5 [default = 1];
+ optional int32 nservers_per_procs = 6 [default = 1];
+
+ // servers and workers in different processes?
+ optional bool server_worker_separate = 20 [default = false];
+
+ ......
+}
+</pre></div></div>
+<p>The mostly used fields are as follows:</p>
+
+<ul>
+
+<li><tt>nworkers_per_group</tt> and <tt>nworkers_per_procs</tt>: decide the partitioning of worker side ParamShard.</li>
+
+<li><tt>nservers_per_group</tt> and <tt>nservers_per_procs</tt>: decide the partitioning of server side ParamShard.</li>
+
+<li><tt>server_worker_separate</tt>: separate servers and workers in different processes.</li>
+</ul></div>
+<div class="section">
+<h2><a name="Different_Training_Frameworks"></a>Different Training Frameworks</h2>
+<p>In SINGA, worker groups run asynchronously and workers within one group run synchronously. Users can leverage this general design to run both <b>synchronous</b> and <b>asynchronous</b> training frameworks. Here we illustrate how to configure popular distributed training frameworks in SINGA.</p>
+<p><img src="../images/frameworks.png" style="width: 800px" alt="" />
+<p><b> Fig.1 - Training frameworks in SINGA</b></p>
+<div class="section">
+<h3><a name="Sandblaster"></a>Sandblaster</h3>
+<p>This is a <b>synchronous</b> framework used by Google Brain. Fig.2(a) shows the Sandblaster framework implemented in SINGA. Its configuration is as follows:</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cluster {
+ nworker_groups: 1
+ nserver_groups: 1
+ nworkers_per_group: 3
+ nservers_per_group: 2
+ server_worker_separate: true
+}
+</pre></div></div>
+<p>A single server group is launched to handle all requests from workers. A worker computes on its partition of the model, and only communicates with servers handling related parameters.</p></div>
+<div class="section">
+<h3><a name="AllReduce"></a>AllReduce</h3>
+<p>This is a <b>synchronous</b> framework used by Baidu’s DeepImage. Fig.2(b) shows the AllReduce framework implemented in SINGA. Its configuration is as follows:</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cluster {
+ nworker_groups: 1
+ nserver_groups: 1
+ nworkers_per_group: 3
+ nservers_per_group: 3
+ server_worker_separate: false
+}
+</pre></div></div>
+<p>We bind each worker with a server on the same node, so that each node is responsible for maintaining a partition of parameters and collecting updates from all other nodes.</p></div>
+<div class="section">
+<h3><a name="Downpour"></a>Downpour</h3>
+<p>This is a <b>asynchronous</b> framework used by Google Brain. Fig.2(c) shows the Downpour framework implemented in SINGA. Its configuration is as follows:</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cluster {
+ nworker_groups: 2
+ nserver_groups: 1
+ nworkers_per_group: 2
+ nservers_per_group: 2
+ server_worker_separate: true
+}
+</pre></div></div>
+<p>Similar to the synchronous Sandblaster, all workers send requests to a global server group. We divide workers into several worker groups, each running independently and working on parameters from the last <i>update</i> response.</p></div>
+<div class="section">
+<h3><a name="Distributed_Hogwild"></a>Distributed Hogwild</h3>
+<p>This is a <b>asynchronous</b> framework used by Caffe. Fig.2(d) shows the Distributed Hogwild framework implemented in SINGA. Its configuration is as follows:</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cluster {
+ nworker_groups: 3
+ nserver_groups: 3
+ nworkers_per_group: 1
+ nservers_per_group: 1
+ server_worker_separate: false
+}
+</pre></div></div>
+<p>Each node contains a complete server group and a complete worker group. Parameter updates are done locally, so that communication cost during each training step is minimized. However, the server group must periodically synchronize with neighboring groups to improve the training convergence.</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+
+<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: websites/staging/singa/trunk/content/v0.3.0/general-rnn.html
==============================================================================
--- websites/staging/singa/trunk/content/v0.3.0/general-rnn.html (added)
+++ websites/staging/singa/trunk/content/v0.3.0/general-rnn.html Wed Apr 20 05:12:03 2016
@@ -0,0 +1,473 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-04-20
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160420" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – RNN in SINGA</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+
+<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
+
+
+<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
+
+
+<meta content="Apache SINGA" name="author"/>
+
+
+<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/>
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../docs/overview.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+
+ <li> <a href="../downloads.html" title="Downloads">Downloads</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../docs/index.html" title="v0.3.0">v0.3.0</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../docs/index.html" title="English">English</a>
+</li>
+ <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a>
+</li>
+ <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a>
+</li>
+ <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a>
+</li>
+ </ul>
+ </li>
+
+ <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a>
+</li>
+
+ <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a>
+</li>
+
+ <li class="dropdown-submenu">
+ <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../develop/contribute-code.html" title="Code">Code</a>
+</li>
+ <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+
+ <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+
+ <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+
+ <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+
+ <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" title="NUS Site">NUS Site</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../images/singa-logo.png" alt="Apache SINGA"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../images/singa-title.png" alt="Apache SINGA"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">RNN in SINGA</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../docs/overview.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+
+ <li>
+
+ <a href="../downloads.html" title="Downloads">
+ <span class="none"></span>
+ Downloads</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../docs/index.html" title="v0.3.0">
+ <span class="icon-chevron-right"></span>
+ v0.3.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.2.0/index.html" title="v0.2.0">
+ <span class="none"></span>
+ v0.2.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.1.0/index.html" title="v0.1.0">
+ <span class="none"></span>
+ v0.1.0</a>
+ </li>
+ <li class="nav-header">Development</li>
+
+ <li>
+
+ <a href="../develop/schedule.html" title="Schedule">
+ <span class="none"></span>
+ Schedule</a>
+ </li>
+
+ <li>
+
+ <a href="../develop/how-contribute.html" title="How to Contribute">
+ <span class="icon-chevron-right"></span>
+ How to Contribute</a>
+ </li>
+ <li class="nav-header">Community</li>
+
+ <li>
+
+ <a href="../community/source-repository.html" title="Source Repository">
+ <span class="none"></span>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="../community/mail-lists.html" title="Mailing Lists">
+ <span class="none"></span>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="../community/issue-tracking.html" title="Issue Tracking">
+ <span class="none"></span>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="../community/team-list.html" title="SINGA Team">
+ <span class="none"></span>
+ SINGA Team</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" class="externalLink" title="NUS Site">
+ <span class="none"></span>
+ NUS Site</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <h1>RNN in SINGA</h1>
+<hr />
+<p>Recurrent neural networks (RNN) are widely used for modelling sequential data, e.g., natural language sentences. In this page, we describe how to implement a RNN application (or model) using SINGA built-in RNN layers. We will use the <a class="externalLink" href="https://github.com/karpathy/char-rnn">char-rnn modle</a> as an example, which trains over setences or source code, with each character as an input unit. Particularly, we will train a RNN using GRU over <a class="externalLink" href="http://cs.stanford.edu/people/karpathy/char-rnn/">Linux kernel source code</a>. After training, we expect to generate meaningful code from the model, like the one shown by <a class="externalLink" href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">Karpathy</a>. There is a <a href="rnn.html">vanilla RNN example</a> for language modelling using user defined RNN layers, which is different to using built-in RNN layers discribed in this page.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">/*
+ * If this error is set, we will need anything right after that BSD.
+ */
+static void action_new_function(struct s_stat_info *wb)
+{
+ unsigned long flags;
+ int lel_idx_bit = e->edd, *sys & ~((unsigned long) *FIRST_COMPAT);
+ buf[0] = 0xFFFFFFFF & (bit << 4);
+ min(inc, slist->bytes);
+ printk(KERN_WARNING "Memory allocated %02x/%02x, "
+ "original MLL instead\n"),
+ min(min(multi_run - s->len, max) * num_data_in),
+ frame_pos, sz + first_seg);
+ div_u64_w(val, inb_p);
+ spin_unlock(&disk->queue_lock);
+ mutex_unlock(&s->sock->mutex);
+ mutex_unlock(&func->mutex);
+ return disassemble(info->pending_bh);
+}
+</pre></div></div>
+<div class="section">
+<h2><a name="User_configuration"></a>User configuration</h2>
+<p>The major diffences to the configuration of other models, e.g., feed-forward models include,</p>
+
+<ol style="list-style-type: decimal">
+
+<li>the training algorithm should be changed to BPTT (back-propagation through time).</li>
+
+<li>the layers and their connections should be configured differently.</li>
+</ol>
+<p>The train one batch algorithm can be simply configured as</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">train_one_batch {
+ alg: kBPTT
+}
+</pre></div></div>
+<p>Next, we introduce the configuration of the neural net.</p>
+<p><img src="../images/char-rnn.png" style="width: 550px" alt="" />
+<p><b> Fig.1 - Illustration of the structure of the Char-RNN model</b></p>
+<p>Fig.1 illustrates the net structure of the char-rnn model. The input layer buffers all training data (the Linux kernel code is about 6MB). For each iteration, it reads <tt>unroll_len +1</tt> (<tt>unroll_len</tt> is configured by users) successive characters, e.g., “int a;”, and passes the first <tt>unroll_len</tt> characters to <tt>OneHotLayer</tt>s (one per layer). Every <tt>OneHotLayer</tt> converts its character into the one-hot vector representation. The input layer passes the last <tt>unroll_len</tt> characters as labels to the <tt>RNNLabelLayer</tt> (the label of the i-th character is the i+1 character, i.e., the objective is to predict the next character). Each <tt>GRULayer</tt> receives an one-hot vector and the hidden feature vector from its precedent layer. After some feature transformation, its own feature vector is passed to an inner-product layer and its successive <tt>GRULayer</tt>. The i-th SoftmaxLossLayer measures the cross-entropy loss for predicti
ng the i-th character. According to Karpathy, there could be another stack of <tt>GRULayer</tt>s connecting the first stack of <tt>GRULayer</tt>s, which improves the performance if there is enough training data. The layer configuration is similar to that for other models, e.g., feed-forward models. The major difference is on the connection configuration.</p>
+<div class="section">
+<h3><a name="Unrolling_length"></a>Unrolling length</h3>
+<p>To model the long dependency, recurrent layers need to be unrolled many times, denoted as <tt>unroll_len</tt> (i.e., 50). According to our unified neural net representation, the neural net should have configurations for <tt>unroll_len</tt> recurrent layers. It is tedious to let users configure these layers manually. Hence, SINGA makes it a configuration field for each layer. For example, to unroll the <tt>GRULayer</tt>, users just configure it as,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer {
+ type: kGRU
+ unroll_len: 50
+}
+</pre></div></div>
+<p>Not only the <tt>GRULayer</tt> is unrolled, other layers like <tt>InnerProductLayer</tt> and <tt>SoftmaxLossLayer</tt>, are also unrolled. To simplify the configuration, SINGA provides a <tt>unroll_len</tt> field in the net configuration, which sets the <tt>unroll_len</tt> of each layer configuration if the <tt>unroll_len</tt> is not configured explicitly for that layer. For instance, SINGA would set the <tt>unroll_len</tt> of the <tt>GRULayer</tt> to 50 implicitly for the following configuration.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">net {
+ unroll_len: 50
+ layer {
+ type: kCharRNNInput
+ unroll_len: 1 // configure it explicitly
+ }
+ layer {
+ type: kGRU
+ // no configuration for unroll_len
+ }
+ }
+</pre></div></div></div>
+<div class="section">
+<h3><a name="ConnectionType"></a>ConnectionType</h3>
+<p><img src="http://karpathy.github.io/assets/rnn/diags.jpeg" style="width: 550px" alt="" />
+<p><b> Fig.1 - Different RNN structures from <a class="externalLink" href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">Karpathy</a></b></p>
+<p>There would be many types of connections between layers in RNN models as shown by Karpathy in Fig.2. For each <tt>srclayer</tt>, there is a connection_type for it. Taking the i-th <tt>srclayer</tt> as an example, if its connection type is,</p>
+
+<ul>
+
+<li>kOneToOne, then each unrolled layer is connected with one unrolled layer from the i-th <tt>srclayer</tt>.</li>
+
+<li>kOneToALL, then each unrolled layer is connected with all unrolled layers from the i-th <tt>srclayer</tt>.</li>
+</ul></div></div>
+<div class="section">
+<h2><a name="Implementation"></a>Implementation</h2>
+<div class="section">
+<h3><a name="Neural_net_configuration_preprocessing"></a>Neural net configuration preprocessing</h3>
+<p>User configured neural net is preprocessed to unroll the recurrent layers, i.e., duplicating the configuration of the <tt>GRULayer</tt>s, renaming the name of each layer with unrolling index, and re-configuring the <tt>srclayer</tt> field. After preprocessing, each layer’s name is changed to <tt><unrolling_index>#<user_configured_name>.</tt> Consequently, the (unrolled) neural net configuration passed to NeuralNet class includes all layers and their connections. The NeuralNet class creates and setup each layer in the same way as for other models. For example, after partitioning, each layer’s name is changed to <tt><layer_name>@<partition_index></tt>. One difference is that it has some special code for sharing Param data and grad Blobs for layers unrolled from the same original layer.</p>
+<p>Users can visualize the neural net structure using the Python script <tt>tool/graph.py</tt> and the files in <i>WORKSPACE/visualization/</i>. For example, after the training program is started,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">python tool/graph.py examples/char-rnn/visualization/train_net.json
+</pre></div></div>
+<p>The generated image file is shown in Fig.3 for <tt>unroll_len=5</tt>,</p>
+<p><img src="../images/char-rnn-net.jpg" style="width: 550px" alt="" />
+<p><b> Fig.3 - Net structure generated by SINGA</b></p></div>
+<div class="section">
+<h3><a name="BPTTWorker"></a>BPTTWorker</h3>
+<p>The BPTT (back-propagation through time) algorithm is typically used to compute gradients of the objective loss w.r.t. parameters for RNN models. It forwards propagates through all unrolled layers (i.e., timepoints) to compute features of each layer, and backwards propagates to compute gradients of parameters. It is the same as the BP algorithm for feed-forward models if the recurrent layers are unrolled infinite times. In practice, due to the constraint of memory, the truncated BPTT is widely used. It unrolls the recurrent layers a fixed (truncated) times (controlled by <tt>unroll_len</tt>). In SINGA, a BPTTWorker is provided to run the truncated BPTT algorithm for each mini-batch (i.e., iteration). The pseudo code is</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">BPTTWorker::Forward(phase, net) {
+ for each layer in net
+ if layer.unroll_index() == 0
+ Get(layer.params()); // fetch params values from servers
+ srclayers = layer.srclayer();
+ if phase & kTest
+ srclayers.push_back(net->GetConextLayer(layer))
+ layer.ComputeFeature(phase, srclayers)
+}
+
+BPTTWorker::Backward(phase, net) {
+ for each layer in reverse(net.layers())
+ layer.ComputeGradient(layer.srclayers())
+ if layer.unroll_index() == 0
+ Update(layer.params()); // send params gradients to servers
+}
+</pre></div></div>
+<p>The testing phase is processed specially. Because the test phase may sample a long sequence of data (e.g., sampling a piece of Linux kernel code), which requires many unrolled layers (e.g., more than 1000 characters/layers). But we cannot unroll the recurrent layers too many times due to memory constraint. The special line add the 0-th unrolled layer as one of its own source layer. Consequently, it dynamically adds a recurrent connection to the recurrent layer (e.g., GRULayer). Then we can sample from the model for infinite times. Taking the char-rnn model as an example, the test job can be configured as</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">test_steps: 10000
+train_one_batch {
+ Alg: kBPTT
+}
+net {
+ // do not set the unroll_len
+ layer {
+ // do not set the unroll_len
+ }
+ …
+}
+</pre></div></div>
+<p>The instructions for <a href="test.html">running test</a> is the same for feed-forward models.</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+
+<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: websites/staging/singa/trunk/content/v0.3.0/gpu.html
==============================================================================
--- websites/staging/singa/trunk/content/v0.3.0/gpu.html (added)
+++ websites/staging/singa/trunk/content/v0.3.0/gpu.html Wed Apr 20 05:12:03 2016
@@ -0,0 +1,404 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-04-20
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160420" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Training on GPU</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+
+<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
+
+
+<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
+
+
+<meta content="Apache SINGA" name="author"/>
+
+
+<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/>
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../docs/overview.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+
+ <li> <a href="../downloads.html" title="Downloads">Downloads</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../docs/index.html" title="v0.3.0">v0.3.0</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../docs/index.html" title="English">English</a>
+</li>
+ <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a>
+</li>
+ <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a>
+</li>
+ <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a>
+</li>
+ </ul>
+ </li>
+
+ <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a>
+</li>
+
+ <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a>
+</li>
+
+ <li class="dropdown-submenu">
+ <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../develop/contribute-code.html" title="Code">Code</a>
+</li>
+ <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+
+ <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+
+ <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+
+ <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+
+ <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" title="NUS Site">NUS Site</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../images/singa-logo.png" alt="Apache SINGA"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../images/singa-title.png" alt="Apache SINGA"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Training on GPU</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../docs/overview.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+
+ <li>
+
+ <a href="../downloads.html" title="Downloads">
+ <span class="none"></span>
+ Downloads</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../docs/index.html" title="v0.3.0">
+ <span class="icon-chevron-right"></span>
+ v0.3.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.2.0/index.html" title="v0.2.0">
+ <span class="none"></span>
+ v0.2.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.1.0/index.html" title="v0.1.0">
+ <span class="none"></span>
+ v0.1.0</a>
+ </li>
+ <li class="nav-header">Development</li>
+
+ <li>
+
+ <a href="../develop/schedule.html" title="Schedule">
+ <span class="none"></span>
+ Schedule</a>
+ </li>
+
+ <li>
+
+ <a href="../develop/how-contribute.html" title="How to Contribute">
+ <span class="icon-chevron-right"></span>
+ How to Contribute</a>
+ </li>
+ <li class="nav-header">Community</li>
+
+ <li>
+
+ <a href="../community/source-repository.html" title="Source Repository">
+ <span class="none"></span>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="../community/mail-lists.html" title="Mailing Lists">
+ <span class="none"></span>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="../community/issue-tracking.html" title="Issue Tracking">
+ <span class="none"></span>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="../community/team-list.html" title="SINGA Team">
+ <span class="none"></span>
+ SINGA Team</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" class="externalLink" title="NUS Site">
+ <span class="none"></span>
+ NUS Site</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <h1>Training on GPU</h1>
+<hr />
+<p>Considering GPU is much faster than CPU for linear algebra operations, it is essential to support the training of deep learning models (which involves a lot of linear algebra operations) on GPU cards. SINGA now supports training on a single node (i.e., process) with multiple GPU cards. Training in a GPU cluster with multiple nodes is under development.</p>
+<div class="section">
+<h2><a name="Instructions"></a>Instructions</h2>
+<div class="section">
+<h3><a name="Compilation"></a>Compilation</h3>
+<p>To enable the training on GPU, you need to compile SINGA with <a class="externalLink" href="http://www.nvidia.com/object/cuda_home_new.html">CUDA</a> from Nvidia,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./configure --enable-cuda --with-cuda=<path to cuda folder>
+</pre></div></div>
+<p>In addition, if you want to use the <a class="externalLink" href="https://developer.nvidia.com/cudnn">CUDNN library</a> for convolutional neural network provided by Nvidia, you need to enable CUDNN,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./configure --enable-cuda --with-cuda=<path to cuda folder> --enable-cudnn --with-cudnn=<path to cudnn folder>
+</pre></div></div>
+<p>SINGA now supports CUDNN V3 and V4.</p></div>
+<div class="section">
+<h3><a name="Configuration"></a>Configuration</h3>
+<p>The job configuration for GPU training is similar to that for training on CPU. There is one more field to configure, <tt>gpu</tt>, which indicate the device ID of the GPU you want to use. The simplest configuration is</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># job.conf
+...
+gpu: 0
+...
+</pre></div></div>
+<div class="section">
+<h4><a name="Single_node_with_multiple_GPUs"></a>Single node with multiple GPUs</h4>
+<p>This configuration will run the worker on GPU 0. If you want to launch multiple workers, each on a separate GPU, you can configure it as</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># job.conf
+...
+gpu: 0
+gpu: 2
+...
+cluster {
+ nworkers_per_group: 2
+ nworkers_per_process: 2
+}
+</pre></div></div>
+<p>Using the above configuration, SINGA would partition each mini-batch evenly onto two workers which run on GPU 0 and GPU 2 respectively. For more information on running multiple workers in a single node, please refer to <a href="frameworks.html">Training Framework</a>. Please be careful to configure the same number of workers and number of <tt>gpu</tt>s. Otherwise some workers would run on GPU and the rest would run on CPU. This kind of hybrid training is not well supported for now.</p>
+<p>For some layers, their implementation is transparent to GPU/CPU, like the InnerProductLayer GRULayer, ReLULayer, etc. Hence, you can use the same configuration for these layers to run on GPU or CPU. For other layers, especially the layers involved in ConvNet, SINGA uses different implementations for GPU and CPU. Particularly, the GPU version is implemented using CUDNN library. To train a ConvNet on GPU, you configure the layers as</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer {
+ type: kCudnnConv
+ ...
+}
+layer {
+ type: kCudnnPool
+ ...
+}
+</pre></div></div>
+<p>The <a href="cnn.html">cifar10 example</a> and <a href="alexnet.html">Alexnet example</a> have complete configurations for ConvNet.</p></div>
+<div class="section">
+<h4><a name="GPU_cluster"></a>GPU cluster</h4>
+<p>For distributed training over a (GPU) cluster, you just need to configure SINGA with <tt>--enable-dist</tt>, which would then compile SINGA with zookeeper and ZeroMQ.</p></div></div></div>
+<div class="section">
+<h2><a name="Implementation_details"></a>Implementation details</h2>
+<p>SINGA implements the GPU training by assigning each worker a GPU device at the beginning of training (by the Driver class). Then the work can call GPU functions and run them on the assigned GPU. GPU is typically used for linear algebra computation in layer functions, because GPU is good at such computation. There is a <a href="">Context</a> singleton, which stores the handles and random generators for each device. The layer code should detect its running device and then call the CPU or GPU functions correspondingly.</p>
+<p>To make the layer implementation easier SINGA provides some linear algebra functions (in <i>math_blob.h</i>), which are transparent to the running device for users. Internally, they query the Context singleton to get the device information and call CPU or GPU to do the computation. Consequently, users can implement layers without awareness of the underlying running device.</p>
+<p>If the functionality cannot be implemented using SINGA provided functions in <i>math_blob.h</i>, the layer code needs to handle the CPU and GPU devices explicitly by querying the Context singleton. For layers that cannot run on GPU, e.g., input/output layers and connection layers which have little computation but much IO or network workload, there is no need to consider the GPU device. When these layers are configured in a neural net, they will run on CPU (since they don’t call GPU functions).</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+
+<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: websites/staging/singa/trunk/content/v0.3.0/hdfs.html
==============================================================================
--- websites/staging/singa/trunk/content/v0.3.0/hdfs.html (added)
+++ websites/staging/singa/trunk/content/v0.3.0/hdfs.html Wed Apr 20 05:12:03 2016
@@ -0,0 +1,514 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-04-20
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160420" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Using HDFS with SINGA</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+
+<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
+
+
+<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
+
+
+<meta content="Apache SINGA" name="author"/>
+
+
+<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/>
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../docs/overview.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../docs/quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+
+ <li> <a href="../downloads.html" title="Downloads">Downloads</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../docs/index.html" title="v0.3.0">v0.3.0</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../docs/index.html" title="English">English</a>
+</li>
+ <li> <a href="../docs/zh/index.html" title="ä¸æ">ä¸æ</a>
+</li>
+ <li> <a href="../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a>
+</li>
+ <li> <a href="../docs/kr/index.html" title="íêµì´">íêµì´</a>
+</li>
+ </ul>
+ </li>
+
+ <li> <a href="../v0.2.0/index.html" title="v0.2.0">v0.2.0</a>
+</li>
+
+ <li> <a href="../v0.1.0/index.html" title="v0.1.0">v0.1.0</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../develop/schedule.html" title="Schedule">Schedule</a>
+</li>
+
+ <li class="dropdown-submenu">
+ <a href="../develop/how-contribute.html" title="How to Contribute">How to Contribute</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../develop/contribute-code.html" title="Code">Code</a>
+</li>
+ <li> <a href="../develop/contribute-docs.html" title="Documentation">Documentation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../community/source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+
+ <li> <a href="../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+
+ <li> <a href="../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+
+ <li> <a href="../community/team-list.html" title="SINGA Team">SINGA Team</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+
+ <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" title="NUS Site">NUS Site</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../images/singa-logo.png" alt="Apache SINGA"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../images/singa-title.png" alt="Apache SINGA"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Using HDFS with SINGA</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../docs/overview.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../docs/quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+
+ <li>
+
+ <a href="../downloads.html" title="Downloads">
+ <span class="none"></span>
+ Downloads</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../docs/index.html" title="v0.3.0">
+ <span class="icon-chevron-right"></span>
+ v0.3.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.2.0/index.html" title="v0.2.0">
+ <span class="none"></span>
+ v0.2.0</a>
+ </li>
+
+ <li>
+
+ <a href="../v0.1.0/index.html" title="v0.1.0">
+ <span class="none"></span>
+ v0.1.0</a>
+ </li>
+ <li class="nav-header">Development</li>
+
+ <li>
+
+ <a href="../develop/schedule.html" title="Schedule">
+ <span class="none"></span>
+ Schedule</a>
+ </li>
+
+ <li>
+
+ <a href="../develop/how-contribute.html" title="How to Contribute">
+ <span class="icon-chevron-right"></span>
+ How to Contribute</a>
+ </li>
+ <li class="nav-header">Community</li>
+
+ <li>
+
+ <a href="../community/source-repository.html" title="Source Repository">
+ <span class="none"></span>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="../community/mail-lists.html" title="Mailing Lists">
+ <span class="none"></span>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="../community/issue-tracking.html" title="Issue Tracking">
+ <span class="none"></span>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="../community/team-list.html" title="SINGA Team">
+ <span class="none"></span>
+ SINGA Team</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" class="externalLink" title="NUS Site">
+ <span class="none"></span>
+ NUS Site</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <h1>Using HDFS with SINGA</h1>
+<p>This guide explains how to make use of HDFS as the data store for SINGA jobs. </p>
+
+<ol style="list-style-type: decimal">
+
+<li><a href="#quickstart">Quick start using Docker</a></li>
+
+<li><a href="#hdfs">Setup HDFS</a></li>
+
+<li><a href="#examples">Examples</a></li>
+</ol>
+<p>– <a name="quickstart"></a></p>
+<div class="section">
+<h2><a name="Quick_start_using_Docker"></a>Quick start using Docker</h2>
+<p>We provide a Docker container built on top of <tt>singa/mesos</tt> (see the <a class="externalLink" href="http://singa.incubator.apache.org/docs/docker.html">guide on building SINGA on Docker</a>). </p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">git clone https://github.com/ug93tad/incubator-singa
+cd incubator-singa
+git checkout SINGA-97-docker
+cd tool/docker/hdfs
+sudo docker build -t singa/hdfs .
+</pre></div></div>
+<p>Once built, the container image <tt>singa/hdfs</tt> contains the installation of HDFS C++ library (<tt>libhdfs3</tt>) and the latest SINGA code. Many distributed nodes can be launched, and HDFS be set up, by following the <a class="externalLink" href="http://singa.incubator.apache.org/docs/mesos.html">guide for running distributed SINGA on Mesos</a>. </p>
+<p>In the following, we assume the HDFS setup with <tt>node0</tt> being the namenode, and <tt>nodei (i>0)</tt> being the datanodes. </p>
+<p><a name="hdfs"></a></p></div>
+<div class="section">
+<h2><a name="Setup_HDFS"></a>Setup HDFS</h2>
+<p>There are at least 2 C/C++ client libraries for interacting with HDFS. One is from Hadoop (<tt>libhdfs</tt>), which is a <a class="externalLink" href="https://wiki.apache.org/hadoop/LibHDFS">JNI-based library</a>, meaning that communication will go through JVM. The other is <tt>libhdfs3</tt> which is a <a class="externalLink" href="https://github.com/PivotalRD/libhdfs3">native C++ library developed by Pivotal</a>, in which the client communicate directly with HDFS via RPC. The current implementation uses the second one. </p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install <tt>libhdfs3</tt>: follow the <a class="externalLink" href="https://github.com/PivotalRD/libhdfs3#installation">official guide</a>.</p></li>
+
+<li>
+<p><b>Additional setup</b>: recent versions of Hadoop (>2.4.x) support short-circuit local reads which bypass network communications (TCP sockets) when retrieving data at the local nodes. <tt>libhdfs3</tt> will throws errors (but will still continue to work) when it finds that short-circuit read is not set. To deal with this complaints, and improve performance, add the following configuration to <tt>hdfs-site.xml</tt> <b>and to <tt>hdfs-client.xml</tt></b></p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"> <property>
+<name>dfs.client.read.shortcircuit</name>
+<value>true</value>
+ </property>
+ <property>
+<name>dfs.domain.socket.path</name>
+<value>/var/lib/hadoop-hdfs/dn_socket</value>
+ </property>
+</pre></div></div>
+<p>Next, at each client, set <tt>LIBHDFS3_CONF</tt> variable to point to <tt>hdfs-client.xml</tt> file:</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"> export LIBHDFS3_CONF=$HADOOP_HOME/etc/hadoop/hdfs-client.xml
+</pre></div></div></li>
+</ol>
+<p><a name="examples"></a></p></div>
+<div class="section">
+<h2><a name="Examples"></a>Examples</h2>
+<p>We explain how to run CIFAR10 and MNIST examples. Before training, the data must be uploaded to HDFS. </p>
+<div class="section">
+<h3><a name="CIFAR10"></a>CIFAR10</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Upload the data to HDFS (done at any of the HDFS nodes)</p>
+
+<ul>
+
+<li>Change <tt>job.conf</tt> to use HDFS: in <tt>examples/cifar10/job.conf</tt>, set <tt>backend</tt> property to <tt>hdfsfile</tt></li>
+
+<li>Create and upload data:</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cd examples/cifar10
+cp Makefile.example Makefile
+make create
+hadoop dfs -mkdir /examples/cifar10
+hadoop dfs -copyFromLocal cifar-10-batches-bin /examples/cifar10/
+</pre></div></div>
+<p>If successful, the files should be seen in HDFS via <tt>hadoop dfs -ls /examples/cifar10</tt></p></li>
+
+<li>
+<p>Training:</p>
+
+<ul>
+
+<li>Make sure <tt>conf/singa.conf</tt> has correct path to Zookeeper service:</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">zookeeper_host: "node0:2181"
+</pre></div></div>
+
+<ul>
+
+<li>Make sure <tt>job.conf</tt> has correct paths to the train and test datasets:</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">// train layer
+path: "hdfs://node0:9000/examples/cifar10/train_data.bin"
+mean_file: "hdfs://node0:9000/examples/cifar10/image_mean.bin"
+// test layer
+path: "hdfs://node0:9000/examples/cifar10/test_data.bin"
+mean_file: "hdfs://node0:9000/examples/cifar10/image_mean.bin"
+</pre></div></div>
+
+<ul>
+
+<li>Start training: execute the following command at every node</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./singa -conf examples/cifar10/job.conf -singa_conf singa.conf -singa_job 0
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h3><a name="MNIST"></a>MNIST</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Upload the data to HDFS (done at any of the HDFS nodes)</p>
+
+<ul>
+
+<li>Change <tt>job.conf</tt> to use HDFS: in <tt>examples/mnist/job.conf</tt>, set <tt>backend</tt> property to <tt>hdfsfile</tt></li>
+
+<li>Create and upload data:</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cd examples/mnist
+cp Makefile.example Makefile
+make create
+make compile
+./create_data.bin train-images-idx3-ubyte train-labels-idx1-ubyte hdfs://node0:9000/examples/mnist/train_data.bin
+./create_data.bin t10k-images-idx3-ubyte t10k-labels-idx1-ubyte hdfs://node0:9000/examples/mnist/test_data.bin
+</pre></div></div>
+<p>If successful, the files should be seen in HDFS via <tt>hadoop dfs -ls /examples/mnist</tt></p></li>
+
+<li>
+<p>Training:</p>
+
+<ul>
+
+<li>Make sure <tt>conf/singa.conf</tt> has correct path to Zookeeper service:</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">zookeeper_host: "node0:2181"
+</pre></div></div>
+
+<ul>
+
+<li>Make sure <tt>job.conf</tt> has correct paths to the train and test datasets:</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">// train layer
+path: "hdfs://node0:9000/examples/mnist/train_data.bin"
+// test layer
+path: "hdfs://node0:9000/examples/mnist/test_data.bin"
+</pre></div></div>
+
+<ul>
+
+<li>Start training: execute the following command at every node</li>
+ </ul>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./singa -conf examples/mnist/job.conf -singa_conf singa.conf -singa_job 0
+</pre></div></div></li>
+</ol></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+
+<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>