You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by bu...@apache.org on 2016/04/12 08:24:54 UTC
svn commit: r985457 [25/35] - in /websites/staging/singa/trunk/content: ./
community/ develop/ docs/ docs/jp/ docs/kr/ docs/zh/ releases/ v0.1.0/
v0.2.0/ v0.2.0/jp/ v0.2.0/kr/ v0.2.0/zh/
Added: websites/staging/singa/trunk/content/v0.2.0/kr/quick-start.html
==============================================================================
--- websites/staging/singa/trunk/content/v0.2.0/kr/quick-start.html (added)
+++ websites/staging/singa/trunk/content/v0.2.0/kr/quick-start.html Tue Apr 12 06:24:50 2016
@@ -0,0 +1,486 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-04-12
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160412" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – íµ ì¤íí¸</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+
+<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
+
+
+<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
+
+
+<meta content="Apache SINGA" name="author"/>
+
+
+<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/>
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../docs/overview.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../../docs/quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+
+ <li> <a href="../../downloads.html" title="Downloads">Downloads</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../../docs/index.html" title="v0.3.0">v0.3.0</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../../docs/index.html" title="English">English</a>
+</li>
+ <li> <a href="../../docs/zh/index.html" title="ä¸æ">ä¸æ</a>
+</li>
+ <li> <a href="../../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a>
+</li>
+ <li> <a href="../../docs/kr/index.html" title="íêµì´">íêµì´</a>
+</li>
+ </ul>
+ </li>
+
+ <li> <a href="../../v0.2.0/index.html" title="v0.2.0">v0.2.0</a>
+</li>
+
+ <li> <a href="../../v0.1.0/index.html" title="v0.1.0">v0.1.0</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../develop/schedule.html" title="Schedule">Schedule</a>
+</li>
+
+ <li class="dropdown-submenu">
+ <a href="../../develop/how-contribute.html" title="How to Contribute">How to Contribute</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../../develop/contribute-code.html" title="Code">Code</a>
+</li>
+ <li> <a href="../../develop/contribute-docs.html" title="Documentation">Documentation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../community/source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+
+ <li> <a href="../../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+
+ <li> <a href="../../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+
+ <li> <a href="../../community/team-list.html" title="SINGA Team">SINGA Team</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+
+ <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" title="NUS Site">NUS Site</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../../images/singa-logo.png" alt="Apache SINGA"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../../images/singa-title.png" alt="Apache SINGA"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">íµ ì¤íí¸</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../../docs/overview.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../../docs/quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+
+ <li>
+
+ <a href="../../downloads.html" title="Downloads">
+ <span class="none"></span>
+ Downloads</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../../docs/index.html" title="v0.3.0">
+ <span class="icon-chevron-right"></span>
+ v0.3.0</a>
+ </li>
+
+ <li>
+
+ <a href="../../v0.2.0/index.html" title="v0.2.0">
+ <span class="none"></span>
+ v0.2.0</a>
+ </li>
+
+ <li>
+
+ <a href="../../v0.1.0/index.html" title="v0.1.0">
+ <span class="none"></span>
+ v0.1.0</a>
+ </li>
+ <li class="nav-header">Development</li>
+
+ <li>
+
+ <a href="../../develop/schedule.html" title="Schedule">
+ <span class="none"></span>
+ Schedule</a>
+ </li>
+
+ <li>
+
+ <a href="../../develop/how-contribute.html" title="How to Contribute">
+ <span class="icon-chevron-right"></span>
+ How to Contribute</a>
+ </li>
+ <li class="nav-header">Community</li>
+
+ <li>
+
+ <a href="../../community/source-repository.html" title="Source Repository">
+ <span class="none"></span>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/mail-lists.html" title="Mailing Lists">
+ <span class="none"></span>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/issue-tracking.html" title="Issue Tracking">
+ <span class="none"></span>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/team-list.html" title="SINGA Team">
+ <span class="none"></span>
+ SINGA Team</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" class="externalLink" title="NUS Site">
+ <span class="none"></span>
+ NUS Site</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <h1>퀵 스타트</h1>
+<hr />
+<div class="section">
+<h2><a name="SINGA_"></a>SINGA 인스톨</h2>
+<p>SINGA 인스톨은 <a href="installation.html">여기</a>를 참조하십시오.</p>
+<div class="section">
+<h3><a name="Zookeeper_"></a>Zookeeper 실행</h3>
+<p>SINGA 트레이닝은 <a class="externalLink" href="https://zookeeper.apache.org/">zookeeper</a>를 이용합니다. 우선 zookeeper 서비스가 시작되어 있는지 확인하십시오.</p>
+<p>준비된 thirdparty 스크립트를 사용하여 zookeeper를 설치 한 경우 다음 스크립트를 실행하십시오.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">#goto top level folder
+cd SINGA_ROOT
+./bin/zk-service.sh start
+</pre></div></div>
+<p>(<tt>./bin/zk-service.sh stop</tt> // zookeeper 중지).</p>
+<p>기본 포트를 사용하지 않고 zookeeper를 시작시킬 때는 <tt>conf/singa.conf</tt>을 편집하십시오.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">zookeeper_host : "localhost : YOUR_PORT"
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Stand-alone__"></a>Stand-alone 모드에서 실행</h2>
+<p>Stand-alone 모드에서 SINGA을 실행할 때, <a class="externalLink" href="http://mesos.apache.org/">Mesos</a> 와 <a class="externalLink" href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html">YARN</a> 과 같은 클러스터 관리툴을 이용하지 않는 경우를 말합니다.</p>
+<div class="section">
+<h3><a name="Single__"></a>Single 노드에서의 트레이닝</h3>
+<p>하나의 프로세스가 시작됩니다. 예를 들어, <a class="externalLink" href="http://www.cs.toronto.edu/~kriz/cifar.html">CIFAR-10</a> 데이터 세트를 이용하여 <a class="externalLink" href="http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks">CNN 모델</a>을 트레이닝 시킵니다. 하이퍼 파라미터는 <a class="externalLink" href="https://code.google.com/p/cuda-convnet/">cuda-convnet</a>에 따라 설정되어 있습니다. 자세한 내용은 <a href="cnn.html">CNN 샘플</a> 페이지를 참조하십시오.</p>
+<div class="section">
+<h4><a name="a__"></a>데이터와 작업 설정</h4>
+<p>데이터 세트 다운로드와 Triaing 이나 Test 를 위한 데이터 샤드의 생성은 다음과 같이 실시합니다.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cd examples/cifar10/
+cp Makefile.example Makefile
+make download
+make create
+</pre></div></div>
+<p>Training 과 Test 데이터 세트는 각각 <i>cifar10-train-shard</i> 그리고 <i>cifar10-test-shard</i> 폴더에 만들어집니다. 모든 이미지의 특징 평균을 기술한 <i>image_mean.bin</i> 파일도 함께 생성됩니다.</p>
+<p>CNN 모델 트레이닝에 필요한 소스코드는 모두 SINGA에 포함되어 있습니다. 코드를 추가 할 필요는 없습니다. 작업 설정 파일(<i>job.conf</i>) 을 지정하여 스크립트(<i>../../bin/singa-run.sh</i>)를 실행합니다. SINGA 코드를 변경하거나 추가 할 경우는, 프로그래밍가이드 (programming-guide.html)를 참조하십시오.</p></div>
+<div class="section">
+<h4><a name="a__"></a>병렬화 없이 트레이닝</h4>
+<p>Cluster Topology의 기본값은 하나의 worker와 하나의 server가 있습니다. 데이터와 모델의 병렬 처리는 되지 않습니다.</p>
+<p>트레이닝을 시작하기 위하여 다음 스크립트를 실행합니다.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># goto top level folder
+cd ../../
+./bin/singa-run.sh -conf examples/cifar10/job.conf
+</pre></div></div>
+<p>현재 실행중인 작업의 리스트를 보려면</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./bin/singa-console.sh list
+
+JOB ID | NUM PROCS
+---------- | -----------
+24 | 1
+</pre></div></div>
+<p>작업을 종료하려면</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./bin/singa-console.sh kill JOB_ID
+</pre></div></div>
+<p>로그 및 작업 정보는 <i>/tmp/singa-log</i> 폴더에 저장됩니다. <i>conf/singa.conf</i> 파일의 <tt>log-dir</tt>에서 변경 가능합니다.</p></div>
+<div class="section">
+<h4><a name="a__"></a>비동기 병렬 트레이닝</h4>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># job.conf
+...
+cluster {
+ nworker_groups : 2
+ nworkers_per_procs : 2
+ workspace : "examples/cifar10/"
+}
+</pre></div></div>
+<p>여러 worker 그룹을 실행함으로써 <a href="architecture.html">비동기 트레이닝</a>을 할 수 있습니다. 예를 들어, <i>job.conf</i> 을 위와 같이 변경합니다. 기본적으로 하나의 worker 그룹이 하나의 worker를 갖도록 설정되어 있습니다. 위의 설정은 하나의 프로세스에 2개의 worker가 설정되어 있기 때문에 2개의 worker 그룹이 동일한 프로세스로 실행됩니다. ો
0;과 인메모리 <a href="frameworks.html">Downpour</a> 트레이닝 프레임워크로 실행됩니다.</p>
+<p>사용자는 데이터의 분산을 신경 쓸 필요는 없습니다. 랜덤 오프셋에 따라 각 worker 그룹에 데이터가 보내집니다. 각 worker는 다른 데이터 파티션을 담당합니다.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># job.conf
+...
+neuralnet {
+ layer {
+ ...
+ sharddata_conf {
+ random_skip : 5000
+ }
+ }
+ ...
+}
+</pre></div></div>
+<p>스크립트 실행 :</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf examples/cifar10/job.conf
+</pre></div></div></div>
+<div class="section">
+<h4><a name="a__"></a>동기화 병렬 트레이닝</h4>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># job.conf
+...
+cluster {
+ nworkers_per_group : 2
+ nworkers_per_procs : 2
+ workspace : "examples/cifar10/"
+}
+</pre></div></div>
+<p>하나의 worker 그룹으로 여러 worker를 실행하여 <a href="architecture.html">동기 트레이닝</a>을 수행 할 수 있습니다. 예를 들어, <i>job.conf</i> 파일을 위와 같이 변경합니다. 위의 설정은 하나의 worker 그룹에 2개의 worker가 설정되었습니다. worker 들은 그룹 내에서 동기화합니다. 이것은 인메모리 <a href="frameworks.html">sandblaster</a>로 실행됩니다. 모델은 2개의 worker로 분할됩니다
;. 각 레이어가 2개의 worker로 분산됩니다. 배분 된 레이어는 원본 레이어와 기능은 같지만 특징 인스턴스의 수가 <tt>B / g</tt> 로 됩니다. 여기서 <tt>B</tt>는 미니밧치 인스턴스의 숫자로 <tt>g</tt>는 그룹의 worker 수 입니다. <a href="neural-net.html">다른 스킴</a>을 이용한 레이어 (뉴럴네트워크) 파티션 방법도 있습니다.</p>
+<p>다른 설정들은 모두 “병렬화 없음”의 경우와 동일합니다.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf examples/cifar10/job.conf
+</pre></div></div></div></div>
+<div class="section">
+<h3><a name="a_"></a>클러스터에서의 트레이닝</h3>
+<p>클러스터 설정을 변경하여 위 트레이닝 프레임워크를 확장합니다.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">nworker_per_procs : 1
+</pre></div></div>
+<p>모든 프로세스는 하나의 worker 스레드를 생성합니다. 결과 worker 우리는 다른 프로세스 (노드)에서 생성됩니다. 클러스터의 노드를 특정하려면 <i>SINGA_ROOT/conf/</i> 의 <i>hostfile</i> 의 설​​정이 필요합니다.</p>
+<p>e.g.,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">logbase-a01
+logbase-a02
+</pre></div></div>
+<p>zookeeper location도 설정해야합니다.</p>
+<p>e.g.,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># conf/singa.conf
+zookeeper_host : "logbase-a01"
+</pre></div></div>
+<p>스크립트의 실행은 “Single 노드 트레이닝”과 동일합니다.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">./bin/singa-run.sh -conf examples/cifar10/job.conf
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Mesos_"></a>Mesos에서 실행</h2>
+<p><i>working</i> …</p></div>
+<div class="section">
+<h2><a name="a"></a>다음</h2>
+<p>SINGA 의 코드 변경 및 추가에 대한 자세한 내용은 <a href="programming-guide.html">프로그래밍 가이드</a>를 참조하십시오.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+
+<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: websites/staging/singa/trunk/content/v0.2.0/kr/rbm.html
==============================================================================
--- websites/staging/singa/trunk/content/v0.2.0/kr/rbm.html (added)
+++ websites/staging/singa/trunk/content/v0.2.0/kr/rbm.html Tue Apr 12 06:24:50 2016
@@ -0,0 +1,643 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-04-12
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160412" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – RBM Example</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+
+<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
+
+
+<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
+
+
+<meta content="Apache SINGA" name="author"/>
+
+
+<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/>
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../docs/overview.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../../docs/quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+
+ <li> <a href="../../downloads.html" title="Downloads">Downloads</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../../docs/index.html" title="v0.3.0">v0.3.0</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../../docs/index.html" title="English">English</a>
+</li>
+ <li> <a href="../../docs/zh/index.html" title="ä¸æ">ä¸æ</a>
+</li>
+ <li> <a href="../../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a>
+</li>
+ <li> <a href="../../docs/kr/index.html" title="íêµì´">íêµì´</a>
+</li>
+ </ul>
+ </li>
+
+ <li> <a href="../../v0.2.0/index.html" title="v0.2.0">v0.2.0</a>
+</li>
+
+ <li> <a href="../../v0.1.0/index.html" title="v0.1.0">v0.1.0</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../develop/schedule.html" title="Schedule">Schedule</a>
+</li>
+
+ <li class="dropdown-submenu">
+ <a href="../../develop/how-contribute.html" title="How to Contribute">How to Contribute</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../../develop/contribute-code.html" title="Code">Code</a>
+</li>
+ <li> <a href="../../develop/contribute-docs.html" title="Documentation">Documentation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../community/source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+
+ <li> <a href="../../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+
+ <li> <a href="../../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+
+ <li> <a href="../../community/team-list.html" title="SINGA Team">SINGA Team</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+
+ <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" title="NUS Site">NUS Site</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../../images/singa-logo.png" alt="Apache SINGA"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../../images/singa-title.png" alt="Apache SINGA"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">RBM Example</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../../docs/overview.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../../docs/quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+
+ <li>
+
+ <a href="../../downloads.html" title="Downloads">
+ <span class="none"></span>
+ Downloads</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../../docs/index.html" title="v0.3.0">
+ <span class="icon-chevron-right"></span>
+ v0.3.0</a>
+ </li>
+
+ <li>
+
+ <a href="../../v0.2.0/index.html" title="v0.2.0">
+ <span class="none"></span>
+ v0.2.0</a>
+ </li>
+
+ <li>
+
+ <a href="../../v0.1.0/index.html" title="v0.1.0">
+ <span class="none"></span>
+ v0.1.0</a>
+ </li>
+ <li class="nav-header">Development</li>
+
+ <li>
+
+ <a href="../../develop/schedule.html" title="Schedule">
+ <span class="none"></span>
+ Schedule</a>
+ </li>
+
+ <li>
+
+ <a href="../../develop/how-contribute.html" title="How to Contribute">
+ <span class="icon-chevron-right"></span>
+ How to Contribute</a>
+ </li>
+ <li class="nav-header">Community</li>
+
+ <li>
+
+ <a href="../../community/source-repository.html" title="Source Repository">
+ <span class="none"></span>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/mail-lists.html" title="Mailing Lists">
+ <span class="none"></span>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/issue-tracking.html" title="Issue Tracking">
+ <span class="none"></span>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/team-list.html" title="SINGA Team">
+ <span class="none"></span>
+ SINGA Team</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" class="externalLink" title="NUS Site">
+ <span class="none"></span>
+ NUS Site</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <h1>RBM Example</h1>
+<hr />
+<p>This example uses SINGA to train 4 RBM models and one auto-encoder model over the <a class="externalLink" href="http://yann.lecun.com/exdb/mnist/">MNIST dataset</a>. The auto-encoder model is trained to reduce the dimensionality of the MNIST image feature. The RBM models are trained to initialize parameters of the auto-encoder model. This example application is from <a class="externalLink" href="http://www.cs.toronto.edu/~hinton/science.pdf">Hinton’s science paper</a>.</p>
+<div class="section">
+<h2><a name="Running_instructions"></a>Running instructions</h2>
+<p>Running scripts are provided in <i>SINGA_ROOT/examples/rbm</i> folder.</p>
+<p>The MNIST dataset has 70,000 handwritten digit images. The <a href="data.html">data preparation</a> page has details on converting this dataset into SINGA recognizable format. Users can simply run the following commands to download and convert the dataset.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># at SINGA_ROOT/examples/mnist/
+$ cp Makefile.example Makefile
+$ make download
+$ make create
+</pre></div></div>
+<p>The training is separated into two phases, namely pre-training and fine-tuning. The pre-training phase trains 4 RBMs in sequence,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># at SINGA_ROOT/
+$ ./bin/singa-run.sh -conf examples/rbm/rbm1.conf
+$ ./bin/singa-run.sh -conf examples/rbm/rbm2.conf
+$ ./bin/singa-run.sh -conf examples/rbm/rbm3.conf
+$ ./bin/singa-run.sh -conf examples/rbm/rbm4.conf
+</pre></div></div>
+<p>The fine-tuning phase trains the auto-encoder by,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">$ ./bin/singa-run.sh -conf examples/rbm/autoencoder.conf
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Training_details"></a>Training details</h2>
+<div class="section">
+<h3><a name="RBM1"></a>RBM1</h3>
+<p><img src="../images/example-rbm1.png" align="center" width="200px" alt="" /> <span><b>Figure 1 - RBM1.</b></span></p>
+<p>The neural net structure for training RBM1 is shown in Figure 1. The data layer and parser layer provides features for training RBM1. The visible layer (connected with parser layer) of RBM1 accepts the image feature (784 dimension). The hidden layer is set to have 1000 neurons (units). These two layers are configured as,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer{
+ name: "RBMVis"
+ type: kRBMVis
+ srclayers:"mnist"
+ srclayers:"RBMHid"
+ rbm_conf{
+ hdim: 1000
+ }
+ param{
+ name: "w1"
+ init{
+ type: kGaussian
+ mean: 0.0
+ std: 0.1
+ }
+ }
+ param{
+ name: "b11"
+ init{
+ type: kConstant
+ value: 0.0
+ }
+ }
+}
+
+layer{
+ name: "RBMHid"
+ type: kRBMHid
+ srclayers:"RBMVis"
+ rbm_conf{
+ hdim: 1000
+ }
+ param{
+ name: "w1_"
+ share_from: "w1"
+ }
+ param{
+ name: "b12"
+ init{
+ type: kConstant
+ value: 0.0
+ }
+ }
+}
+</pre></div></div>
+<p>For RBM, the weight matrix is shared by the visible and hidden layers. For instance, <tt>w1</tt> is shared by <tt>vis</tt> and <tt>hid</tt> layers shown in Figure 1. In SINGA, we can configure the <tt>share_from</tt> field to enable <a href="param.html">parameter sharing</a> as shown above for the param <tt>w1</tt> and <tt>w1_</tt>.</p>
+<p><a href="train-one-batch.html#contrastive-divergence">Contrastive Divergence</a> is configured as the algorithm for <a href="train-one-batch.html">TrainOneBatch</a>. Following Hinton’s paper, we configure the <a href="updater.html">updating protocol</a> as follows,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># Updater Configuration
+updater{
+ type: kSGD
+ momentum: 0.2
+ weight_decay: 0.0002
+ learning_rate{
+ base_lr: 0.1
+ type: kFixed
+ }
+}
+</pre></div></div>
+<p>Since the parameters of RBM0 will be used to initialize the auto-encoder, we should configure the <tt>workspace</tt> field to specify a path for the checkpoint folder. For example, if we configure it as,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">cluster {
+ workspace: "examples/rbm/rbm1/"
+}
+</pre></div></div>
+<p>Then SINGA will <a href="checkpoint.html">checkpoint the parameters</a> into <i>examples/rbm/rbm1/</i>.</p></div>
+<div class="section">
+<h3><a name="RBM1"></a>RBM1</h3>
+<p><img src="../images/example-rbm2.png" align="center" width="200px" alt="" /> <span><b>Figure 2 - RBM2.</b></span></p>
+<p>Figure 2 shows the net structure of training RBM2. The visible units of RBM2 accept the output from the Sigmoid1 layer. The Inner1 layer is a <tt>InnerProductLayer</tt> whose parameters are set to the <tt>w1</tt> and <tt>b12</tt> learned from RBM1. The neural net configuration is (with layers for data layer and parser layer omitted).</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer{
+ name: "Inner1"
+ type: kInnerProduct
+ srclayers:"mnist"
+ innerproduct_conf{
+ num_output: 1000
+ }
+ param{ name: "w1" }
+ param{ name: "b12"}
+}
+
+layer{
+ name: "Sigmoid1"
+ type: kSigmoid
+ srclayers:"Inner1"
+}
+
+layer{
+ name: "RBMVis"
+ type: kRBMVis
+ srclayers:"Sigmoid1"
+ srclayers:"RBMHid"
+ rbm_conf{
+ hdim: 500
+ }
+ param{
+ name: "w2"
+ ...
+ }
+ param{
+ name: "b21"
+ ...
+ }
+}
+
+layer{
+ name: "RBMHid"
+ type: kRBMHid
+ srclayers:"RBMVis"
+ rbm_conf{
+ hdim: 500
+ }
+ param{
+ name: "w2_"
+ share_from: "w2"
+ }
+ param{
+ name: "b22"
+ ...
+ }
+}
+</pre></div></div>
+<p>To load w0 and b02 from RBM0’s checkpoint file, we configure the <tt>checkpoint_path</tt> as,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">checkpoint_path: "examples/rbm/rbm1/checkpoint/step6000-worker0"
+cluster{
+ workspace: "examples/rbm/rbm2"
+}
+</pre></div></div>
+<p>The workspace is changed for checkpointing <tt>w2</tt>, <tt>b21</tt> and <tt>b22</tt> into <i>examples/rbm/rbm2/</i>.</p></div>
+<div class="section">
+<h3><a name="RBM3"></a>RBM3</h3>
+<p><img src="../images/example-rbm3.png" align="center" width="200px" alt="" /> <span><b>Figure 3 - RBM3.</b></span></p>
+<p>Figure 3 shows the net structure of training RBM3. In this model, a layer with 250 units is added as the hidden layer of RBM3. The visible units of RBM3 accepts output from Sigmoid2 layer. Parameters of Inner1 and Innner2 are set to <tt>w1,b12,w2,b22</tt> which can be load from the checkpoint file of RBM2, i.e., “examples/rbm/rbm2/”.</p></div>
+<div class="section">
+<h3><a name="RBM4"></a>RBM4</h3>
+<p><img src="../images/example-rbm4.png" align="center" width="200px" alt="" /> <span><b>Figure 4 - RBM4.</b></span></p>
+<p>Figure 4 shows the net structure of training RBM4. It is similar to Figure 3, but according to <a class="externalLink" href="http://www.cs.toronto.edu/~hinton/science.pdf">Hinton’s science paper</a>, the hidden units of the top RBM (RBM4) have stochastic real-valued states drawn from a unit variance Gaussian whose mean is determined by the input from the RBM’s logistic visible units. So we add a <tt>gaussian</tt> field in the RBMHid layer to control the sampling distribution (Gaussian or Bernoulli). In addition, this RBM has a much smaller learning rate (0.001). The neural net configuration for the RBM4 and the updating protocol is (with layers for data layer and parser layer omitted),</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># Updater Configuration
+updater{
+ type: kSGD
+ momentum: 0.9
+ weight_decay: 0.0002
+ learning_rate{
+ base_lr: 0.001
+ type: kFixed
+ }
+}
+
+layer{
+ name: "RBMVis"
+ type: kRBMVis
+ srclayers:"Sigmoid3"
+ srclayers:"RBMHid"
+ rbm_conf{
+ hdim: 30
+ }
+ param{
+ name: "w4"
+ ...
+ }
+ param{
+ name: "b41"
+ ...
+ }
+}
+
+layer{
+ name: "RBMHid"
+ type: kRBMHid
+ srclayers:"RBMVis"
+ rbm_conf{
+ hdim: 30
+ gaussian: true
+ }
+ param{
+ name: "w4_"
+ share_from: "w4"
+ }
+ param{
+ name: "b42"
+ ...
+ }
+}
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Auto-encoder"></a>Auto-encoder</h3>
+<p>In the fine-tuning stage, the 4 RBMs are “unfolded” to form encoder and decoder networks that are initialized using the parameters from the previous 4 RBMs.</p>
+<p><img src="../images/example-autoencoder.png" align="center" width="500px" alt="" /> <span><b>Figure 5 - Auto-Encoders.</b></span></p>
+<p>Figure 5 shows the neural net structure for training the auto-encoder. <a href="train-one-batch.html">Back propagation (kBP)</a> is configured as the algorithm for <tt>TrainOneBatch</tt>. We use the same cluster configuration as RBM models. For updater, we use <a href="updater.html#adagradupdater">AdaGrad</a> algorithm with fixed learning rate.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">### Updater Configuration
+updater{
+ type: kAdaGrad
+ learning_rate{
+ base_lr: 0.01
+ type: kFixed
+ }
+}
+</pre></div></div>
+<p>According to <a class="externalLink" href="http://www.cs.toronto.edu/~hinton/science.pdf">Hinton’s science paper</a>, we configure a EuclideanLoss layer to compute the reconstruction error. The neural net configuration is (with some of the middle layers omitted),</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">layer{ name: "data" }
+layer{ name:"mnist" }
+layer{
+ name: "Inner1"
+ param{ name: "w1" }
+ param{ name: "b12" }
+}
+layer{ name: "Sigmoid1" }
+...
+layer{
+ name: "Inner8"
+ innerproduct_conf{
+ num_output: 784
+ transpose: true
+ }
+ param{
+ name: "w8"
+ share_from: "w1"
+ }
+ param{ name: "b11" }
+}
+layer{ name: "Sigmoid8" }
+
+# Euclidean Loss Layer Configuration
+layer{
+ name: "loss"
+ type:kEuclideanLoss
+ srclayers:"Sigmoid8"
+ srclayers:"mnist"
+}
+</pre></div></div>
+<p>To load pre-trained parameters from the 4 RBMs’ checkpoint file we configure <tt>checkpoint_path</tt> as</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">### Checkpoint Configuration
+checkpoint_path: "examples/rbm/checkpoint/rbm1/checkpoint/step6000-worker0"
+checkpoint_path: "examples/rbm/checkpoint/rbm2/checkpoint/step6000-worker0"
+checkpoint_path: "examples/rbm/checkpoint/rbm3/checkpoint/step6000-worker0"
+checkpoint_path: "examples/rbm/checkpoint/rbm4/checkpoint/step6000-worker0"
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Visualization_Results"></a>Visualization Results</h2>
+
+<div>
+<img src="../images/rbm-weight.PNG" align="center" width="300px" alt="" />
+
+<img src="../images/rbm-feature.PNG" align="center" width="300px" alt="" />
+<br />
+<span><b>Figure 6 - Bottom RBM weight matrix.</b></span>
+ 
+ 
+ 
+ 
+
+<span><b>Figure 7 - Top layer features.</b></span>
+</div>
+<p>Figure 6 visualizes sample columns of the weight matrix of RBM1, We can see the Gabor-like filters are learned. Figure 7 depicts the features extracted from the top-layer of the auto-encoder, wherein one point represents one image. Different colors represent different digits. We can see that most images are well clustered according to the ground truth.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+
+<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: websites/staging/singa/trunk/content/v0.2.0/kr/rnn.html
==============================================================================
--- websites/staging/singa/trunk/content/v0.2.0/kr/rnn.html (added)
+++ websites/staging/singa/trunk/content/v0.2.0/kr/rnn.html Tue Apr 12 06:24:50 2016
@@ -0,0 +1,674 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-04-12
+ | Rendered using Apache Maven Fluido Skin 1.4
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160412" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache SINGA – Recurrent Neural Networks for Language Modelling</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.4.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.4.min.js"></script>
+
+
+
+<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
+
+
+<script type="text/x-mathjax-config">MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});</script>
+
+
+<meta content="Apache SINGA" name="author"/>
+
+
+<meta content="SINGA, Deep Learning, Distributed training" name="keywords"/>
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="https://github.com/apache/incubator-singa">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top navbar-inverse">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache SINGA <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../docs/overview.html" title="Introduction">Introduction</a>
+</li>
+
+ <li> <a href="../../docs/quick-start.html" title="Quick Start">Quick Start</a>
+</li>
+
+ <li> <a href="../../downloads.html" title="Downloads">Downloads</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentaion <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li class="dropdown-submenu">
+ <a href="../../docs/index.html" title="v0.3.0">v0.3.0</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../../docs/index.html" title="English">English</a>
+</li>
+ <li> <a href="../../docs/zh/index.html" title="ä¸æ">ä¸æ</a>
+</li>
+ <li> <a href="../../docs/jp/index.html" title="æ¥æ¬èª">æ¥æ¬èª</a>
+</li>
+ <li> <a href="../../docs/kr/index.html" title="íêµì´">íêµì´</a>
+</li>
+ </ul>
+ </li>
+
+ <li> <a href="../../v0.2.0/index.html" title="v0.2.0">v0.2.0</a>
+</li>
+
+ <li> <a href="../../v0.1.0/index.html" title="v0.1.0">v0.1.0</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Development <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../develop/schedule.html" title="Schedule">Schedule</a>
+</li>
+
+ <li class="dropdown-submenu">
+ <a href="../../develop/how-contribute.html" title="How to Contribute">How to Contribute</a>
+ <ul class="dropdown-menu">
+ <li> <a href="../../develop/contribute-code.html" title="Code">Code</a>
+</li>
+ <li> <a href="../../develop/contribute-docs.html" title="Documentation">Documentation</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="../../community/source-repository.html" title="Source Repository">Source Repository</a>
+</li>
+
+ <li> <a href="../../community/mail-lists.html" title="Mailing Lists">Mailing Lists</a>
+</li>
+
+ <li> <a href="../../community/issue-tracking.html" title="Issue Tracking">Issue Tracking</a>
+</li>
+
+ <li> <a href="../../community/team-list.html" title="SINGA Team">SINGA Team</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://www.apache.org/" title="Apache Software Foundation">Apache Software Foundation</a>
+</li>
+
+ <li> <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" title="NUS Site">NUS Site</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../../index.html" id="bannerLeft" title="Apache SINGA">
+ <img src="../../images/singa-logo.png" alt="Apache SINGA"/>
+ </a>
+ </div>
+ <div class="pull-right"> <div id="bannerRight">
+ <img src="../../images/singa-title.png" alt="Apache SINGA"/>
+ </div>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="../../index.html" title="Apache SINGA">
+ Apache SINGA</a>
+ <span class="divider">/</span>
+ </li>
+ <li class="active ">Recurrent Neural Networks for Language Modelling</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span2">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Apache SINGA</li>
+
+ <li>
+
+ <a href="../../docs/overview.html" title="Introduction">
+ <span class="none"></span>
+ Introduction</a>
+ </li>
+
+ <li>
+
+ <a href="../../docs/quick-start.html" title="Quick Start">
+ <span class="none"></span>
+ Quick Start</a>
+ </li>
+
+ <li>
+
+ <a href="../../downloads.html" title="Downloads">
+ <span class="none"></span>
+ Downloads</a>
+ </li>
+ <li class="nav-header">Documentaion</li>
+
+ <li>
+
+ <a href="../../docs/index.html" title="v0.3.0">
+ <span class="icon-chevron-right"></span>
+ v0.3.0</a>
+ </li>
+
+ <li>
+
+ <a href="../../v0.2.0/index.html" title="v0.2.0">
+ <span class="none"></span>
+ v0.2.0</a>
+ </li>
+
+ <li>
+
+ <a href="../../v0.1.0/index.html" title="v0.1.0">
+ <span class="none"></span>
+ v0.1.0</a>
+ </li>
+ <li class="nav-header">Development</li>
+
+ <li>
+
+ <a href="../../develop/schedule.html" title="Schedule">
+ <span class="none"></span>
+ Schedule</a>
+ </li>
+
+ <li>
+
+ <a href="../../develop/how-contribute.html" title="How to Contribute">
+ <span class="icon-chevron-right"></span>
+ How to Contribute</a>
+ </li>
+ <li class="nav-header">Community</li>
+
+ <li>
+
+ <a href="../../community/source-repository.html" title="Source Repository">
+ <span class="none"></span>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/mail-lists.html" title="Mailing Lists">
+ <span class="none"></span>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/issue-tracking.html" title="Issue Tracking">
+ <span class="none"></span>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="../../community/team-list.html" title="SINGA Team">
+ <span class="none"></span>
+ SINGA Team</a>
+ </li>
+ <li class="nav-header">External Links</li>
+
+ <li>
+
+ <a href="http://www.apache.org/" class="externalLink" title="Apache Software Foundation">
+ <span class="none"></span>
+ Apache Software Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.comp.nus.edu.sg/~dbsystem/singa/" class="externalLink" title="NUS Site">
+ <span class="none"></span>
+ NUS Site</a>
+ </li>
+ </ul>
+
+
+
+ <hr />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://incubator.apache.org" title="apache-incubator" class="builtBy">
+ <img class="builtBy" alt="Apache Incubator" src="http://incubator.apache.org/images/egg-logo.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span10" >
+
+ <h1>Recurrent Neural Networks for Language Modelling</h1>
+<hr />
+<p>Recurrent Neural Networks (RNN) are widely used for modelling sequential data, such as music and sentences. In this example, we use SINGA to train a <a class="externalLink" href="http://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf">RNN model</a> proposed by Tomas Mikolov for <a class="externalLink" href="https://en.wikipedia.org/wiki/Language_model">language modeling</a>. The training objective (loss) is to minimize the <a class="externalLink" href="https://en.wikipedia.org/wiki/Perplexity">perplexity per word</a>, which is equivalent to maximize the probability of predicting the next word given the current word in a sentence.</p>
+<p>Different to the <a href="cnn.html">CNN</a>, <a href="mlp.html">MLP</a> and <a href="rbm.html">RBM</a> examples which use built-in layers(layer) and records(data), none of the layers in this example are built-in. Hence users would learn to implement their own layers and data records through this example.</p>
+<div class="section">
+<h2><a name="Running_instructions"></a>Running instructions</h2>
+<p>In <i>SINGA_ROOT/examples/rnnlm/</i>, scripts are provided to run the training job. First, the data is prepared by</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">$ cp Makefile.example Makefile
+$ make download
+$ make create
+</pre></div></div>
+<p>Second, to compile the source code under <i>examples/rnnlm/</i>, run</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">$ make rnnlm
+</pre></div></div>
+<p>An executable file <i>rnnlm.bin</i> will be generated.</p>
+<p>Third, the training is started by passing <i>rnnlm.bin</i> and the job configuration to <i>singa-run.sh</i>,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># at SINGA_ROOT/
+# export LD_LIBRARY_PATH=.libs:$LD_LIBRARY_PATH
+$ ./bin/singa-run.sh -exec examples/rnnlm/rnnlm.bin -conf examples/rnnlm/job.conf
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Implementations"></a>Implementations</h2>
+<p><img src="../images/rnnlm.png" align="center" width="400px" alt="" /> <span><b>Figure 1 - Net structure of the RNN model.</b></span></p>
+<p>The neural net structure is shown Figure 1. Word records are loaded by <tt>DataLayer</tt>. For every iteration, at most <tt>max_window</tt> word records are processed. If a sentence ending character is read, the <tt>DataLayer</tt> stops loading immediately. <tt>EmbeddingLayer</tt> looks up a word embedding matrix to extract feature vectors for words loaded by the <tt>DataLayer</tt>. These features are transformed by the <tt>HiddenLayer</tt> which propagates the features from left to right. The output feature for word at position k is influenced by words from position 0 to k-1. Finally, <tt>LossLayer</tt> computes the cross-entropy loss (see below) by predicting the next word of each word. The cross-entropy loss is computed as</p>
+<p><tt>$$L(w_t)=-log P(w_{t+1}|w_t)$$</tt></p>
+<p>Given <tt>$w_t$</tt> the above equation would compute over all words in the vocabulary, which is time consuming. <a class="externalLink" href="https://f25ea9ccb7d3346ce6891573d543960492b92c30.googledrive.com/host/0ByxdPXuxLPS5RFM5dVNvWVhTd0U/rnnlm-0.4b.tgz">RNNLM Toolkit</a> accelerates the computation as</p>
+<p><tt>$$P(w_{t+1}|w_t) = P(C_{w_{t+1}}|w_t) * P(w_{t+1}|C_{w_{t+1}})$$</tt></p>
+<p>Words from the vocabulary are partitioned into a user-defined number of classes. The first term on the left side predicts the class of the next word, and then predicts the next word given its class. Both the number of classes and the words from one class are much smaller than the vocabulary size. The probabilities can be calculated much faster.</p>
+<p>The perplexity per word is computed by,</p>
+<p><tt>$$PPL = 10^{- avg_t log_{10} P(w_{t+1}|w_t)}$$</tt></p>
+<div class="section">
+<h3><a name="Data_preparation"></a>Data preparation</h3>
+<p>We use a small dataset provided by the <a class="externalLink" href="https://f25ea9ccb7d3346ce6891573d543960492b92c30.googledrive.com/host/0ByxdPXuxLPS5RFM5dVNvWVhTd0U/rnnlm-0.4b.tgz">RNNLM Toolkit</a>. It has 10,000 training sentences, with 71350 words in total and 3720 unique words. The subsequent steps follow the instructions in <a href="data.html">Data Preparation</a> to convert the raw data into records and insert them into data stores.</p>
+<div class="section">
+<h4><a name="Download_source_data"></a>Download source data</h4>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># in SINGA_ROOT/examples/rnnlm/
+cp Makefile.example Makefile
+make download
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Define_record_format"></a>Define record format</h4>
+<p>We define the word record as follows,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># in SINGA_ROOT/examples/rnnlm/rnnlm.proto
+message WordRecord {
+ optional string word = 1;
+ optional int32 word_index = 2;
+ optional int32 class_index = 3;
+ optional int32 class_start = 4;
+ optional int32 class_end = 5;
+}
+</pre></div></div>
+<p>It includes the word string and its index in the vocabulary. Words in the vocabulary are sorted based on their frequency in the training dataset. The sorted list is cut into 100 sublists such that each sublist has 1/100 total word frequency. Each sublist is called a class. Hence each word has a <tt>class_index</tt> ([0,100)). The <tt>class_start</tt> is the index of the first word in the same class as <tt>word</tt>. The <tt>class_end</tt> is the index of the first word in the next class.</p></div>
+<div class="section">
+<h4><a name="Create_data_stores"></a>Create data stores</h4>
+<p>We use code from RNNLM Toolkit to read words, and sort them into classes. The main function in <i>create_store.cc</i> first creates word classes based on the training dataset. Second it calls the following function to create data store for the training, validation and test dataset.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">int create_data(const char *input_file, const char *output_file);
+</pre></div></div>
+<p><tt>input</tt> is the path to training/validation/testing text file from the RNNLM Toolkit, <tt>output</tt> is output store file. This function starts with</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">singa::io::KVFile store;
+store.Open(output, signa::io::kCreate);
+</pre></div></div>
+<p>Then it reads the words one by one. For each word it creates a <tt>WordRecord</tt> instance, and inserts it into the store,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">int wcnt = 0; // word count
+WordRecord wordRecord;
+while(1) {
+ readWord(wordstr, fin);
+ if (feof(fin)) break;
+ ...// fill in the wordRecord;
+ string val;
+ wordRecord.SerializeToString(&val);
+ int length = snprintf(key, BUFFER_LEN, "%05d", wcnt++);
+ store.Write(string(key, length), val);
+}
+</pre></div></div>
+<p>Compilation and running commands are provided in the <i>Makefile.example</i>. After executing</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">make create
+</pre></div></div>
+<p><i>train_data.bin</i>, <i>test_data.bin</i> and <i>valid_data.bin</i> will be created.</p></div></div>
+<div class="section">
+<h3><a name="Layer_implementation"></a>Layer implementation</h3>
+<p>4 user-defined layers are implemented for this application. Following the guide for implementing <a href="layer#implementing-a-new-layer-subclass">new Layer subclasses</a>, we extend the <a href="../api/classsinga_1_1LayerProto.html">LayerProto</a> to include the configuration messages of user-defined layers as shown below (3 out of the 7 layers have specific configurations),</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">import "job.proto"; // Layer message for SINGA is defined
+
+//For implementation of RNNLM application
+extend singa.LayerProto {
+ optional EmbeddingProto embedding_conf = 101;
+ optional LossProto loss_conf = 102;
+ optional DataProto data_conf = 103;
+}
+</pre></div></div>
+<p>In the subsequent sections, we describe the implementation of each layer, including its configuration message.</p>
+<div class="section">
+<h4><a name="RNNLayer"></a>RNNLayer</h4>
+<p>This is the base layer of all other layers for this applications. It is defined as follows,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">class RNNLayer : virtual public Layer {
+public:
+ inline int window() { return window_; }
+protected:
+ int window_;
+};
+</pre></div></div>
+<p>For this application, two iterations may process different number of words. Because sentences have different lengths. The <tt>DataLayer</tt> decides the effective window size. All other layers call its source layers to get the effective window size and resets <tt>window_</tt> in <tt>ComputeFeature</tt> function.</p></div>
+<div class="section">
+<h4><a name="DataLayer"></a>DataLayer</h4>
+<p>DataLayer is for loading Records.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">class DataLayer : public RNNLayer, singa::InputLayer {
+ public:
+ void Setup(const LayerProto& proto, const vector<Layer*>& srclayers) override;
+ void ComputeFeature(int flag, const vector<Layer*>& srclayers) override;
+ int max_window() const {
+ return max_window_;
+ }
+ private:
+ int max_window_;
+ singa::io::Store* store_;
+};
+</pre></div></div>
+<p>The Setup function gets the user configured max window size.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">max_window_ = proto.GetExtension(input_conf).max_window();
+</pre></div></div>
+<p>The <tt>ComputeFeature</tt> function loads at most max_window records. It could also stop when the sentence ending character is encountered.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">...// shift the last record to the first
+window_ = max_window_;
+for (int i = 1; i <= max_window_; i++) {
+ // load record; break if it is the ending character
+}
+</pre></div></div>
+<p>The configuration of <tt>DataLayer</tt> is like</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">name: "data"
+user_type: "kData"
+[data_conf] {
+ path: "examples/rnnlm/train_data.bin"
+ max_window: 10
+}
+</pre></div></div></div>
+<div class="section">
+<h4><a name="EmbeddingLayer"></a>EmbeddingLayer</h4>
+<p>This layer gets records from <tt>DataLayer</tt>. For each record, the word index is parsed and used to get the corresponding word feature vector from the embedding matrix.</p>
+<p>The class is declared as follows,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">class EmbeddingLayer : public RNNLayer {
+ ...
+ const std::vector<Param*> GetParams() const override {
+ std::vector<Param*> params{embed_};
+ return params;
+ }
+ private:
+ int word_dim_, vocab_size_;
+ Param* embed_;
+}
+</pre></div></div>
+<p>The <tt>embed_</tt> field is a matrix whose values are parameter to be learned. The matrix size is <tt>vocab_size_</tt> x <tt>word_dim_</tt>.</p>
+<p>The Setup function reads configurations for <tt>word_dim_</tt> and <tt>vocab_size_</tt>. Then it allocates feature Blob for <tt>max_window</tt> words and setups <tt>embed_</tt>.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">int max_window = srclayers[0]->data(this).shape()[0];
+word_dim_ = proto.GetExtension(embedding_conf).word_dim();
+data_.Reshape(vector<int>{max_window, word_dim_});
+...
+embed_->Setup(vector<int>{vocab_size_, word_dim_});
+</pre></div></div>
+<p>The <tt>ComputeFeature</tt> function simply copies the feature vector from the <tt>embed_</tt> matrix into the feature Blob.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># reset effective window size
+window_ = datalayer->window();
+auto records = datalayer->records();
+...
+for (int t = 0; t < window_; t++) {
+ int idx <- word index
+ Copy(words[t], embed[idx]);
+}
+</pre></div></div>
+<p>The <tt>ComputeGradient</tt> function copies back the gradients to the <tt>embed_</tt> matrix.</p>
+<p>The configuration for <tt>EmbeddingLayer</tt> is like,</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">user_type: "kEmbedding"
+[embedding_conf] {
+ word_dim: 15
+ vocab_size: 3720
+}
+srclayers: "data"
+param {
+ name: "w1"
+ init {
+ type: kUniform
+ low:-0.3
+ high:0.3
+ }
+}
+</pre></div></div></div>
+<div class="section">
+<h4><a name="HiddenLayer"></a>HiddenLayer</h4>
+<p>This layer unrolls the recurrent connections for at most max_window times. The feature for position k is computed based on the feature from the embedding layer (position k) and the feature at position k-1 of this layer. The formula is</p>
+<p><tt>$$f[k]=\sigma (f[t-1]*W+src[t])$$</tt></p>
+<p>where <tt>$W$</tt> is a matrix with <tt>word_dim_</tt> x <tt>word_dim_</tt> parameters.</p>
+<p>If you want to implement a recurrent neural network following our design, this layer is of vital importance for you to refer to.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">class HiddenLayer : public RNNLayer {
+ ...
+ const std::vector<Param*> GetParams() const override {
+ std::vector<Param*> params{weight_};
+ return params;
+ }
+private:
+ Param* weight_;
+};
+</pre></div></div>
+<p>The <tt>Setup</tt> function setups the weight matrix as</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">weight_->Setup(std::vector<int>{word_dim, word_dim});
+</pre></div></div>
+<p>The <tt>ComputeFeature</tt> function gets the effective window size (<tt>window_</tt>) from its source layer i.e., the embedding layer. Then it propagates the feature from position 0 to position <tt>window_</tt> -1. The detailed descriptions for this process are illustrated as follows.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">void HiddenLayer::ComputeFeature() {
+ for(int t = 0; t < window_size; t++){
+ if(t == 0)
+ Copy(data[t], src[t]);
+ else
+ data[t]=sigmoid(data[t-1]*W + src[t]);
+ }
+}
+</pre></div></div>
+<p>The <tt>ComputeGradient</tt> function computes the gradient of the loss w.r.t. W and the source layer. Particularly, for each position k, since data[k] contributes to data[k+1] and the feature at position k in its destination layer (the loss layer), grad[k] should contains the gradient from two parts. The destination layer has already computed the gradient from the loss layer into grad[k]; In the <tt>ComputeGradient</tt> function, we need to add the gradient from position k+1.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">void HiddenLayer::ComputeGradient(){
+ ...
+ for (int k = window_ - 1; k >= 0; k--) {
+ if (k < window_ - 1) {
+ grad[k] += dot(grad[k + 1], weight.T()); // add gradient from position t+1.
+ }
+ grad[k] =... // compute gL/gy[t], y[t]=data[t-1]*W+src[t]
+ }
+ gweight = dot(data.Slice(0, window_-1).T(), grad.Slice(1, window_));
+ Copy(gsrc, grad);
+}
+</pre></div></div>
+<p>After the loop, we get the gradient of the loss w.r.t y[k], which is used to compute the gradient of W and the src[k].</p></div>
+<div class="section">
+<h4><a name="LossLayer"></a>LossLayer</h4>
+<p>This layer computes the cross-entropy loss and the <tt>$log_{10}P(w_{t+1}|w_t)$</tt> (which could be averaged over all words by users to get the PPL value).</p>
+<p>There are two configuration fields to be specified by users.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">message LossProto {
+ optional int32 nclass = 1;
+ optional int32 vocab_size = 2;
+}
+</pre></div></div>
+<p>There are two weight matrices to be learned</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">class LossLayer : public RNNLayer {
+ ...
+ private:
+ Param* word_weight_, *class_weight_;
+}
+</pre></div></div>
+<p>The ComputeFeature function computes the two probabilities respectively.</p>
+<p><tt>$$P(C_{w_{t+1}}|w_t) = Softmax(w_t * class\_weight_)$$</tt> <tt>$$P(w_{t+1}|C_{w_{t+1}}) = Softmax(w_t * word\_weight[class\_start:class\_end])$$</tt></p>
+<p><tt>$w_t$</tt> is the feature from the hidden layer for the k-th word, its ground truth next word is <tt>$w_{t+1}$</tt>. The first equation computes the probability distribution over all classes for the next word. The second equation computes the probability distribution over the words in the ground truth class for the next word.</p>
+<p>The ComputeGradient function computes the gradient of the source layer (i.e., the hidden layer) and the two weight matrices.</p></div></div>
+<div class="section">
+<h3><a name="Updater_Configuration"></a>Updater Configuration</h3>
+<p>We employ kFixedStep type of the learning rate change method and the configuration is as follows. We decay the learning rate once the performance does not increase on the validation dataset.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint">updater{
+ type: kSGD
+ learning_rate {
+ type: kFixedStep
+ fixedstep_conf:{
+ step:0
+ step:48810
+ step:56945
+ step:65080
+ step:73215
+ step_lr:0.1
+ step_lr:0.05
+ step_lr:0.025
+ step_lr:0.0125
+ step_lr:0.00625
+ }
+ }
+}
+</pre></div></div></div>
+<div class="section">
+<h3><a name="TrainOneBatch_Function"></a>TrainOneBatch() Function</h3>
+<p>We use BP (BackPropagation) algorithm to train the RNN model here. The corresponding configuration can be seen below.</p>
+
+<div class="source">
+<div class="source"><pre class="prettyprint"># In job.conf file
+train_one_batch {
+ alg: kBackPropagation
+}
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Cluster_Configuration"></a>Cluster Configuration</h3>
+<p>The default cluster configuration can be used, i.e., single worker and single server in a single process.</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row-fluid">
+
+<p>Copyright © 2015 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+
+
+ </div>
+ </footer>
+ </body>
+</html>