You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by wa...@apache.org on 2019/06/29 14:42:26 UTC
svn commit: r1862313 [5/34] - in /incubator/singa/site/trunk: ./ _sources/
_sources/community/ _sources/docs/ _sources/docs/model_zoo/
_sources/docs/model_zoo/caffe/ _sources/docs/model_zoo/char-rnn/
_sources/docs/model_zoo/cifar10/ _sources/docs/model...
Modified: incubator/singa/site/trunk/docs/installation.html
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/docs/installation.html?rev=1862313&r1=1862312&r2=1862313&view=diff
==============================================================================
--- incubator/singa/site/trunk/docs/installation.html (original)
+++ incubator/singa/site/trunk/docs/installation.html Sat Jun 29 14:42:24 2019
@@ -116,6 +116,7 @@
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="software_stack.html">Software Stack</a></li>
+<li class="toctree-l2"><a class="reference internal" href="benchmark.html">Benchmark for Distributed training</a></li>
<li class="toctree-l2"><a class="reference internal" href="device.html">Device</a></li>
<li class="toctree-l2"><a class="reference internal" href="tensor.html">Tensor</a></li>
<li class="toctree-l2"><a class="reference internal" href="layer.html">Layer</a></li>
@@ -242,12 +243,12 @@
After installing miniconda, execute the one of the following commands to install
SINGA.</p>
<ol>
-<li><p class="first">CPU only</p>
+<li><p>CPU only</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">conda</span> <span class="n">install</span> <span class="o">-</span><span class="n">c</span> <span class="n">nusdbsystem</span> <span class="n">singa</span><span class="o">-</span><span class="n">cpu</span>
</pre></div>
</div>
</li>
-<li><p class="first">GPU with CUDA and cuDNN</p>
+<li><p>GPU with CUDA and cuDNN</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">conda</span> <span class="n">install</span> <span class="o">-</span><span class="n">c</span> <span class="n">nusdbsystem</span> <span class="n">singa</span><span class="o">-</span><span class="n">gpu</span>
</pre></div>
</div>
@@ -306,29 +307,30 @@ the instructions of installing the packa
Refer to SINGA <a class="reference external" href="https://github.com/apache/incubator-singa/blob/master/tool/docker/">Dockerfiles</a>
for the instructions of installing them on Ubuntu 16.04.</p>
<ul class="simple">
-<li>cmake (>=2.8)</li>
-<li>gcc (>=4.8.1)</li>
-<li>google protobuf (>=2.5)</li>
-<li>blas (tested with openblas >=0.2.10)</li>
-<li>swig(>=3.0.10) for compiling PySINGA</li>
-<li>numpy(>=1.11.0) for compiling PySINGA</li>
+<li><p>cmake (>=2.8)</p></li>
+<li><p>gcc (>=4.8.1)</p></li>
+<li><p>google protobuf (>=2.5)</p></li>
+<li><p>blas (tested with openblas >=0.2.10)</p></li>
+<li><p>swig(>=3.0.10) for compiling PySINGA</p></li>
+<li><p>numpy(>=1.11.0) for compiling PySINGA</p></li>
</ul>
<ol class="simple">
-<li>create a <code class="docutils literal notranslate"><span class="pre">build</span></code> folder inside incubator-singa and go into that folder</li>
-<li>run <code class="docutils literal notranslate"><span class="pre">cmake</span> <span class="pre">[options]</span> <span class="pre">..</span></code>
-by default all options are OFF except <code class="docutils literal notranslate"><span class="pre">USE_PYTHON</span></code><ul>
-<li><code class="docutils literal notranslate"><span class="pre">USE_MODULES=ON</span></code>, used if protobuf and blas are not installed a prior</li>
-<li><code class="docutils literal notranslate"><span class="pre">USE_CUDA=ON</span></code>, used if CUDA and cuDNN is available</li>
-<li><code class="docutils literal notranslate"><span class="pre">USE_PYTHON3=ON</span></code>, used for compiling with Python 3 support. (The default is Python 2)</li>
-<li><code class="docutils literal notranslate"><span class="pre">USE_OPENCL=ON</span></code>, used for compiling with OpenCL support</li>
-<li><code class="docutils literal notranslate"><span class="pre">USE_MKLDNN=ON</span></code>, used for compiling with Intel MKL-dnn support</li>
-<li><code class="docutils literal notranslate"><span class="pre">PACKAGE=ON</span></code>, used for building the Debian package</li>
-<li><code class="docutils literal notranslate"><span class="pre">ENABLE_TEST</span></code>, used for compiling unit test cases</li>
+<li><p>create a <code class="docutils literal notranslate"><span class="pre">build</span></code> folder inside incubator-singa and go into that folder</p></li>
+<li><p>run <code class="docutils literal notranslate"><span class="pre">cmake</span> <span class="pre">[options]</span> <span class="pre">..</span></code>
+by default all options are OFF except <code class="docutils literal notranslate"><span class="pre">USE_PYTHON</span></code></p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">USE_MODULES=ON</span></code>, used if protobuf and blas are not installed a prior</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">USE_CUDA=ON</span></code>, used if CUDA and cuDNN is available</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">USE_PYTHON3=ON</span></code>, used for compiling with Python 3 support. (The default is Python 2)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">USE_OPENCL=ON</span></code>, used for compiling with OpenCL support</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">USE_MKLDNN=ON</span></code>, used for compiling with Intel MKL-dnn support</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">PACKAGE=ON</span></code>, used for building the Debian package</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">ENABLE_TEST</span></code>, used for compiling unit test cases</p></li>
</ul>
</li>
-<li>compile the code, <code class="docutils literal notranslate"><span class="pre">make</span></code></li>
-<li>goto python folder</li>
-<li>run <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">.</span></code> or <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">-e</span> <span class="pre">.</span></code> The second command creates symlinks instead of copying files into python site-package folder.</li>
+<li><p>compile the code, <code class="docutils literal notranslate"><span class="pre">make</span></code></p></li>
+<li><p>goto python folder</p></li>
+<li><p>run <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">.</span></code> or <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">-e</span> <span class="pre">.</span></code> The second command creates symlinks instead of copying files into python site-package folder.</p></li>
</ol>
<p>Execute step 4 and 5 are to install PySINGA when USE_PYTHON=ON.</p>
<p>After compiling SINGA with ENABLE_TEST=ON, you can run the unit tests by</p>
@@ -406,9 +408,9 @@ $ sudo yum install opencl-headers, vienn
</div>
<p>Additionally, you will need the OpenCL Installable Client Driver (ICD) for the platforms that you want to run OpenCL on.</p>
<ul class="simple">
-<li>For AMD and nVidia GPUs, the driver package should also install the correct OpenCL ICD.</li>
-<li>For Intel CPUs and/or GPUs, get the driver from the <a class="reference external" href="https://software.intel.com/en-us/articles/opencl-drivers">Intel website.</a> Note that the drivers provided on that website only supports recent CPUs and Iris GPUs.</li>
-<li>For older Intel CPUs, you can use the <code class="docutils literal notranslate"><span class="pre">beignet-opencl-icd</span></code> package.</li>
+<li><p>For AMD and nVidia GPUs, the driver package should also install the correct OpenCL ICD.</p></li>
+<li><p>For Intel CPUs and/or GPUs, get the driver from the <a class="reference external" href="https://software.intel.com/en-us/articles/opencl-drivers">Intel website.</a> Note that the drivers provided on that website only supports recent CPUs and Iris GPUs.</p></li>
+<li><p>For older Intel CPUs, you can use the <code class="docutils literal notranslate"><span class="pre">beignet-opencl-icd</span></code> package.</p></li>
</ul>
<p>Note that running OpenCL on CPUs is not currently recommended because it is slow.
Memory transfer is on the order of whole seconds (1000âs of ms on CPUs as compared to 1âs of ms on GPUs).</p>
@@ -447,18 +449,18 @@ $ make package
<div class="section" id="faq">
<h2>FAQ<a class="headerlink" href="#faq" title="Permalink to this headline">¶</a></h2>
<ul>
-<li><p class="first">Q: Error from âimport singaâ using PySINGA installed from wheel.</p>
+<li><p>Q: Error from âimport singaâ using PySINGA installed from wheel.</p>
<p>A: Please check the detailed error from <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">-c</span> <span class="pre">"from</span> <span class="pre">singa</span> <span class="pre">import</span> <span class="pre">_singa_wrap"</span></code>. Sometimes it is
caused by the dependent libraries, e.g. there are multiple versions of protobuf, missing of cudnn, numpy version mismatch. Following
steps show the solutions for different cases</p>
<ol>
-<li><p class="first">Check the cudnn and cuda and gcc versions, cudnn5 and cuda7.5 and gcc4.8/4.9 are preferred. if gcc is 5.0, then downgrade it.
+<li><p>Check the cudnn and cuda and gcc versions, cudnn5 and cuda7.5 and gcc4.8/4.9 are preferred. if gcc is 5.0, then downgrade it.
If cudnn is missing or not match with the wheel version, you can download the correct version of cudnn into ~/local/cudnn/ and</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ echo "export LD_LIBRARY_PATH=/home/<yourname>/local/cudnn/lib64:$LD_LIBRARY_PATH" >> ~/.bashrc
</pre></div>
</div>
</li>
-<li><p class="first">If it is the problem related to protobuf, then download the newest whl files which have <a class="reference external" href="https://issues.apache.org/jira/browse/SINGA-255">compiled protobuf and openblas into the whl</a> file of PySINGA.
+<li><p>If it is the problem related to protobuf, then download the newest whl files which have <a class="reference external" href="https://issues.apache.org/jira/browse/SINGA-255">compiled protobuf and openblas into the whl</a> file of PySINGA.
Or you can install protobuf from source into a local folder, say ~/local/;
Decompress the tar file, and then</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ ./configure --prefix=/home/<yourname>local
@@ -468,9 +470,8 @@ Decompress the tar file, and then</p>
</pre></div>
</div>
</li>
-<li><p class="first">If it cannot find other libs including python, then create virtual env using pip or conda;</p>
-</li>
-<li><p class="first">If it is not caused by the above reasons, go to the folder of <code class="docutils literal notranslate"><span class="pre">_singa_wrap.so</span></code>,</p>
+<li><p>If it cannot find other libs including python, then create virtual env using pip or conda;</p></li>
+<li><p>If it is not caused by the above reasons, go to the folder of <code class="docutils literal notranslate"><span class="pre">_singa_wrap.so</span></code>,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ python
>> import importlib
>> importlib.import_module('_singa_wrap')
@@ -486,7 +487,7 @@ Decompress the tar file, and then</p>
</pre></div>
</div>
</li>
-<li><p class="first">Q: Error from running <code class="docutils literal notranslate"><span class="pre">cmake</span> <span class="pre">..</span></code>, which cannot find the dependent libraries.</p>
+<li><p>Q: Error from running <code class="docutils literal notranslate"><span class="pre">cmake</span> <span class="pre">..</span></code>, which cannot find the dependent libraries.</p>
<p>A: If you havenât installed the libraries, install them. If you installed
the libraries in a folder that is outside of the system folder, e.g. /usr/local,
you need to export the following variables</p>
@@ -495,7 +496,7 @@ you need to export the following variabl
</pre></div>
</div>
</li>
-<li><p class="first">Q: Error from <code class="docutils literal notranslate"><span class="pre">make</span></code>, e.g. the linking phase</p>
+<li><p>Q: Error from <code class="docutils literal notranslate"><span class="pre">make</span></code>, e.g. the linking phase</p>
<p>A: If your libraries are in other folders than system default paths, you need
to export the following varaibles</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ export LIBRARY_PATH=<path to your lib file folder>
@@ -503,20 +504,20 @@ to export the following varaibles</p>
</pre></div>
</div>
</li>
-<li><p class="first">Q: Error from header files, e.g. âcblas.h no such file or directory existsâ</p>
+<li><p>Q: Error from header files, e.g. âcblas.h no such file or directory existsâ</p>
<p>A: You need to include the folder of the cblas.h into CPLUS_INCLUDE_PATH,
e.g.,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
</pre></div>
</div>
</li>
-<li><p class="first">Q:While compiling SINGA, I get error <code class="docutils literal notranslate"><span class="pre">SSE2</span> <span class="pre">instruction</span> <span class="pre">set</span> <span class="pre">not</span> <span class="pre">enabled</span></code></p>
+<li><p>Q:While compiling SINGA, I get error <code class="docutils literal notranslate"><span class="pre">SSE2</span> <span class="pre">instruction</span> <span class="pre">set</span> <span class="pre">not</span> <span class="pre">enabled</span></code></p>
<p>A:You can try following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ make CFLAGS='-msse2' CXXFLAGS='-msse2'
</pre></div>
</div>
</li>
-<li><p class="first">Q:I get <code class="docutils literal notranslate"><span class="pre">ImportError:</span> <span class="pre">cannot</span> <span class="pre">import</span> <span class="pre">name</span> <span class="pre">enum_type_wrapper</span></code> from google.protobuf.internal when I try to import .py files.</p>
+<li><p>Q:I get <code class="docutils literal notranslate"><span class="pre">ImportError:</span> <span class="pre">cannot</span> <span class="pre">import</span> <span class="pre">name</span> <span class="pre">enum_type_wrapper</span></code> from google.protobuf.internal when I try to import .py files.</p>
<p>A: You need to install the python binding of protobuf, which could be installed via</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ sudo apt-get install protobuf
</pre></div>
@@ -529,7 +530,7 @@ e.g.,</p>
</pre></div>
</div>
</li>
-<li><p class="first">Q: When I build OpenBLAS from source, I am told that I need a Fortran compiler.</p>
+<li><p>Q: When I build OpenBLAS from source, I am told that I need a Fortran compiler.</p>
<p>A: You can compile OpenBLAS by</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ make ONLY_CBLAS=1
</pre></div>
@@ -539,7 +540,7 @@ e.g.,</p>
</pre></div>
</div>
</li>
-<li><p class="first">Q: When I build protocol buffer, it reports that GLIBC++_3.4.20 not found in /usr/lib64/libstdc++.so.6.</p>
+<li><p>Q: When I build protocol buffer, it reports that GLIBC++_3.4.20 not found in /usr/lib64/libstdc++.so.6.</p>
<p>A: This means the linker found libstdc++.so.6 but that library
belongs to an older version of GCC than was used to compile and link the
program. The program depends on code defined in
@@ -556,7 +557,7 @@ following command,</p>
</pre></div>
</div>
</li>
-<li><p class="first">Q: When I build glog, it reports that âsrc/logging_unittest.cc:83:20: error: âgflagsâ is not a namespace-nameâ</p>
+<li><p>Q: When I build glog, it reports that âsrc/logging_unittest.cc:83:20: error: âgflagsâ is not a namespace-nameâ</p>
<p>A: It maybe that you have installed gflags with a different namespace such as âgoogleâ. so glog canât find âgflagsâ namespace.
Because it is not necessary to have gflags to build glog. So you can change the configure.ac file to ignore gflags.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="mf">1.</span> <span class="n">cd</span> <span class="n">to</span> <span class="n">glog</span> <span class="n">src</span> <span class="n">directory</span>
@@ -566,17 +567,17 @@ Because it is not necessary to have gfla
</div>
<p>After this, you can build glog again.</p>
</li>
-<li><p class="first">Q: When using virtual environment, everytime I run pip install, it would reinstall numpy. However, the numpy would not be used when I <code class="docutils literal notranslate"><span class="pre">import</span> <span class="pre">numpy</span></code></p>
+<li><p>Q: When using virtual environment, everytime I run pip install, it would reinstall numpy. However, the numpy would not be used when I <code class="docutils literal notranslate"><span class="pre">import</span> <span class="pre">numpy</span></code></p>
<p>A: It could be caused by the <code class="docutils literal notranslate"><span class="pre">PYTHONPATH</span></code> which should be set to empty when you are using virtual environment to avoid the conflicts with the path of
the virtual environment.</p>
</li>
-<li><p class="first">Q: When compiling PySINGA from source, there is a compilation error due to the missing of <numpy/objectarray.h></p>
+<li><p>Q: When compiling PySINGA from source, there is a compilation error due to the missing of <numpy/objectarray.h></p>
<p>A: Please install numpy and export the path of numpy header files as</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> $ export CPLUS_INCLUDE_PATH=`python -c "import numpy; print numpy.get_include()"`:$CPLUS_INCLUDE_PATH
</pre></div>
</div>
</li>
-<li><p class="first">Q: When I run PySINGA in Mac OS X, I got the error âFatal Python error: PyThreadState_Get: no current thread Abort trap: 6â</p>
+<li><p>Q: When I run PySINGA in Mac OS X, I got the error âFatal Python error: PyThreadState_Get: no current thread Abort trap: 6â</p>
<p>A: This error happens typically when you have multiple version of Python on your system and you installed SINGA via pip (this problem is resolved for installation via conda),
e.g, the one comes with the OS and the one installed by Homebrew. The Python linked by PySINGA must be the same as the Python interpreter.
You can check your interpreter by <code class="docutils literal notranslate"><span class="pre">which</span> <span class="pre">python</span></code> and check the Python linked by PySINGA via <code class="docutils literal notranslate"><span class="pre">otool</span> <span class="pre">-L</span> <span class="pre"><path</span> <span class="pre">to</span> <span class="pre">_singa_wrap.so></span></code>.
Modified: incubator/singa/site/trunk/docs/layer.html
URL: http://svn.apache.org/viewvc/incubator/singa/site/trunk/docs/layer.html?rev=1862313&r1=1862312&r2=1862313&view=diff
==============================================================================
--- incubator/singa/site/trunk/docs/layer.html (original)
+++ incubator/singa/site/trunk/docs/layer.html Sat Jun 29 14:42:24 2019
@@ -104,10 +104,11 @@
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Documentation</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="software_stack.html">Software Stack</a></li>
+<li class="toctree-l2"><a class="reference internal" href="benchmark.html">Benchmark for Distributed training</a></li>
<li class="toctree-l2"><a class="reference internal" href="device.html">Device</a></li>
<li class="toctree-l2"><a class="reference internal" href="tensor.html">Tensor</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Layer</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="#python-api">Python API</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#module-singa.layer">Python API</a></li>
<li class="toctree-l3"><a class="reference internal" href="#cpp-api">CPP API</a></li>
</ul>
</li>
@@ -209,8 +210,949 @@
<div class="section" id="layer">
<h1>Layer<a class="headerlink" href="#layer" title="Permalink to this headline">¶</a></h1>
-<div class="section" id="python-api">
-<h2>Python API<a class="headerlink" href="#python-api" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="module-singa.layer">
+<span id="python-api"></span><h2>Python API<a class="headerlink" href="#module-singa.layer" title="Permalink to this headline">¶</a></h2>
+<p>Python layers wrap the C++ layers to provide simpler construction APIs.</p>
+<p>Example usages:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">singa</span> <span class="k">import</span> <span class="n">layer</span>
+<span class="kn">from</span> <span class="nn">singa</span> <span class="k">import</span> <span class="n">tensor</span>
+<span class="kn">from</span> <span class="nn">singa</span> <span class="k">import</span> <span class="n">device</span>
+
+<span class="n">layer</span><span class="o">.</span><span class="n">engine</span> <span class="o">=</span> <span class="s1">'cudnn'</span> <span class="c1"># to use cudnn layers</span>
+<span class="n">dev</span> <span class="o">=</span> <span class="n">device</span><span class="o">.</span><span class="n">create_cuda_gpu</span><span class="p">()</span>
+
+<span class="c1"># create a convolution layer</span>
+<span class="n">conv</span> <span class="o">=</span> <span class="n">layer</span><span class="o">.</span><span class="n">Conv2D</span><span class="p">(</span><span class="s1">'conv'</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">pad</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">input_sample_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">))</span>
+
+<span class="c1"># init param values</span>
+<span class="n">w</span><span class="p">,</span> <span class="n">b</span> <span class="o">=</span> <span class="n">conv</span><span class="o">.</span><span class="n">param_values</span><span class="p">()</span>
+<span class="n">w</span><span class="o">.</span><span class="n">guassian</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.01</span><span class="p">)</span>
+<span class="n">b</span><span class="o">.</span><span class="n">set_value</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
+<span class="n">conv</span><span class="o">.</span><span class="n">to_device</span><span class="p">(</span><span class="n">dev</span><span class="p">)</span> <span class="c1"># move the layer data onto a CudaGPU device</span>
+
+<span class="n">x</span> <span class="o">=</span> <span class="n">tensor</span><span class="o">.</span><span class="n">Tensor</span><span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span> <span class="n">dev</span><span class="p">)</span>
+<span class="n">x</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
+<span class="n">y</span> <span class="o">=</span> <span class="n">conv</span><span class="o">.</span><span class="n">foward</span><span class="p">(</span><span class="kc">True</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
+
+<span class="n">dy</span> <span class="o">=</span> <span class="n">tensor</span><span class="o">.</span><span class="n">Tensor</span><span class="p">()</span>
+<span class="n">dy</span><span class="o">.</span><span class="n">reset_like</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
+<span class="n">dy</span><span class="o">.</span><span class="n">set_value</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
+<span class="c1"># dp is a list of tensors for parameter gradients</span>
+<span class="n">dx</span><span class="p">,</span> <span class="n">dp</span> <span class="o">=</span> <span class="n">conv</span><span class="o">.</span><span class="n">backward</span><span class="p">(</span><span class="n">kTrain</span><span class="p">,</span> <span class="n">dy</span><span class="p">)</span>
+</pre></div>
+</div>
+<dl class="data">
+<dt id="singa.layer.engine">
+<code class="descclassname">singa.layer.</code><code class="descname">engine</code><em class="property"> = 'cudnn'</em><a class="headerlink" href="#singa.layer.engine" title="Permalink to this definition">¶</a></dt>
+<dd><p>engine is the prefix of layer identifier.</p>
+<p>The value could be one of [<strong>âcudnnâ, âsingacppâ, âsingacudaâ, âsingaclâ</strong>], for
+layers implemented using the cudnn library, Cpp, Cuda and OpenCL respectively.
+For example, CudnnConvolution layer is identified by âcudnn_convolutionâ;
+âsingacpp_convolutionâ is for Convolution layer;
+Some layersâ implementation use only Tensor functions, thererfore they are
+transparent to the underlying devices. For threse layers, they would have
+multiple identifiers, e.g., singacpp_dropout, singacuda_dropout and
+singacl_dropout are all for the Dropout layer. In addition, it has an extra
+identifier âsingaâ, i.e. âsinga_dropoutâ also stands for the Dropout layer.</p>
+<p>engine is case insensitive. Each python layer would create the correct specific
+layer using the engine attribute.</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Layer">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Layer</code><span class="sig-paren">(</span><em>name</em>, <em>conf=None</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></p>
+<p>Base Python layer class.</p>
+<dl class="simple">
+<dt>Typically, the life cycle of a layer instance includes:</dt><dd><ol class="arabic simple">
+<li><p>construct layer without input_sample_shapes, goto 2;
+construct layer with input_sample_shapes, goto 3;</p></li>
+<li><p>call setup to create the parameters and setup other meta fields</p></li>
+<li><p>call forward or access layer members</p></li>
+<li><p>call backward and get parameters for update</p></li>
+</ol>
+</dd>
+</dl>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>name</strong> (<em>str</em>) â layer name</p>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.Layer.setup">
+<code class="descname">setup</code><span class="sig-paren">(</span><em>in_shapes</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.setup" title="Permalink to this definition">¶</a></dt>
+<dd><p>Call the C++ setup function to create params and set some meta data.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>in_shapes</strong> â if the layer accepts a single input Tensor, in_shapes is
+a single tuple specifying the inpute Tensor shape; if the layer
+accepts multiple input Tensor (e.g., the concatenation layer),
+in_shapes is a tuple of tuples, each for one input Tensor</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.caffe_layer">
+<code class="descname">caffe_layer</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.caffe_layer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Create a singa layer based on caffe layer configuration.</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.param_names">
+<code class="descname">param_names</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.param_names" title="Permalink to this definition">¶</a></dt>
+<dd><dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a list of strings, one for the name of one parameter Tensor</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.param_values">
+<code class="descname">param_values</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.param_values" title="Permalink to this definition">¶</a></dt>
+<dd><p>Return param value tensors.</p>
+<p>Parameter tensors are not stored as layer members because cpp Tensor
+could be moved onto diff devices due to the change of layer device,
+which would result in inconsistency.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a list of tensors, one for each paramter</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>flag</em>, <em>x</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Forward propagate through this layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â True (kTrain) for training (kEval); False for evaluating;
+other values for furture use.</p></li>
+<li><p><strong>x</strong> (<a class="reference internal" href="tensor.html#singa.tensor.Tensor" title="singa.tensor.Tensor"><em>Tensor</em></a><em> or </em><em>list<Tensor></em>) â an input tensor if the layer is
+connected from a single layer; a list of tensors if the layer
+is connected from multiple layers.</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p>a tensor if the layer is connected to a single layer; a list of
+tensors if the layer is connected to multiple layers;</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>flag</em>, <em>dy</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Backward propagate gradients through this layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> (<em>int</em>) â for future use.</p></li>
+<li><p><strong>dy</strong> (<a class="reference internal" href="tensor.html#singa.tensor.Tensor" title="singa.tensor.Tensor"><em>Tensor</em></a><em> or </em><em>list<Tensor></em>) â the gradient tensor(s) y w.r.t the
+objective loss</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p><dx, <dp1, dp2..>>, dx is a (set of) tensor(s) for the gradient of x
+, dpi is the gradient of the i-th parameter</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.to_device">
+<code class="descname">to_device</code><span class="sig-paren">(</span><em>device</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.to_device" title="Permalink to this definition">¶</a></dt>
+<dd><p>Move layer state tensors onto the given device.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>device</strong> â swig converted device, created using singa.device</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Layer.as_type">
+<code class="descname">as_type</code><span class="sig-paren">(</span><em>dtype</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Layer.as_type" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Dummy">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Dummy</code><span class="sig-paren">(</span><em>name</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Dummy" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>A dummy layer that does nothing but just forwards/backwards the data
+(the input/output is a single tensor).</p>
+<dl class="method">
+<dt id="singa.layer.Dummy.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Dummy.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Dummy.setup">
+<code class="descname">setup</code><span class="sig-paren">(</span><em>input_sample_shape</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Dummy.setup" title="Permalink to this definition">¶</a></dt>
+<dd><p>Call the C++ setup function to create params and set some meta data.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>in_shapes</strong> â if the layer accepts a single input Tensor, in_shapes is
+a single tuple specifying the inpute Tensor shape; if the layer
+accepts multiple input Tensor (e.g., the concatenation layer),
+in_shapes is a tuple of tuples, each for one input Tensor</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Dummy.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>flag</em>, <em>x</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Dummy.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Return the input x</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Dummy.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>falg</em>, <em>dy</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Dummy.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Return dy, []</p>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Conv2D">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Conv2D</code><span class="sig-paren">(</span><em>name</em>, <em>nb_kernels</em>, <em>kernel=3</em>, <em>stride=1</em>, <em>border_mode='same'</em>, <em>cudnn_prefer='fastest'</em>, <em>workspace_byte_limit=1024</em>, <em>data_format='NCHW'</em>, <em>use_bias=True</em>, <em>W_specs=None</em>, <em>b_specs=None</em>, <em>pad=None</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Conv2D" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Construct a layer for 2D convolution.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>nb_kernels</strong> (<em>int</em>) â num of the channels (kernels) of the input Tensor</p></li>
+<li><p><strong>kernel</strong> â an integer or a pair of integers for kernel height and width</p></li>
+<li><p><strong>stride</strong> â an integer or a pair of integers for stride height and width</p></li>
+<li><p><strong>border_mode</strong> (<em>string</em>) â padding mode, case in-sensitive,
+âvalidâ -> padding is 0 for height and width
+âsameâ -> padding is half of the kernel (floor), the kernel must be
+odd number.</p></li>
+<li><p><strong>cudnn_prefer</strong> (<em>string</em>) â the preferred algorithm for cudnn convolution
+which could be âfastestâ, âautotuneâ, âlimited_workspaceâ and
+âno_workspaceâ</p></li>
+<li><p><strong>workspace_byte_limit</strong> (<em>int</em>) â max workspace size in MB (default is 512MB)</p></li>
+<li><p><strong>data_format</strong> (<em>string</em>) â either âNCHWâ or âNHWCâ</p></li>
+<li><p><strong>use_bias</strong> (<em>bool</em>) â True or False</p></li>
+<li><p><strong>pad</strong> â an integer or a pair of integers for padding height and width</p></li>
+<li><p><strong>W_specs</strong> (<em>dict</em>) â used to specify the weight matrix specs, fields
+include,
+ânameâ for parameter name
+âlr_multâ for learning rate multiplier
+âdecay_multâ for weight decay multiplier
+âinitâ for init method, which could be âgaussianâ, âuniformâ,
+âxavierâ and ââ
+âstdâ, âmeanâ, âhighâ, âlowâ for corresponding init methods
+TODO(wangwei) âclampâ for gradient constraint, value is scalar
+âregularizerâ for regularization, currently support âl2â</p></li>
+<li><p><strong>b_specs</strong> (<em>dict</em>) â hyper-parameters for bias vector, similar as W_specs</p></li>
+<li><p><strong>name</strong> (<em>string</em>) â layer name.</p></li>
+<li><p><strong>input_sample_shape</strong> â 3d tuple for the shape of the input Tensor
+without the batchsize, e.g., (channel, height, width) or
+(height, width, channel)</p></li>
+</ul>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.Conv2D.setup">
+<code class="descname">setup</code><span class="sig-paren">(</span><em>in_shape</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Conv2D.setup" title="Permalink to this definition">¶</a></dt>
+<dd><p>Set up the kernel, stride and padding; then call the C++ setup
+function to create params and set some meta data.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>is a tuple of int for the input sample shape</strong> (<em>in_shapes</em>) â </p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Conv1D">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Conv1D</code><span class="sig-paren">(</span><em>name</em>, <em>nb_kernels</em>, <em>kernel=3</em>, <em>stride=1</em>, <em>border_mode='same'</em>, <em>cudnn_prefer='fastest'</em>, <em>workspace_byte_limit=1024</em>, <em>use_bias=True</em>, <em>W_specs={'init': 'Xavier'}</em>, <em>b_specs={'init': 'Constant'</em>, <em>'value': 0}</em>, <em>pad=None</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Conv1D" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Conv2D" title="singa.layer.Conv2D"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Conv2D</span></code></a></p>
+<p>Construct a layer for 1D convolution.</p>
+<p>Most of the args are the same as those for Conv2D except the kernel,
+stride, pad, which is a scalar instead of a tuple.
+input_sample_shape is a tuple with a single value for the input feature
+length</p>
+<dl class="method">
+<dt id="singa.layer.Conv1D.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Conv1D.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Pooling2D">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Pooling2D</code><span class="sig-paren">(</span><em>name</em>, <em>mode</em>, <em>kernel=3</em>, <em>stride=2</em>, <em>border_mode='same'</em>, <em>pad=None</em>, <em>data_format='NCHW'</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Pooling2D" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>2D pooling layer providing max/avg pooling.</p>
+<p>All args are the same as those for Conv2D, except the following one</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>mode</strong> â pooling type, model_pb2.PoolingConf.MAX or
+model_pb2.PoolingConf.AVE</p>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.Pooling2D.setup">
+<code class="descname">setup</code><span class="sig-paren">(</span><em>in_shape</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Pooling2D.setup" title="Permalink to this definition">¶</a></dt>
+<dd><p>Set up the kernel, stride and padding; then call the C++ setup
+function to create params and set some meta data.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>is a tuple of int for the input sample shape</strong> (<em>in_shapes</em>) â </p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.MaxPooling2D">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">MaxPooling2D</code><span class="sig-paren">(</span><em>name</em>, <em>kernel=3</em>, <em>stride=2</em>, <em>border_mode='same'</em>, <em>pad=None</em>, <em>data_format='NCHW'</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.MaxPooling2D" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Pooling2D" title="singa.layer.Pooling2D"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Pooling2D</span></code></a></p>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.AvgPooling2D">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">AvgPooling2D</code><span class="sig-paren">(</span><em>name</em>, <em>kernel=3</em>, <em>stride=2</em>, <em>border_mode='same'</em>, <em>pad=None</em>, <em>data_format='NCHW'</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.AvgPooling2D" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Pooling2D" title="singa.layer.Pooling2D"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Pooling2D</span></code></a></p>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.MaxPooling1D">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">MaxPooling1D</code><span class="sig-paren">(</span><em>name</em>, <em>kernel=3</em>, <em>stride=2</em>, <em>border_mode='same'</em>, <em>pad=None</em>, <em>data_format='NCHW'</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.MaxPooling1D" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.MaxPooling2D" title="singa.layer.MaxPooling2D"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.MaxPooling2D</span></code></a></p>
+<dl class="method">
+<dt id="singa.layer.MaxPooling1D.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.MaxPooling1D.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.AvgPooling1D">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">AvgPooling1D</code><span class="sig-paren">(</span><em>name</em>, <em>kernel=3</em>, <em>stride=2</em>, <em>border_mode='same'</em>, <em>pad=None</em>, <em>data_format='NCHW'</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.AvgPooling1D" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.AvgPooling2D" title="singa.layer.AvgPooling2D"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.AvgPooling2D</span></code></a></p>
+<dl class="method">
+<dt id="singa.layer.AvgPooling1D.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.AvgPooling1D.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.BatchNormalization">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">BatchNormalization</code><span class="sig-paren">(</span><em>name</em>, <em>momentum=0.9</em>, <em>beta_specs=None</em>, <em>gamma_specs=None</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.BatchNormalization" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Batch-normalization.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>momentum</strong> (<em>float</em>) â for running average mean and variance.</p></li>
+<li><p><strong>beta_specs</strong> (<em>dict</em>) â dictionary includes the fields for the beta
+param:
+ânameâ for parameter name
+âlr_multâ for learning rate multiplier
+âdecay_multâ for weight decay multiplier
+âinitâ for init method, which could be âgaussianâ, âuniformâ,
+âxavierâ and ââ
+âstdâ, âmeanâ, âhighâ, âlowâ for corresponding init methods
+âclampâ for gradient constraint, value is scalar
+âregularizerâ for regularization, currently support âl2â</p></li>
+<li><p><strong>gamma_specs</strong> (<em>dict</em>) â similar to beta_specs, but for the gamma param.</p></li>
+<li><p><strong>name</strong> (<em>string</em>) â layer name</p></li>
+<li><p><strong>input_sample_shape</strong> (<em>tuple</em>) â with at least one integer</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.L2Norm">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">L2Norm</code><span class="sig-paren">(</span><em>name</em>, <em>input_sample_shape</em>, <em>epsilon=1e-08</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.L2Norm" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Normalize each sample to have L2 norm = 1</p>
+<dl class="method">
+<dt id="singa.layer.L2Norm.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.L2Norm.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.L2Norm.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>is_train</em>, <em>x</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.L2Norm.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Forward propagate through this layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â True (kTrain) for training (kEval); False for evaluating;
+other values for furture use.</p></li>
+<li><p><strong>x</strong> (<a class="reference internal" href="tensor.html#singa.tensor.Tensor" title="singa.tensor.Tensor"><em>Tensor</em></a><em> or </em><em>list<Tensor></em>) â an input tensor if the layer is
+connected from a single layer; a list of tensors if the layer
+is connected from multiple layers.</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p>a tensor if the layer is connected to a single layer; a list of
+tensors if the layer is connected to multiple layers;</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.L2Norm.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>is_train</em>, <em>dy</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.L2Norm.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Backward propagate gradients through this layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> (<em>int</em>) â for future use.</p></li>
+<li><p><strong>dy</strong> (<a class="reference internal" href="tensor.html#singa.tensor.Tensor" title="singa.tensor.Tensor"><em>Tensor</em></a><em> or </em><em>list<Tensor></em>) â the gradient tensor(s) y w.r.t the
+objective loss</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p><dx, <dp1, dp2..>>, dx is a (set of) tensor(s) for the gradient of x
+, dpi is the gradient of the i-th parameter</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.LRN">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">LRN</code><span class="sig-paren">(</span><em>name</em>, <em>size=5</em>, <em>alpha=1</em>, <em>beta=0.75</em>, <em>mode='cross_channel'</em>, <em>k=1</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.LRN" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Local response normalization.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>size</strong> (<em>int</em>) â # of channels to be crossed
+normalization.</p></li>
+<li><p><strong>mode</strong> (<em>string</em>) â âcross_channelâ</p></li>
+<li><p><strong>input_sample_shape</strong> (<em>tuple</em>) â 3d tuple, (channel, height, width)</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Dense">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Dense</code><span class="sig-paren">(</span><em>name</em>, <em>num_output</em>, <em>use_bias=True</em>, <em>W_specs=None</em>, <em>b_specs=None</em>, <em>W_transpose=False</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Dense" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Apply linear/affine transformation, also called inner-product or
+fully connected layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>num_output</strong> (<em>int</em>) â output feature length.</p></li>
+<li><p><strong>use_bias</strong> (<em>bool</em>) â add a bias vector or not to the transformed feature</p></li>
+<li><p><strong>W_specs</strong> (<em>dict</em>) â specs for the weight matrix
+ânameâ for parameter name
+âlr_multâ for learning rate multiplier
+âdecay_multâ for weight decay multiplier
+âinitâ for init method, which could be âgaussianâ, âuniformâ,
+âxavierâ and ââ
+âstdâ, âmeanâ, âhighâ, âlowâ for corresponding init methods
+âclampâ for gradient constraint, value is scalar
+âregularizerâ for regularization, currently support âl2â</p></li>
+<li><p><strong>b_specs</strong> (<em>dict</em>) â specs for the bias vector, same fields as W_specs.</p></li>
+<li><p><strong>W_transpose</strong> (<em>bool</em>) â if true, output=x*W.T+b;</p></li>
+<li><p><strong>input_sample_shape</strong> (<em>tuple</em>) â input feature length</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Dropout">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Dropout</code><span class="sig-paren">(</span><em>name</em>, <em>p=0.5</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Dropout" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Droput layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>p</strong> (<em>float</em>) â probability for dropping out the element, i.e., set to 0</p></li>
+<li><p><strong>name</strong> (<em>string</em>) â layer name</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Activation">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Activation</code><span class="sig-paren">(</span><em>name</em>, <em>mode='relu'</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Activation" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Activation layers.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>name</strong> (<em>string</em>) â layer name</p></li>
+<li><p><strong>mode</strong> (<em>string</em>) â âreluâ, âsigmoidâ, or âtanhâ</p></li>
+<li><p><strong>input_sample_shape</strong> (<em>tuple</em>) â shape of a single sample</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Softmax">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Softmax</code><span class="sig-paren">(</span><em>name</em>, <em>axis=1</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Softmax" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Apply softmax.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>axis</strong> (<em>int</em>) â reshape the input as a matrix with the dimension
+[0,axis) as the row, the [axis, -1) as the column.</p></li>
+<li><p><strong>input_sample_shape</strong> (<em>tuple</em>) â shape of a single sample</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Flatten">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Flatten</code><span class="sig-paren">(</span><em>name</em>, <em>axis=1</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Flatten" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Reshape the input tensor into a matrix.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>axis</strong> (<em>int</em>) â reshape the input as a matrix with the dimension
+[0,axis) as the row, the [axis, -1) as the column.</p></li>
+<li><p><strong>input_sample_shape</strong> (<em>tuple</em>) â shape for a single sample</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Merge">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Merge</code><span class="sig-paren">(</span><em>name</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Merge" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Sum all input tensors.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>input_sample_shape</strong> â sample shape of the input. The sample shape of all
+inputs should be the same.</p>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.Merge.setup">
+<code class="descname">setup</code><span class="sig-paren">(</span><em>in_shape</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Merge.setup" title="Permalink to this definition">¶</a></dt>
+<dd><p>Call the C++ setup function to create params and set some meta data.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>in_shapes</strong> â if the layer accepts a single input Tensor, in_shapes is
+a single tuple specifying the inpute Tensor shape; if the layer
+accepts multiple input Tensor (e.g., the concatenation layer),
+in_shapes is a tuple of tuples, each for one input Tensor</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Merge.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Merge.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Merge.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>flag</em>, <em>inputs</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Merge.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Merge all input tensors by summation.</p>
+<p>TODO(wangwei) do element-wise merge operations, e.g., avg, count
+:param flag: not used.
+:param inputs: a list of tensors
+:type inputs: list</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>A single tensor as the sum of all input tensors</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Merge.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>flag</em>, <em>grad</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Merge.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Replicate the grad for each input source layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>grad</strong> (<a class="reference internal" href="tensor.html#singa.tensor.Tensor" title="singa.tensor.Tensor"><em>Tensor</em></a>) â </p>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p>A list of replicated grad, one per source layer</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Split">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Split</code><span class="sig-paren">(</span><em>name</em>, <em>num_output</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Split" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Replicate the input tensor.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>num_output</strong> (<em>int</em>) â number of output tensors to generate.</p></li>
+<li><p><strong>input_sample_shape</strong> â includes a single integer for the input sample
+feature size.</p></li>
+</ul>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.Split.setup">
+<code class="descname">setup</code><span class="sig-paren">(</span><em>in_shape</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Split.setup" title="Permalink to this definition">¶</a></dt>
+<dd><p>Call the C++ setup function to create params and set some meta data.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>in_shapes</strong> â if the layer accepts a single input Tensor, in_shapes is
+a single tuple specifying the inpute Tensor shape; if the layer
+accepts multiple input Tensor (e.g., the concatenation layer),
+in_shapes is a tuple of tuples, each for one input Tensor</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Split.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Split.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Split.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>flag</em>, <em>input</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Split.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Replicate the input tensor into mutiple tensors.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â not used</p></li>
+<li><p><strong>input</strong> â a single input tensor</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p>a list a output tensor (each one is a copy of the input)</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Split.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>flag</em>, <em>grads</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Split.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Sum all grad tensors to generate a single output tensor.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><p><strong>grads</strong> (<em>list of Tensor</em>) â </p>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p>a single tensor as the sum of all grads</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Concat">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Concat</code><span class="sig-paren">(</span><em>name</em>, <em>axis</em>, <em>input_sample_shapes=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Concat" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Concatenate tensors vertically (axis = 0) or horizontally (axis = 1).</p>
+<p>Currently, only support tensors with 2 dimensions.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>axis</strong> (<em>int</em>) â 0 for concat row; 1 for concat columns;</p></li>
+<li><p><strong>input_sample_shapes</strong> â a list of sample shape tuples, one per input tensor</p></li>
+</ul>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.Concat.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>flag</em>, <em>inputs</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Concat.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Concatenate all input tensors.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â same as Layer::forward()</p></li>
+<li><p><strong>input</strong> â a list of tensors</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p>a single concatenated tensor</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Concat.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>flag</em>, <em>dy</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Concat.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Backward propagate gradients through this layer.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â same as Layer::backward()</p></li>
+<li><p><strong>dy</strong> (<a class="reference internal" href="tensor.html#singa.tensor.Tensor" title="singa.tensor.Tensor"><em>Tensor</em></a>) â the gradient tensors of y w.r.t objective loss</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p><dl class="simple">
+<dt><dx, []>, dx is a list tensors for the gradient of the inputs; []</dt><dd><p>is an empty list.</p>
+</dd>
+</dl>
+</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.Slice">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">Slice</code><span class="sig-paren">(</span><em>name</em>, <em>axis</em>, <em>slice_point</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Slice" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Slice the input tensor into multiple sub-tensors vertially (axis=0) or
+horizontally (axis=1).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>axis</strong> (<em>int</em>) â 0 for slice rows; 1 for slice columns;</p></li>
+<li><p><strong>slice_point</strong> (<em>list</em>) â positions along the axis to do slice; there are n-1
+points for n sub-tensors;</p></li>
+<li><p><strong>input_sample_shape</strong> â input tensor sample shape</p></li>
+</ul>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.Slice.get_output_sample_shape">
+<code class="descname">get_output_sample_shape</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Slice.get_output_sample_shape" title="Permalink to this definition">¶</a></dt>
+<dd><p>Called after setup to get the shape of the output sample(s).</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>a tuple for a single output Tensor or a list of tuples if this layer
+has multiple outputs</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Slice.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>flag</em>, <em>x</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Slice.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Slice the input tensor on the given axis.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â same as Layer::forward()</p></li>
+<li><p><strong>x</strong> â a single input tensor</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p>a list a output tensor</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.Slice.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>flag</em>, <em>grads</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.Slice.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Concate all grad tensors to generate a single output tensor</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â same as Layer::backward()</p></li>
+<li><p><strong>grads</strong> â a list of tensors, one for the gradient of one sliced tensor</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p><dl class="simple">
+<dt>a single tensor for the gradient of the original user, and an empty</dt><dd><p>list.</p>
+</dd>
+</dl>
+</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.RNN">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">RNN</code><span class="sig-paren">(</span><em>name</em>, <em>hidden_size</em>, <em>rnn_mode='lstm'</em>, <em>dropout=0.0</em>, <em>num_stacks=1</em>, <em>input_mode='linear'</em>, <em>bidirectional=False</em>, <em>param_specs=None</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.RNN" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.Layer" title="singa.layer.Layer"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.Layer</span></code></a></p>
+<p>Recurrent layer with 4 types of units, namely lstm, gru, tanh and relu.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>hidden_size</strong> â hidden feature size, the same for all stacks of layers.</p></li>
+<li><p><strong>rnn_mode</strong> â decides the rnn unit, which could be one of âlstmâ, âgruâ,
+âtanhâ and âreluâ, refer to cudnn manual for each mode.</p></li>
+<li><p><strong>num_stacks</strong> â num of stacks of rnn layers. It is different to the
+unrolling seqence length.</p></li>
+<li><p><strong>input_mode</strong> â âlinearâ convert the input feature x by by a linear
+transformation to get a feature vector of size hidden_size;
+âskipâ does nothing but requires the input feature size equals
+hidden_size</p></li>
+<li><p><strong>bidirection</strong> â True for bidirectional RNN</p></li>
+<li><p><strong>param_specs</strong> â config for initializing the RNN parameters.</p></li>
+<li><p><strong>input_sample_shape</strong> â includes a single integer for the input sample
+feature size.</p></li>
+</ul>
+</dd>
+</dl>
+<dl class="method">
+<dt id="singa.layer.RNN.forward">
+<code class="descname">forward</code><span class="sig-paren">(</span><em>flag</em>, <em>inputs</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.RNN.forward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Forward inputs through the RNN.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>flag</strong> â True(kTrain) for training; False(kEval) for evaluation;
+others values for future use.</p></li>
+<li><p><strong><x1</strong><strong>, </strong><strong>x2</strong><strong>,</strong><strong>..xn</strong><strong>, </strong><strong>hx</strong><strong>, </strong><strong>cx></strong><strong>, </strong><strong>where xi is the input tensor for the</strong> (<em>inputs</em><em>,</em>) â i-th position, its shape is (batch_size, input_feature_length);
+the batch_size of xi must >= that of xi+1; hx is the initial
+hidden state of shape (num_stacks * bidirection?2:1, batch_size,
+hidden_size). cx is the initial cell state tensor of the same
+shape as hy. cx is valid for only lstm. For other RNNs there is
+no cx. Both hx and cx could be dummy tensors without shape and
+data.</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p><dl class="simple">
+<dt><y1, y2, ⦠yn, hy, cy>, where yi is the output tensor for the i-th</dt><dd><p>position, its shape is (batch_size,
+hidden_size * bidirection?2:1). hy is the final hidden state
+tensor. cx is the final cell state tensor. cx is only used for
+lstm.</p>
+</dd>
+</dl>
+</p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="method">
+<dt id="singa.layer.RNN.backward">
+<code class="descname">backward</code><span class="sig-paren">(</span><em>flag</em>, <em>grad</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.RNN.backward" title="Permalink to this definition">¶</a></dt>
+<dd><p>Backward gradients through the RNN.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters</dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>for future use.</strong> (<em>flag</em><em>,</em>) â </p></li>
+<li><p><strong><dy1</strong><strong>, </strong><strong>dy2</strong><strong>,</strong><strong>..dyn</strong><strong>, </strong><strong>dhy</strong><strong>, </strong><strong>dcy></strong><strong>, </strong><strong>where dyi is the gradient for the</strong> (<em>grad</em><em>,</em>) â </p></li>
+<li><p><strong>output</strong><strong>, </strong><strong>its shape is</strong><strong> (</strong><strong>batch_size</strong><strong>, </strong><strong>hidden_size*bidirection?2</strong> (<em>i-th</em>) â 1);
+dhy is the gradient for the final hidden state, its shape is
+(num_stacks * bidirection?2:1, batch_size,
+hidden_size). dcy is the gradient for the final cell state.
+cx is valid only for lstm. For other RNNs there is
+no cx. Both dhy and dcy could be dummy tensors without shape and
+data.</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns</dt>
+<dd class="field-even"><p><dl class="simple">
+<dt><dx1, dx2, ⦠dxn, dhx, dcx>, where dxi is the gradient tensor for</dt><dd><p>the i-th input, its shape is (batch_size,
+input_feature_length). dhx is the gradient for the initial
+hidden state. dcx is the gradient for the initial cell state,
+which is valid only for lstm.</p>
+</dd>
+</dl>
+</p>
+</dd>
+</dl>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.LSTM">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">LSTM</code><span class="sig-paren">(</span><em>name</em>, <em>hidden_size</em>, <em>dropout=0.0</em>, <em>num_stacks=1</em>, <em>input_mode='linear'</em>, <em>bidirectional=False</em>, <em>param_specs=None</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.LSTM" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.RNN" title="singa.layer.RNN"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.RNN</span></code></a></p>
+</dd></dl>
+
+<dl class="class">
+<dt id="singa.layer.GRU">
+<em class="property">class </em><code class="descclassname">singa.layer.</code><code class="descname">GRU</code><span class="sig-paren">(</span><em>name</em>, <em>hidden_size</em>, <em>dropout=0.0</em>, <em>num_stacks=1</em>, <em>input_mode='linear'</em>, <em>bidirectional=False</em>, <em>param_specs=None</em>, <em>input_sample_shape=None</em><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.GRU" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#singa.layer.RNN" title="singa.layer.RNN"><code class="xref py py-class docutils literal notranslate"><span class="pre">singa.layer.RNN</span></code></a></p>
+</dd></dl>
+
+<dl class="function">
+<dt id="singa.layer.get_layer_list">
+<code class="descclassname">singa.layer.</code><code class="descname">get_layer_list</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#singa.layer.get_layer_list" title="Permalink to this definition">¶</a></dt>
+<dd><p>Return a list of strings which include the identifiers (tags) of all
+supported layers</p>
+</dd></dl>
+
</div>
<div class="section" id="cpp-api">
<h2>CPP API<a class="headerlink" href="#cpp-api" title="Permalink to this headline">¶</a></h2>