You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by zh...@apache.org on 2017/01/09 04:10:21 UTC
[6/7] incubator-singa git commit: SINGA-274 Create Debian packages
SINGA-274 Create Debian packages
update the build.sh script to accept control argument (cuda, python, build, modules) for generate the packages.
jenkins can now invoke the build.sh to do the packaging automatically like that for wheel file.
refer to the README.md for details.
Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/f4e4a784
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/f4e4a784
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/f4e4a784
Branch: refs/heads/master
Commit: f4e4a7843fe335861ad244bfdaee05ebae662d50
Parents: 515ed0e
Author: wangwei <wa...@comp.nus.edu.sg>
Authored: Thu Jan 5 07:25:04 2017 +0000
Committer: wangwei <wa...@comp.nus.edu.sg>
Committed: Sun Jan 8 11:07:35 2017 +0800
----------------------------------------------------------------------
README.md | 124 +++++++++++--
tool/debian/build.sh | 182 ++++++++++++++-----
tool/debian/python-singa/DEBIAN/control | 13 ++
tool/debian/python-singa/DEBIAN/postinst | 4 +
tool/debian/python-singa/DEBIAN/postrm | 4 +
.../debian/singa-cpp_1.0.0_amd64/DEBIAN/control | 12 --
.../singa-python_1.0.0_amd64/DEBIAN/control | 15 --
.../singa-python_1.0.0_amd64/DEBIAN/postinst | 4 -
.../singa-python_1.0.0_amd64/DEBIAN/postrm | 4 -
tool/debian/singa/DEBIAN/control | 12 ++
tool/jenkins/README.md | 36 +++-
11 files changed, 306 insertions(+), 104 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 2673f58..e631d39 100644
--- a/README.md
+++ b/README.md
@@ -1,18 +1,120 @@
-#Apache SINGA
+# Jenkins CI Support
-[](https://travis-ci.org/apache/incubator-singa)
-
+## Introduction
+This documentation is to guide SINGA developers to setup Jenkins service.
-Distributed deep learning system
+We use jenkins to support continuous integration.
+After each commit, we want to automatically compile and test SINGA
+under different OS and settings.
+Those built binaries need to be archived for users to download.
-[http://singa.apache.org](http://singa.apache.org)
+## Install Jenkins
+[Jenkins Official Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins)
+The slave nodes for running different building environments are configured under 'Manage Jenkins'->'Manage nodes'.
-## Quick Start
+## Configure Jenkins Multi-configuration Project for Unit Testing and Package Generation
+Create a multi-configuration project and configure project as follows:
-* [Installation](doc/en/docs/installation.md)
-* [Examples](examples)
+### Description
+This job automatically pulls latest commits from Apache incubator-singa github repository, then for different environments
+* compile and test SINGA
+* create PySINGA wheel files
+* create Debian packages
-##Mailing Lists
+### General
+ * Discard old builds - Max # of builds to keep - 50
+ * GitHub project - ``https://github.com/apache/incubator-singa``
-* [Development Mailing List](mailto:dev-subscribe@singa.incubator.apache.org) ([Archive](http://mail-archives.apache.org/mod_mbox/singa-dev/))
-* [Commits Mailing List](mailto:commits-subscribe@singa.incubator.apache.org) ([Archive](http://mail-archives.apache.org/mod_mbox/singa-commits/))
+### Source Code Management
+ * Git - Repository URL - ``https://github.com/apache/incubator-singa``
+ * Git - Branch Specifier - ``*/master``
+
+### Build Triggers
+ * Poll SCM - Schedule - ``H/30 * * * *`` (pull every 30 minutes)
+
+### Configuration Matrix
+ * User-defined Axis - name ``lang`` values ``CPP CUDA``
+ * Slave - name ``env`` Node/label ``tick all nodes``
+
+### Build
+ * Execute shell - command - ``bash -ex tool/jenkins/jenkins_wheel.sh $lang``
+
+### Post-build Actions
+ * Publish JUnit test result report - Test report XMLs - ``**/gtest.xml, **/unittest.xml``
+ * (optional) Archive the artifacts - ``build/python/dist/**.whl, build/debian/**.deb``
+ * Send build artifacts (wheel) over SSH
+ * jenkins_wheel.sh packages the .whl file into $BUILD_ID.tar.gz. Inside the tar file,
+ the folder layout is `build_id/commit_hash/os_lang/*.whl`, where `os_lang` is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version.
+ * In `Manage Jenkins`-`Configure System`, configure the SSH for connecting to the remote public server and set the target folder location
+ * Source files - `build/python/dist/*.tar.gz`
+ * Remove prefix - `build/python/dist`
+ * Remote directory - `wheel`
+ * Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is
+
+ cd <target_folder>/wheel && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
+
+ * The file links on the remote public server would be like
+
+ wheel/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
+ wheel/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
+
+ * Send build artifacts (Debian package) over SSH for wheel
+ * ../debian/build.sh packages the .deb file into $BUILD_ID.tar.gz. Inside the tar file,
+ the folder layout is `build_id/commit_hash/os_lang/*.deb`, where `os_lang` is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version.
+ * In `Manage Jenkins`-`Configure System`, configure the SSH for connecting to the remote public server and set the target folder location
+ * Source files - `build/debian/*.tar.gz`
+ * Remove prefix - `build/debian
+ * Remote directory - `debian`
+ * Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is
+
+ cd <target_folder>/debian && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
+
+ * The file links on the remote public server would be like
+
+ debian/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
+ debian/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
+
+## Configure Jenkins for SINGA Website Updates
+
+### Description and Configuration
+
+This job is triggered upon any changes to the files of the `doc/` folder.
+It does the following tasks,
+
+1. installs the latest PySINGA
+2. pull the latest source code
+3. generate the html files for the documentation
+4. update the SINGA website
+
+The Jenkins job configuration is similar as above except the following fields,
+
+* Source Code Management - Git - Additional Behaviors - Include Region `doc/*`
+* Build - Execute Shell - Command `bash -ex tool/jenkins/jenkins_doc.sh`
+* No `Post-build Actions`
+
+### Docker Images
+
+The Docker image for the Jenkins slave node is at `docker/ubuntu16.04/runtime/Dockerfile`.
+To build the docker image,
+
+ # under the docker/ubuntu16.04/runtime/ folder
+ $ docker built -t singa:doc .
+
+To start the slave node
+
+ $ docker run --name singa-doc -d singa:doc
+ $ docker exec -it singa-doc /bin/bash
+ $ svn co https://svn.apache.org/repos/asf/incubator/singa/site/trunk
+ # update ~/.subversion/config to set 'store-password=yes'
+ # to set password free commit, we have to do a manual commit at first.
+ # change any file (add spaces) inside trunk/ to commit a message
+ $ svn commit -m "test" --username <committer id> --password <passwd>
+
+## Docker Images
+We provide in `docker` a number of singa docker images for Jenkins to use as slaves.
+To run the docker images,
+
+ nvidia-docker run --name <jenkins-slaveXX> -d <Image ID>
+
+## Access Control
+Use `Role Strategy Plugin` to give read access for anonymous users.
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/build.sh
----------------------------------------------------------------------
diff --git a/tool/debian/build.sh b/tool/debian/build.sh
index 3fb4858..8433516 100755
--- a/tool/debian/build.sh
+++ b/tool/debian/build.sh
@@ -16,52 +16,136 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# singa-cpp
-
-mkdir build
-cd build
-cmake ../../.. -DUSE_MODULES=ON
-make
-cd ..
-
-mkdir -p singa-cpp_1.0.0_amd64/usr/local/include
-cp -r ../../include/singa singa-cpp_1.0.0_amd64/usr/local/include
-cp -r build/include/singa/proto singa-cpp_1.0.0_amd64/usr/local/include/singa
-cp build/include/singa/singa_config.h singa-cpp_1.0.0_amd64/usr/local/include/singa
-
-mkdir -p singa-cpp_1.0.0_amd64/usr/local/lib
-cp build/lib/libsinga.so singa-cpp_1.0.0_amd64/usr/local/lib
-
-mkdir -p singa-cpp_1.0.0_amd64/usr/share/doc/singa
-cp ../../LICENSE singa-cpp_1.0.0_amd64/usr/share/doc/singa/
-
-dpkg -b singa-cpp_1.0.0_amd64
-
-rm -rf ./build
-
-
-# singa-python
-
-mkdir build
-cd build
-cmake ../../.. -DUSE_PYTHON=ON -DUSE_MODULES=ON
-make
-cd python
-make
-cd ../..
-
-mkdir -p singa-python_1.0.0_amd64/usr/local/lib/singa/singa/
-cp -r ../../python/singa singa-python_1.0.0_amd64/usr/local/lib/singa
-cp -r ../../python/rafiki singa-python_1.0.0_amd64/usr/local/lib/singa
-
-cp build/python/setup.py singa-python_1.0.0_amd64/usr/local/lib/singa
-cp build/python/singa/singa_wrap.py singa-python_1.0.0_amd64/usr/local/lib/singa/singa
-cp build/python/singa/_singa_wrap.so singa-python_1.0.0_amd64/usr/local/lib/singa/singa/
-cp -r build/python/singa/proto singa-python_1.0.0_amd64/usr/local/lib/singa/singa
-
-mkdir -p singa-python_1.0.0_amd64/usr/share/doc/singa
-cp ../../LICENSE singa-python_1.0.0_amd64/usr/share/doc/singa/
-
-dpkg -b singa-python_1.0.0_amd64
-
-rm -rf ./build
+# execute this script at SINGA_ROOT
+BUILD="OFF"
+CUDA="OFF"
+CUDNN="OFF"
+MODULES="OFF"
+PYTHON="OFF"
+
+echo "OS version: " $OS_VERSION
+COMMIT=`git rev-parse --short HEAD`
+echo "Commit ID: " $COMMIT
+
+while [[ $# -gt 0 ]]
+do
+key="$1"
+case $key in
+ -b|--build)
+ BUILD="ON"
+ shift
+ ;;
+ -m|--modules)
+ MODULES="ON"
+ shift
+ ;;
+ -c|--cuda|--CUDA)
+ CUDA="ON"
+ CUDNN="ON"
+ shift
+ ;;
+ -p|--python)
+ PYTHON="ON"
+ shift
+ ;;
+ -h|--help)
+ echo "Usage (execute at SINGA_ROOT folder): build.sh -b|--build -m|--modules -c|--cuda -p|--python"
+ echo " build: compile SINGA; if not set, we assume SINGA is compiled in ROOT/build/"
+ echo " modules: if set, protobuf and openblas are linked statically, hence they are removed from 'depends' in control file"
+ echo " cuda: if set, SINGA is compiled with CUDA and CUDNN"
+ echo " python: if set, package pysinga together"
+ exit 0
+ ;;
+ *)
+ echo "WRONG argument:" $key
+# for jenkins
+ shift
+ ;;
+esac
+done
+
+# get singa version
+SINGA_VERSION=`grep "PACKAGE_VERSION" CMakeLists.txt |sed 's/SET(PACKAGE_VERSION\s*\"\([0-9]\.[0-9]\.[0-9]\)\"\s*)/\1/'`
+echo "SINGA version: " $SINGA_VERSION
+
+
+# compile singa
+if [ $BUILD = "ON" ]; then
+ rm -rf build
+ mkdir build
+ cd build
+ cmake -DUSE_MODULES=$MODULES -DUSE_CUDA=$CUDA -DUSE_CUDNN=$CUDNN -DUSE_PYTHON=$PYTHON ..
+ make
+ cd ..
+fi
+
+# create the folder for the package
+FOLDER_PREFIX="singa"
+if [ -n ${BUILD_ID+x} ]; then
+ FOLDER_PREFIX=$BUILD_ID/$COMMIT/$OS_VERSION
+fi
+
+FOLDER=$FOLDER_PREFIX-cpp
+if [ $CUDA = "ON" ]; then
+ FOLDER=$FOLDER_PREFIX-cuda$CUDA_VERSION-cudnn$CUDNN_VERSION
+fi
+
+if [ $PYTHON = "ON" ]
+then
+ FOLDER=$FOLDER/python-singa
+else
+ FOLDER=$FOLDER/singa
+fi
+
+echo "Path: " build/debian/$FOLDER
+mkdir -p build/debian/$FOLDER
+
+if [ $PYTHON = "ON" ]
+then
+ cp -r tool/debian/python-singa/* build/debian/$FOLDER/
+else
+ cp -r tool/debian/singa/* build/debian/$FOLDER/
+fi
+
+# change SINGA version in the control file
+sed -i "s/\(Version: [0-9]\.[0-9]\.[0-9]\)/Version: $SINGA_VERSION/" build/debian/$FOLDER/DEBIAN/control
+
+# remove unnecessary dependencies
+if [ $MODULES = "ON" ]; then
+ sed -i 's/<libopenblas-dev\>,*//' build/debian/$FOLDER/DEBIAN/control
+ sed -i 's/<libprotobuf-dev\>,*//' build/debian/$FOLDER/DEBIAN/control
+ sed -i 's/<protobuf-compiler\>,*//' build/debian/$FOLDER/DEBIAN/control
+fi
+
+# copy cpp and cuda files
+ICL_FOLDER=build/debian/$FOLDER/usr/local/include
+mkdir -p ${ICL_FOLDER}
+cp -r include/singa ${ICL_FOLDER}/
+cp -r build/include/singa/proto ${ICL_FOLDER}/singa
+cp build/include/singa/singa_config.h ${ICL_FOLDER}/singa
+
+LIB_FOLDER=build/debian/$FOLDER/usr/local/lib
+mkdir -p ${LIB_FOLDER}
+cp build/lib/libsinga.so ${LIB_FOLDER}
+
+mkdir -p build/debian/$FOLDER/usr/share/doc/singa
+cp LICENSE build/debian/$FOLDER/usr/share/doc/singa
+
+# copy pysinga files
+if [ $PYTHON = "ON" ]; then
+ PY_FOLDER=build/debian/$FOLDER/usr/local/lib/singa
+ mkdir -p $PY_FOLDER
+ cp -r python/singa $PY_FOLDER/
+ cp -r python/rafiki $PY_FOLDER/
+
+ cp build/python/setup.py $PY_FOLDER/
+ cp build/python/singa/singa_wrap.py $PY_FOLDER/singa/
+ cp build/python/singa/_singa_wrap.so $PY_FOLDER/singa/
+ cp -r build/python/singa/proto $PY_FOLDER/singa/
+fi
+
+dpkg -b build/debian/$FOLDER/
+
+cd build/debian
+tar czf $BUILD_ID.tar.gz $FOLDER.deb
+exit 0
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/python-singa/DEBIAN/control
----------------------------------------------------------------------
diff --git a/tool/debian/python-singa/DEBIAN/control b/tool/debian/python-singa/DEBIAN/control
new file mode 100644
index 0000000..7d1bb76
--- /dev/null
+++ b/tool/debian/python-singa/DEBIAN/control
@@ -0,0 +1,13 @@
+Package: singa-python
+Version: 1.0.1
+Installed-Size: 30488
+Maintainer: Moaz Reyad <mo...@gmail.com>
+Section: devel
+Source: https://github.com/apache/incubator-singa
+Homepage: http://singa.apache.org
+Architecture: amd64
+Priority: optional
+Depends: libc6, libstdc++6, libprotobuf-dev, libopenblas-dev, libgoogle-glog-dev, python, libpython2.7, python-setuptools, python-numpy
+Description: Singa is an Apache Incubating project for developing an open source deep learning library.
+ The Singa project was initiated by the DB System Group at National University of Singapore in 2014. It focused on distributed deep learning by partitioning the model and data onto nodes in a cluster and parallelize the training. The prototype was accepted by Apache Incubator in March 2015.
+
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/python-singa/DEBIAN/postinst
----------------------------------------------------------------------
diff --git a/tool/debian/python-singa/DEBIAN/postinst b/tool/debian/python-singa/DEBIAN/postinst
new file mode 100755
index 0000000..7aee702
--- /dev/null
+++ b/tool/debian/python-singa/DEBIAN/postinst
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cd /usr/local/lib/singa
+python setup.py install
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/python-singa/DEBIAN/postrm
----------------------------------------------------------------------
diff --git a/tool/debian/python-singa/DEBIAN/postrm b/tool/debian/python-singa/DEBIAN/postrm
new file mode 100755
index 0000000..11add34
--- /dev/null
+++ b/tool/debian/python-singa/DEBIAN/postrm
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cd /usr/local/lib/singa
+rm -r build dist singa.egg-info
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/singa-cpp_1.0.0_amd64/DEBIAN/control
----------------------------------------------------------------------
diff --git a/tool/debian/singa-cpp_1.0.0_amd64/DEBIAN/control b/tool/debian/singa-cpp_1.0.0_amd64/DEBIAN/control
deleted file mode 100644
index 401370a..0000000
--- a/tool/debian/singa-cpp_1.0.0_amd64/DEBIAN/control
+++ /dev/null
@@ -1,12 +0,0 @@
-Package: Singa
-Version: 1.0.0
-Architecture: amd64
-Maintainer: Moaz Reyad <mo...@gmail.com>
-Depends: libgoogle-glog-dev, libprotobuf-dev, libstdc++6 , libc6
-Installed-Size: 24820
-Section: devel
-Priority: optional
-Source: https://github.com/apache/incubator-singa
-Homepage: http://singa.apache.org
-Description: Singa is an Apache Incubating project for developing an open source deep learning library.
- The Singa project was initiated by the DB System Group at National University of Singapore in 2014. It focused on distributed deep learning by partitioning the model and data onto nodes in a cluster and parallelize the training. The prototype was accepted by Apache Incubator in March 2015. Four versions have been released as shown In the following table. Since V1.0, SINGA is general to support traditional machine learning models such as logistic regression. Singa's software stack includes three major components, namely, core, IO and model. The core component provides memory management and tensor operations; IO has classes for reading (and writing) data from (to) disk and network; The model component provides data structures and algorithms for machine learning models, e.g., layers for neural network models, optimizers/initializer/metric/loss for general machine learning models.
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/singa-python_1.0.0_amd64/DEBIAN/control
----------------------------------------------------------------------
diff --git a/tool/debian/singa-python_1.0.0_amd64/DEBIAN/control b/tool/debian/singa-python_1.0.0_amd64/DEBIAN/control
deleted file mode 100644
index 949d124..0000000
--- a/tool/debian/singa-python_1.0.0_amd64/DEBIAN/control
+++ /dev/null
@@ -1,15 +0,0 @@
-Package: singa-python
-Version: 1.0.0
-Installed-Size: 30488
-Maintainer: Moaz Reyad <mo...@gmail.com>
-Section: devel
-Source: https://github.com/apache/incubator-singa
-Homepage: http://singa.apache.org
-Architecture: amd64
-Priority: optional
-Depends: libc6, libstdc++6, libprotobuf-dev, libgoogle-glog-dev, python, libpython2.7, python-setuptools, python-numpy
-Description: Singa is an Apache Incubating project for developing an open source deep learning library.
- The Singa project was initiated by the DB System Group at National University of Singapore in 2014. It focused on distributed deep learning by partitioning the model and data onto nodes in a cluster and parallelize the training. The prototype was accepted by Apache Incubator in March 2015. Four versions have been released as shown In the following table. Since V1.0, SINGA is general to support traditional machine learning models such as logistic regression. Singa's software stack includes three major components, namely, core, IO and model. The core component provides memory management and tensor operations; IO has classes for reading (and writing) data from (to) disk and network; The model component provides data structures and algorithms for machine learning models, e.g., layers for neural network models, optimizers/initializer/metric/loss for general machine learning models.
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postinst
----------------------------------------------------------------------
diff --git a/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postinst b/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postinst
deleted file mode 100755
index 7aee702..0000000
--- a/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postinst
+++ /dev/null
@@ -1,4 +0,0 @@
-#!/bin/bash
-
-cd /usr/local/lib/singa
-python setup.py install
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postrm
----------------------------------------------------------------------
diff --git a/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postrm b/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postrm
deleted file mode 100755
index 11add34..0000000
--- a/tool/debian/singa-python_1.0.0_amd64/DEBIAN/postrm
+++ /dev/null
@@ -1,4 +0,0 @@
-#!/bin/bash
-
-cd /usr/local/lib/singa
-rm -r build dist singa.egg-info
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/debian/singa/DEBIAN/control
----------------------------------------------------------------------
diff --git a/tool/debian/singa/DEBIAN/control b/tool/debian/singa/DEBIAN/control
new file mode 100644
index 0000000..8605427
--- /dev/null
+++ b/tool/debian/singa/DEBIAN/control
@@ -0,0 +1,12 @@
+Package: Singa
+Version: 1.0.1
+Architecture: amd64
+Maintainer: Moaz Reyad <mo...@gmail.com>
+Depends: libgoogle-glog-dev, libprotobuf-dev, libopenblas-dev, libstdc++6, libc6
+Installed-Size: 24820
+Section: devel
+Priority: optional
+Source: https://github.com/apache/incubator-singa
+Homepage: http://singa.apache.org
+Description: Singa is an Apache Incubating project for developing an open source deep learning library.
+ The Singa project was initiated by the DB System Group at National University of Singapore in 2014. It focused on distributed deep learning by partitioning the model and data onto nodes in a cluster and parallelize the training. The prototype was accepted by Apache Incubator in March 2015.
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/f4e4a784/tool/jenkins/README.md
----------------------------------------------------------------------
diff --git a/tool/jenkins/README.md b/tool/jenkins/README.md
index 339e532..e631d39 100644
--- a/tool/jenkins/README.md
+++ b/tool/jenkins/README.md
@@ -12,12 +12,14 @@ Those built binaries need to be archived for users to download.
[Jenkins Official Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins)
The slave nodes for running different building environments are configured under 'Manage Jenkins'->'Manage nodes'.
-## Configure Jenkins Multi-configuration Project for Unit Testing and PySINGA Generation
+## Configure Jenkins Multi-configuration Project for Unit Testing and Package Generation
Create a multi-configuration project and configure project as follows:
### Description
-This job automatically pulls latest commits from apache SINGA github repository.
-It compiles and tests SINGA in different environments and creates PySINGA wheel distribution accordingly.
+This job automatically pulls latest commits from Apache incubator-singa github repository, then for different environments
+* compile and test SINGA
+* create PySINGA wheel files
+* create Debian packages
### General
* Discard old builds - Max # of builds to keep - 50
@@ -39,22 +41,38 @@ It compiles and tests SINGA in different environments and creates PySINGA wheel
### Post-build Actions
* Publish JUnit test result report - Test report XMLs - ``**/gtest.xml, **/unittest.xml``
- * Archive the artifacts - ``build/python/dist/**.whl``
- * Send build artifacts over SSH
- * In jenkins_wheel.sh, package the .whl file into $BUILD_ID.tar.gz. Inside the tar file,
+ * (optional) Archive the artifacts - ``build/python/dist/**.whl, build/debian/**.deb``
+ * Send build artifacts (wheel) over SSH
+ * jenkins_wheel.sh packages the .whl file into $BUILD_ID.tar.gz. Inside the tar file,
the folder layout is `build_id/commit_hash/os_lang/*.whl`, where `os_lang` is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version.
* In `Manage Jenkins`-`Configure System`, configure the SSH for connecting to the remote public server and set the target folder location
* Source files - `build/python/dist/*.tar.gz`
* Remove prefix - `build/python/dist`
+ * Remote directory - `wheel`
* Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is
- cd <target_folder> && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
+ cd <target_folder>/wheel && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
* The file links on the remote public server would be like
- 32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
- 32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
+ wheel/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
+ wheel/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
+ * Send build artifacts (Debian package) over SSH for wheel
+ * ../debian/build.sh packages the .deb file into $BUILD_ID.tar.gz. Inside the tar file,
+ the folder layout is `build_id/commit_hash/os_lang/*.deb`, where `os_lang` is the combination of os version, device programming language (cuda/cpp/opencl) and cudnn version.
+ * In `Manage Jenkins`-`Configure System`, configure the SSH for connecting to the remote public server and set the target folder location
+ * Source files - `build/debian/*.tar.gz`
+ * Remove prefix - `build/debian
+ * Remote directory - `debian`
+ * Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is
+
+ cd <target_folder>/debian && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
+
+ * The file links on the remote public server would be like
+
+ debian/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
+ debian/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
## Configure Jenkins for SINGA Website Updates