You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by wa...@apache.org on 2017/01/16 08:02:30 UTC
[2/5] incubator-singa git commit: SINGA-276 Create docker images
SINGA-276 Create docker images
Created the dockerfiles for tag 'devel' and 'devel-cuda'.
They are described in doc/en/docs/docker.md file.
Instructions and naming rules of docker image tags are also included in that file.
Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/c3ce75e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/c3ce75e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/c3ce75e9
Branch: refs/heads/master
Commit: c3ce75e99346ff2b4b403d344080b5f261cc29a8
Parents: 8dac86c
Author: wangwei <wa...@comp.nus.edu.sg>
Authored: Fri Dec 23 10:06:24 2016 +0800
Committer: wangwei <wa...@comp.nus.edu.sg>
Committed: Fri Jan 13 13:35:16 2017 +0800
----------------------------------------------------------------------
doc/en/docs/docker.md | 58 ++++++++
tool/docker/README.md | 189 ---------------------------
tool/docker/devel/Dockerfile | 20 +++
tool/docker/devel/cuda/Dockerfile | 23 ++++
tool/docker/mesos/.bashrc | 32 -----
tool/docker/mesos/Dockerfile | 40 ------
tool/docker/mesos/core-site.xml | 24 ----
tool/docker/mesos/hdfs-site.xml | 34 -----
tool/docker/mesos/install.sh | 35 -----
tool/docker/mesos/mapred-site.xml | 22 ----
tool/docker/mesos/yarn-site.xml | 32 -----
tool/docker/singa/.bashrc | 30 -----
tool/docker/singa/Dockerfile | 34 -----
tool/docker/singa/Dockerfile_gpu | 80 ------------
tool/docker/singa/ssh.conf | 5 -
tool/mesos/docker/README.md | 189 +++++++++++++++++++++++++++
tool/mesos/docker/mesos/.bashrc | 32 +++++
tool/mesos/docker/mesos/Dockerfile | 40 ++++++
tool/mesos/docker/mesos/core-site.xml | 24 ++++
tool/mesos/docker/mesos/hdfs-site.xml | 34 +++++
tool/mesos/docker/mesos/install.sh | 35 +++++
tool/mesos/docker/mesos/mapred-site.xml | 22 ++++
tool/mesos/docker/mesos/yarn-site.xml | 32 +++++
tool/mesos/docker/singa/.bashrc | 30 +++++
tool/mesos/docker/singa/Dockerfile | 34 +++++
tool/mesos/docker/singa/Dockerfile_gpu | 80 ++++++++++++
tool/mesos/docker/singa/ssh.conf | 5 +
27 files changed, 658 insertions(+), 557 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/doc/en/docs/docker.md
----------------------------------------------------------------------
diff --git a/doc/en/docs/docker.md b/doc/en/docs/docker.md
new file mode 100644
index 0000000..8e5743e
--- /dev/null
+++ b/doc/en/docs/docker.md
@@ -0,0 +1,58 @@
+# Docker Images
+
+
+## Available tags
+
+* `devel`, with SINGA and the development packages installed on Ubuntu16.04 (no GPU)
+* `devel-cuda`, with SINGA, CUDA8.0, CUDNN5, and other development packages installed on Ubuntu16.04
+
+## Use the existing Docker images
+
+Users can pull the Docker images from Dockerhub via
+
+ docker pull apache/singa:devel
+ # or
+ docker pull apache/singa:devel-cuda
+
+Run the docker container using
+
+ docker run -it apache/singa:devel /bin/bash
+ # or
+ docker run -it apache/singa:devel-cuda /bin/bash
+
+The latest SINGA code is under the `incubator-singa` folder.
+
+## Create new Docker images from Dockerfile
+
+New Docker images could be created by executing the following command within the
+Dockerfile folder, e.g., tool/docker/devel/
+
+ docker build -t singa:<TAG> -f Dockerfile
+
+The `<TAG>` is named as
+
+ devel|runtime[-OS][-CUDA|OPENCL][-CUDNN]
+
+* devel: development images with all dependent libs' header files installed and SINGA's source code; runtime: the minimal images which can run SINGA programs.
+* OS: ubuntu, ubuntu14.04, centos, centos6
+* CUDA: cuda, cuda8.0, cuda7.0
+* CUDNN: cudnn, cudnn5, cudnn4
+* OPENCL: opencl, opencl1.2
+
+By default, if the version is not included in the tag, the latest stable version is used.
+The default OS is Ubuntu. The version is the latest stable version (e.g., 16.04 for now).
+For -cuda version, the **cudnn** is included by default. Their versions are also the latest stable version, i.e., cuda8.0 and cudnn5 for now.
+
+Here are some example tags,
+
+`devel`, `devel-cuda`, `runtime`, `runtime-cuda`, `devel-centos7-cuda`, `devel-ubuntu14.04`, `devel-ubuntu14.04-cuda7.5-cudnn4`
+
+Please follow the existing Dockefiles under tool/docker/ to create other Dockefiles.
+The folder structure is like
+
+ level1: devel|runtime
+ level2: Dockerfile, OS
+ level3: Dockerfile, CUDA|OPENCL
+ level4: CUDNN
+
+For example, the path of the Dockerfile for `devel-cuda` is `tool/docker/devel/cuda/Dockerfile`.
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/README.md
----------------------------------------------------------------------
diff --git a/tool/docker/README.md b/tool/docker/README.md
deleted file mode 100644
index fd33e7e..0000000
--- a/tool/docker/README.md
+++ /dev/null
@@ -1,189 +0,0 @@
-This guide explains how to set up a development environment for SINGA. A cluster environment can be launched over one or multiple physical nodes.
-
-1. [Build SINGA base](#build_base)
-2. [Build SINGA with Mesos and Hadoop](#build_mesos)
-3. [Pre-built images](#pre_built)
-4. [Launch and stop SINGA (stand alone mode)](#launch_stand_alone)
-5. [Launch pseudo-distributed SINGA on one node](#launch_pseudo)
-6. [Launch fully distributed SINGA on multiple nodes](#launch_distributed)
-
----
-
-<a name="build_base"></a>
-#### Build SINGA base image
-
-````
-$ cd tool/docker/singa
-$ sudo docker build -t singa/base .
-$ sudo docker images
-REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
-singa/base latest XXXX XXX 2.01 GB
-````
-
-The result is the image containing a built version of SINGA.
-
- 
-
- *Figure 1. singa/base Docker image, containing library dependencies and SINGA built from source.*
-
---
-
-<a name="build_mesos"></a>
-#### Build SINGA with Mesos and Hadoop
-````
-$ cd tool/docker/mesos
-$ sudo docker build -t singa/mesos .
-$ sudo docker images
-REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
-singa/mesos latest XXXX XXX 4.935 GB
-````
- 
-
- *Figure 2. singa/mesos Docker image, containing Hadoop and Mesos built on
-top of SINGA. The default namenode address for Hadoop is `node0:9000`*
-
-**Notes** A common failure observed during the build process is caused by network failure occuring when downloading dependencies. Simply re-run the build command.
-
---
-
-<a name="pre_built"></a>
-#### Pre-built images on epiC cluster
-For users with access to the `epiC` cluster, there are pre-built and loaded Docker images at the following nodes:
-
- ciidaa-c18
- ciidaa-c19
-
-The available images at those nodes are:
-````
-REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
-singa/base latest XXXX XXX 2.01 GB
-singa/mesos latest XXXX XXX 4.935 GB
-weaveworks/weaveexec 1.1.1 XXXX 11 days ago 57.8 MB
-weaveworks/weave 1.1.1 XXXX 11 days ago 17.56 MB
-````
-
---
-
-<a name="launch_stand_alone"></a>
-#### Launch and stop SINGA in stand-alone mode
-To launch a test environment for a single-node SINGA training, simply start a container from `singa/base` image. The following starts a container called
-`XYZ`, then launches a shell in the container:
-
-````
-$ sudo docker run -dt --name XYZ singa/base /usr/bin/supervisord
-$ sudo docker exec -it XYZ /bin/bash
-````
-
- 
-
- *Figure 3. Launch SINGA in stand-alone mode: single node training*
-
-Inside the launched container, the SINGA source directory can be found at `/root/incubator-singa`.
-
-**Stopping the container**
-
-````
-$ sudo docker stop XYZ
-$ sudo docker rm ZYZ
-````
-
---
-
-<a name="launch_pseudo"></a>
-#### Launch SINGA on pseudo-distributed mode (single node)
-To simulate a distributed environment on a single node, one can repeat the
-previous step multiple times, each time giving a different name to the
-container. Network connections between these containers are already supported,
-thus SINGA instances/nodes in these container can readily communicate with each
-other.
-
-The previous approach requires the user to start SINGA instances individually
-at each container. Although there's a bash script for that, we provide a better
-way. In particular, multiple containers can be started from `singa/mesos` image
-which already bundles Mesos and Hadoop with SINGA. Using Mesos makes it easy to
-launch, stop and monitor the distributed execution from a single container.
-Figure 4 shows `N+1` containers running concurrently at the local host.
-
-````
-# sudo docker run -dt --name node0 singa/mesos /usr/bin/supervisord
-# sudo docker run -dt --name node1 singa/mesos /usr/bin/supervisord
-...
-````
- 
-
- *Figure 4. Launch SINGA in pseudo-distributed mode : multiple SINGA nodes over one single machine*
-
-**Starting SINGA distributed training**
-
-Refer to the [Mesos
-guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md)
-for details of how to start training with multiple SINGA instances.
-
-**Important:** the container that assumes the role of Hadoop's namenode (and often Mesos's and Zookeeper's mater node as well) **must** be named `node0`. Otherwise, the user must log in to individual containers and change the Hadoop configuration separately.
-
---
-
-<a name="launch_distributed"></a>
-#### Launch SINGA on fully distributed mode (multiple nodes)
-The previous section has explained how to start a distributed environment on a
-single node. But running many containers on one node does not scale. When there
-are multiple physical hosts available, it is better to distribute the
-containers over them.
-
-The only extra requirement for the fully distributed mode, as compared with the
-pseudo distributed mode, is that the containers from different hosts are able
-to transparently communicate with each other. In the pseudo distributed mode,
-the local docker engine takes care of such communication. Here, we rely on
-[Weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) to make the
-communication transparent. The resulting architecture is shown below.
-
- 
-
- *Figure 5. Launch SINGA in fully distributed mode: multiple SINGA nodes over multiple machines*
-
-**Install Weave at all hosts**
-```
-$ curl -L git.io/weave -o /usr/local/bin/weave
-$ chmod a+x /usr/local/bin/weave
-```
-
-**Starting Weave**
-
-Suppose `node0` will be launched at host with IP `111.222.111.222`.
-
-* At host `111.222.111.222`:
-````
-$ weave launch
-$ eval "$(weave env)" //if there's error, do `sudo -s` and try again
-````
-* At other hosts:
-````
-$ weave launch 111.222.111.222
-$ eval "$(weave env)" //if there's error, do `sudo -s` and try again
-````
-
-**Starting containers**
-
-The user logs in to each host and starts the container (same as in [pseudo-distributed](#launch_pseudo) mode). Note that container acting as the head node of the cluster must be named `node0` (and be running at the host with IP `111.222.111.222`, for example).
-
-**_Important_:** when there are other containers sharing the same host as `node0`, say `node1` and `node2` for example,
-there're additional changes to be made to `node1` and `node2`. Particularly, log in to each container and edit
-`/etc/hosts` file:
-
-````
-# modified by weave
-...
-X.Y.Z node0 node0.bridge //<- REMOVE this line
-..
-````
-This is to ensure that name resolutions (of `node0`'s address) from `node1` and `node2` are correct. By default,
-containers of the same host resolves each other's addresses via the Docker bridge. Instead, we want they to use
-addressed given by Weave.
-
-
-**Starting SINGA distributed training**
-
-Refer to the [Mesos
-guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md)
-for details of how to start training with multiple SINGA instances.
-
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/devel/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/docker/devel/Dockerfile b/tool/docker/devel/Dockerfile
new file mode 100644
index 0000000..c8827cc
--- /dev/null
+++ b/tool/docker/devel/Dockerfile
@@ -0,0 +1,20 @@
+# Base unbuntu 16.04 image
+FROM ubuntu:latest
+
+MAINTAINER incubator-singa dev@singa.incubator.apache.org
+
+# install dependencies
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends build-essential git cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget\
+ && apt-get clean && apt-get autoremove && apt-get autoclean \
+ && rm -rf /var/lib/apt/lists/* \
+ && pip --no-cache-dir install -U pip numpy setuptools
+
+# set environment
+ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH}
+
+# download singa source
+RUN git clone https://github.com/apache/incubator-singa.git
+
+# compile singa and install pysinga
+RUN cd incubator-singa && mkdir build && cd build && cmake .. && make && cd python && pip install . && cd ../../ && rm -rf build
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/devel/cuda/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/docker/devel/cuda/Dockerfile b/tool/docker/devel/cuda/Dockerfile
new file mode 100644
index 0000000..76772ba
--- /dev/null
+++ b/tool/docker/devel/cuda/Dockerfile
@@ -0,0 +1,23 @@
+# Base unbuntu 16.04 image from nvidia/cuda
+# Change tags to build with different cuda/cudnn versions:
+# FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04
+FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04
+
+MAINTAINER incubator-singa dev@singa.incubator.apache.org
+
+# install dependencies
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends git g++ cmake libprotobuf-dev libopenblas-dev protobuf-compiler python-dev python-pip swig wget\
+ && apt-get clean && apt-get autoremove && apt-get autoclean \
+ && rm -rf /var/lib/apt/lists/* \
+ && pip --no-cache-dir install -U pip numpy setuptools
+
+# set environment
+ENV CPLUS_INCLUDE_PATH /usr/local/lib/python2.7/dist-packages/numpy/core/include:${CPLUS_INCLUDE_PATH}
+ENV CMAKE_INCLUDE_PATH /usr/local/cuda/include:${CMAKE_INCLUDE_PATH}
+ENV CMAKE_LIBRARY_PATH /usr/local/cuda/lib64:${CMAKE_LIBRARY_PATH}
+
+# download singa source
+RUN git clone https://github.com/apache/incubator-singa.git
+
+RUN cd incubator-singa && mkdir build && cd build && cmake -DUSE_CUDA=ON .. && make && cd python && pip install . && cd ../../ && rm -rf build
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/.bashrc
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/.bashrc b/tool/docker/mesos/.bashrc
deleted file mode 100644
index c796a27..0000000
--- a/tool/docker/mesos/.bashrc
+++ /dev/null
@@ -1,32 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH
-export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH
-export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
-export PATH=/opt/jdk1.8.0_60/bin:/opt/bin:$PATH
-export HADOOP_HOME=/opt/hadoop-2.6.0
-export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
-export JAVA_HOME=/opt/jdk1.8.0_60
-alias ls="ls --color=always"
-# some more ls aliases
-alias ll="ls -alF"
-alias la="ls -A"
-alias l="ls -CF"
-export SINGA_HOME=/root/incubator-singa
-export PATH=$PATH:$SINGA_HOME/bin
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/Dockerfile b/tool/docker/mesos/Dockerfile
deleted file mode 100644
index 83142b3..0000000
--- a/tool/docker/mesos/Dockerfile
+++ /dev/null
@@ -1,40 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-#build Mesos+Hadoop on top of SINGA
-FROM singa/base
-MAINTAINER Anh Dinh<ug...@gmail.com>
-
-COPY incubator-singa/tool/docker/mesos/.bashrc /root/.bashrc
-COPY incubator-singa/tool/docker/mesos/install.sh /opt/install.sh
-COPY incubator-singa/tool/mesos/* /root/incubator-singa/tool/mesos/
-
-RUN cd /opt && source ./install.sh
-
-RUN apt-get -y install curl cmake libxml2 libxml2-dev uuid-dev protobuf-compiler libprotobuf-dev libgsasl7-dev libkrb5-dev libboost1.54-all-dev
-
-RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz && tar -zxvf hadoop-2.6.0.tar.gz && git clone https://github.com/PivotalRD/libhdfs3.git && cd libhdfs3 && mkdir build && cd build && cmake ../ && make && make install
-
-#to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from the master)
-#RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git clone https://github.com/ug93tad/incubator-singa && cd incubator-singa && git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd /root/incubator-singa/tool/mesos && make
-
-# update SINGA, but keep the examples directory
-RUN source ~/.bashrc && cd /root/incubator-singa/tool/mesos && make
-
-COPY incubator-singa/tool/docker/mesos/*.xml /opt/hadoop-2.6.0/etc/hadoop/
-RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys && echo 'StrictHostKeyChecking no' >> ~/.ssh/config
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/core-site.xml
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/core-site.xml b/tool/docker/mesos/core-site.xml
deleted file mode 100644
index 49a39ca..0000000
--- a/tool/docker/mesos/core-site.xml
+++ /dev/null
@@ -1,24 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<!-- Put site-specific property overrides in this file. -->
-
-<configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://node0:9000</value>
- </property>
-</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/hdfs-site.xml
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/hdfs-site.xml b/tool/docker/mesos/hdfs-site.xml
deleted file mode 100644
index 2e0f4c6..0000000
--- a/tool/docker/mesos/hdfs-site.xml
+++ /dev/null
@@ -1,34 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<!-- Put site-specific property overrides in this file. -->
-
-<configuration>
-<property>
- <name>dfs.replication</name>
- <value>1</value>
- </property>
-
-<property>
- <name>dfs.name.dir</name>
- <value>file:///var/lib/hadoop/hdfs/namenode</value>
- </property>
-
- <property>
- <name>dfs.data.dir</name>
- <value>file:///var/lib/hadoop/hdfs/datanode</value>
- </property>
-</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/install.sh
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/install.sh b/tool/docker/mesos/install.sh
deleted file mode 100644
index 4b6ae79..0000000
--- a/tool/docker/mesos/install.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-source /root/.bashrc
-#download
-cd /opt
-wget -c http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz
-wget https://www.comp.nus.edu.sg/~dinhtta/files/mesos_patch
-tar -zxvf mesos-0.22.0.tar.gz
-
-#patch and install mesos
-cd /opt/mesos-0.22.0
-patch -p5 < ../mesos_patch
-mkdir build; cd build
-../configure
-make
-sudo make install
-
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/mapred-site.xml
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/mapred-site.xml b/tool/docker/mesos/mapred-site.xml
deleted file mode 100644
index 42bb81c..0000000
--- a/tool/docker/mesos/mapred-site.xml
+++ /dev/null
@@ -1,22 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<!-- Put site-specific property overrides in this file. -->
-
-<configuration>
- <property>
- <name>mapreduce.framework.name</name>
- <value>yarn</value>
- </property>
-</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/mesos/yarn-site.xml
----------------------------------------------------------------------
diff --git a/tool/docker/mesos/yarn-site.xml b/tool/docker/mesos/yarn-site.xml
deleted file mode 100644
index c137af9..0000000
--- a/tool/docker/mesos/yarn-site.xml
+++ /dev/null
@@ -1,32 +0,0 @@
-<?xml version="1.0"?>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-<configuration>
-
-<!-- Site specific YARN configuration properties -->
- <property>
- <name>yarn.nodemanager.aux-services</name>
- <value>mapreduce_shuffle</value>
- </property>
-
- <property>
- <name>yarn.resourcemanager.hostname</name>
- <value>node0</value>
- </property>
-
- <property>
- <name>yarn.resourcemanager.bind-host</name>
- <value>0.0.0.0</value>
- </property>
-</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/.bashrc
----------------------------------------------------------------------
diff --git a/tool/docker/singa/.bashrc b/tool/docker/singa/.bashrc
deleted file mode 100644
index cf0712f..0000000
--- a/tool/docker/singa/.bashrc
+++ /dev/null
@@ -1,30 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH
-export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH
-export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
-export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin
-export JAVA_HOME=/opt/jdk1.8.0_60
-export SINGA_HOME=/root/incubator-singa
-export PATH=$PATH:$SINGA_HOME/bin
-alias ls="ls --color=always"
-# some more ls aliases
-alias ll="ls -alF"
-alias la="ls -A"
-alias l="ls -CF"
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/docker/singa/Dockerfile b/tool/docker/singa/Dockerfile
deleted file mode 100644
index 0fc16bc..0000000
--- a/tool/docker/singa/Dockerfile
+++ /dev/null
@@ -1,34 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-#build singa from scratch
-FROM ubuntu:14.04
-MAINTAINER Anh Dinh<ug...@gmail.com>
-RUN rm /bin/sh && ln -s /bin/bash /bin/sh
-
-RUN apt-get update && apt-get -y install g++-4.8 build-essential git vim wget zip automake libtool man python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev openssh-server supervisor
-
-RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz && cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz
-
-COPY incubator-singa/tool/docker/singa/ssh.conf /etc/supervisor/conf.d/
-COPY incubator-singa/tool/docker/singa/.bashrc /root/.bashrc
-COPY incubator-singa/tool/docker/singa/.vimrc /root/.vimrc
-ADD incubator-singa /root/incubator-singa
-
-RUN cd && source /root/.bashrc && cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz && tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure && make clean && make && make install && cd ../../ && cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure && make && make install && cd examples/cifar10 && mv Makefile.example Makefile && make download && make create && service ssh start
-
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/Dockerfile_gpu
----------------------------------------------------------------------
diff --git a/tool/docker/singa/Dockerfile_gpu b/tool/docker/singa/Dockerfile_gpu
deleted file mode 100644
index da57479..0000000
--- a/tool/docker/singa/Dockerfile_gpu
+++ /dev/null
@@ -1,80 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements. See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership. The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License. You may obtain a copy of the License at
-# *
-# * http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-FROM singa/base
-MAINTAINER NVIDIA CORPORATION <di...@nvidia.com>
-
-LABEL com.nvidia.volumes.needed="nvidia_driver"
-
-ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c1c
-ENV NVIDIA_GPGKEY_FPR 889bee522da690103c4b085ed88c3d385c37d3be
-
-RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/GPGKEY && \
- apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +2 > cudasign.pub && \
- echo "$NVIDIA_GPGKEY_SUM cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && \
- echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list
-
-ENV CUDA_VERSION 7.0
-LABEL com.nvidia.cuda.version="7.0"
-
-ENV CUDA_PKG_VERSION 7-0=7.0-28
-RUN apt-get update && apt-get install -y --no-install-recommends --force-yes curl \
- cuda-nvrtc-$CUDA_PKG_VERSION \
- cuda-cusolver-$CUDA_PKG_VERSION \
- cuda-cublas-$CUDA_PKG_VERSION \
- cuda-cufft-$CUDA_PKG_VERSION \
- cuda-curand-$CUDA_PKG_VERSION \
- cuda-cusparse-$CUDA_PKG_VERSION \
- cuda-npp-$CUDA_PKG_VERSION \
- cuda-cudart-$CUDA_PKG_VERSION && \
- ln -s cuda-$CUDA_VERSION /usr/local/cuda
-
-RUN apt-get install -y --no-install-recommends --force-yes \
- cuda-core-$CUDA_PKG_VERSION \
- cuda-misc-headers-$CUDA_PKG_VERSION \
- cuda-command-line-tools-$CUDA_PKG_VERSION \
- cuda-license-$CUDA_PKG_VERSION \
- cuda-nvrtc-dev-$CUDA_PKG_VERSION \
- cuda-cusolver-dev-$CUDA_PKG_VERSION \
- cuda-cublas-dev-$CUDA_PKG_VERSION \
- cuda-cufft-dev-$CUDA_PKG_VERSION \
- cuda-curand-dev-$CUDA_PKG_VERSION \
- cuda-cusparse-dev-$CUDA_PKG_VERSION \
- cuda-npp-dev-$CUDA_PKG_VERSION \
- cuda-cudart-dev-$CUDA_PKG_VERSION \
- cuda-driver-dev-$CUDA_PKG_VERSION
-
-RUN echo "/usr/local/cuda/lib" >> /etc/ld.so.conf.d/cuda.conf && \
- echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf && \
- ldconfig
-
-ENV CUDNN_DOWNLOAD_SUM 98679d5ec039acfd4d81b8bfdc6a6352d6439e921523ff9909d364e706275c2b
-
-RUN curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v3/cudnn-7.0-linux-x64-v3.0-prod.tgz -O && \
- echo "$CUDNN_DOWNLOAD_SUM cudnn-7.0-linux-x64-v3.0-prod.tgz" | sha256sum -c --strict - && \
- tar -xzf cudnn-7.0-linux-x64-v3.0-prod.tgz -C /usr/local && \
- rm cudnn-7.0-linux-x64-v3.0-prod.tgz && \
- ldconfig
-
-RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
- echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
-
-ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
-ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH}
-
-RUN cd && source /root/.bashrc && cd incubator-singa && ./configure --enable-cuda --with-cuda=/usr/local/cuda --enable-cudnn --with-cudnn=/usr/local/cuda && make && make install
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/docker/singa/ssh.conf
----------------------------------------------------------------------
diff --git a/tool/docker/singa/ssh.conf b/tool/docker/singa/ssh.conf
deleted file mode 100644
index ac04d7e..0000000
--- a/tool/docker/singa/ssh.conf
+++ /dev/null
@@ -1,5 +0,0 @@
-[supervisord]
-nodaemon=true
-
-[program:ssh]
-command=/usr/sbin/sshd -D
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/README.md
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/README.md b/tool/mesos/docker/README.md
new file mode 100644
index 0000000..fd33e7e
--- /dev/null
+++ b/tool/mesos/docker/README.md
@@ -0,0 +1,189 @@
+This guide explains how to set up a development environment for SINGA. A cluster environment can be launched over one or multiple physical nodes.
+
+1. [Build SINGA base](#build_base)
+2. [Build SINGA with Mesos and Hadoop](#build_mesos)
+3. [Pre-built images](#pre_built)
+4. [Launch and stop SINGA (stand alone mode)](#launch_stand_alone)
+5. [Launch pseudo-distributed SINGA on one node](#launch_pseudo)
+6. [Launch fully distributed SINGA on multiple nodes](#launch_distributed)
+
+---
+
+<a name="build_base"></a>
+#### Build SINGA base image
+
+````
+$ cd tool/docker/singa
+$ sudo docker build -t singa/base .
+$ sudo docker images
+REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
+singa/base latest XXXX XXX 2.01 GB
+````
+
+The result is the image containing a built version of SINGA.
+
+ 
+
+ *Figure 1. singa/base Docker image, containing library dependencies and SINGA built from source.*
+
+--
+
+<a name="build_mesos"></a>
+#### Build SINGA with Mesos and Hadoop
+````
+$ cd tool/docker/mesos
+$ sudo docker build -t singa/mesos .
+$ sudo docker images
+REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
+singa/mesos latest XXXX XXX 4.935 GB
+````
+ 
+
+ *Figure 2. singa/mesos Docker image, containing Hadoop and Mesos built on
+top of SINGA. The default namenode address for Hadoop is `node0:9000`*
+
+**Notes** A common failure observed during the build process is caused by network failure occuring when downloading dependencies. Simply re-run the build command.
+
+--
+
+<a name="pre_built"></a>
+#### Pre-built images on epiC cluster
+For users with access to the `epiC` cluster, there are pre-built and loaded Docker images at the following nodes:
+
+ ciidaa-c18
+ ciidaa-c19
+
+The available images at those nodes are:
+````
+REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
+singa/base latest XXXX XXX 2.01 GB
+singa/mesos latest XXXX XXX 4.935 GB
+weaveworks/weaveexec 1.1.1 XXXX 11 days ago 57.8 MB
+weaveworks/weave 1.1.1 XXXX 11 days ago 17.56 MB
+````
+
+--
+
+<a name="launch_stand_alone"></a>
+#### Launch and stop SINGA in stand-alone mode
+To launch a test environment for a single-node SINGA training, simply start a container from `singa/base` image. The following starts a container called
+`XYZ`, then launches a shell in the container:
+
+````
+$ sudo docker run -dt --name XYZ singa/base /usr/bin/supervisord
+$ sudo docker exec -it XYZ /bin/bash
+````
+
+ 
+
+ *Figure 3. Launch SINGA in stand-alone mode: single node training*
+
+Inside the launched container, the SINGA source directory can be found at `/root/incubator-singa`.
+
+**Stopping the container**
+
+````
+$ sudo docker stop XYZ
+$ sudo docker rm ZYZ
+````
+
+--
+
+<a name="launch_pseudo"></a>
+#### Launch SINGA on pseudo-distributed mode (single node)
+To simulate a distributed environment on a single node, one can repeat the
+previous step multiple times, each time giving a different name to the
+container. Network connections between these containers are already supported,
+thus SINGA instances/nodes in these container can readily communicate with each
+other.
+
+The previous approach requires the user to start SINGA instances individually
+at each container. Although there's a bash script for that, we provide a better
+way. In particular, multiple containers can be started from `singa/mesos` image
+which already bundles Mesos and Hadoop with SINGA. Using Mesos makes it easy to
+launch, stop and monitor the distributed execution from a single container.
+Figure 4 shows `N+1` containers running concurrently at the local host.
+
+````
+# sudo docker run -dt --name node0 singa/mesos /usr/bin/supervisord
+# sudo docker run -dt --name node1 singa/mesos /usr/bin/supervisord
+...
+````
+ 
+
+ *Figure 4. Launch SINGA in pseudo-distributed mode : multiple SINGA nodes over one single machine*
+
+**Starting SINGA distributed training**
+
+Refer to the [Mesos
+guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md)
+for details of how to start training with multiple SINGA instances.
+
+**Important:** the container that assumes the role of Hadoop's namenode (and often Mesos's and Zookeeper's mater node as well) **must** be named `node0`. Otherwise, the user must log in to individual containers and change the Hadoop configuration separately.
+
+--
+
+<a name="launch_distributed"></a>
+#### Launch SINGA on fully distributed mode (multiple nodes)
+The previous section has explained how to start a distributed environment on a
+single node. But running many containers on one node does not scale. When there
+are multiple physical hosts available, it is better to distribute the
+containers over them.
+
+The only extra requirement for the fully distributed mode, as compared with the
+pseudo distributed mode, is that the containers from different hosts are able
+to transparently communicate with each other. In the pseudo distributed mode,
+the local docker engine takes care of such communication. Here, we rely on
+[Weave](http://weave.works/guides/weave-docker-ubuntu-simple.html) to make the
+communication transparent. The resulting architecture is shown below.
+
+ 
+
+ *Figure 5. Launch SINGA in fully distributed mode: multiple SINGA nodes over multiple machines*
+
+**Install Weave at all hosts**
+```
+$ curl -L git.io/weave -o /usr/local/bin/weave
+$ chmod a+x /usr/local/bin/weave
+```
+
+**Starting Weave**
+
+Suppose `node0` will be launched at host with IP `111.222.111.222`.
+
+* At host `111.222.111.222`:
+````
+$ weave launch
+$ eval "$(weave env)" //if there's error, do `sudo -s` and try again
+````
+* At other hosts:
+````
+$ weave launch 111.222.111.222
+$ eval "$(weave env)" //if there's error, do `sudo -s` and try again
+````
+
+**Starting containers**
+
+The user logs in to each host and starts the container (same as in [pseudo-distributed](#launch_pseudo) mode). Note that container acting as the head node of the cluster must be named `node0` (and be running at the host with IP `111.222.111.222`, for example).
+
+**_Important_:** when there are other containers sharing the same host as `node0`, say `node1` and `node2` for example,
+there're additional changes to be made to `node1` and `node2`. Particularly, log in to each container and edit
+`/etc/hosts` file:
+
+````
+# modified by weave
+...
+X.Y.Z node0 node0.bridge //<- REMOVE this line
+..
+````
+This is to ensure that name resolutions (of `node0`'s address) from `node1` and `node2` are correct. By default,
+containers of the same host resolves each other's addresses via the Docker bridge. Instead, we want they to use
+addressed given by Weave.
+
+
+**Starting SINGA distributed training**
+
+Refer to the [Mesos
+guide](https://github.com/ug93tad/incubator-singa/blob/SINGA-11/tool/mesos/README.md)
+for details of how to start training with multiple SINGA instances.
+
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/.bashrc
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/mesos/.bashrc b/tool/mesos/docker/mesos/.bashrc
new file mode 100644
index 0000000..c796a27
--- /dev/null
+++ b/tool/mesos/docker/mesos/.bashrc
@@ -0,0 +1,32 @@
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+
+export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH
+export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
+export PATH=/opt/jdk1.8.0_60/bin:/opt/bin:$PATH
+export HADOOP_HOME=/opt/hadoop-2.6.0
+export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
+export JAVA_HOME=/opt/jdk1.8.0_60
+alias ls="ls --color=always"
+# some more ls aliases
+alias ll="ls -alF"
+alias la="ls -A"
+alias l="ls -CF"
+export SINGA_HOME=/root/incubator-singa
+export PATH=$PATH:$SINGA_HOME/bin
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/mesos/Dockerfile b/tool/mesos/docker/mesos/Dockerfile
new file mode 100644
index 0000000..83142b3
--- /dev/null
+++ b/tool/mesos/docker/mesos/Dockerfile
@@ -0,0 +1,40 @@
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+
+#build Mesos+Hadoop on top of SINGA
+FROM singa/base
+MAINTAINER Anh Dinh<ug...@gmail.com>
+
+COPY incubator-singa/tool/docker/mesos/.bashrc /root/.bashrc
+COPY incubator-singa/tool/docker/mesos/install.sh /opt/install.sh
+COPY incubator-singa/tool/mesos/* /root/incubator-singa/tool/mesos/
+
+RUN cd /opt && source ./install.sh
+
+RUN apt-get -y install curl cmake libxml2 libxml2-dev uuid-dev protobuf-compiler libprotobuf-dev libgsasl7-dev libkrb5-dev libboost1.54-all-dev
+
+RUN cd /opt && source /root/.bashrc && wget -c http://www.eu.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz && tar -zxvf hadoop-2.6.0.tar.gz && git clone https://github.com/PivotalRD/libhdfs3.git && cd libhdfs3 && mkdir build && cd build && cmake ../ && make && make install
+
+#to be removed after SINGA-11 is merged (pulling from Anh's branch instead of pulling from the master)
+#RUN source ~/.bashrc && mkdir /root/mesos && cd /root/mesos && git clone https://github.com/ug93tad/incubator-singa && cd incubator-singa && git checkout SINGA-11 && cp -r tool/mesos /root/incubator-singa/tool/ && cd /root/incubator-singa/tool/mesos && make
+
+# update SINGA, but keep the examples directory
+RUN source ~/.bashrc && cd /root/incubator-singa/tool/mesos && make
+
+COPY incubator-singa/tool/docker/mesos/*.xml /opt/hadoop-2.6.0/etc/hadoop/
+RUN ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa && cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys && echo 'StrictHostKeyChecking no' >> ~/.ssh/config
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/core-site.xml
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/mesos/core-site.xml b/tool/mesos/docker/mesos/core-site.xml
new file mode 100644
index 0000000..49a39ca
--- /dev/null
+++ b/tool/mesos/docker/mesos/core-site.xml
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+ <property>
+ <name>fs.default.name</name>
+ <value>hdfs://node0:9000</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/hdfs-site.xml
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/mesos/hdfs-site.xml b/tool/mesos/docker/mesos/hdfs-site.xml
new file mode 100644
index 0000000..2e0f4c6
--- /dev/null
+++ b/tool/mesos/docker/mesos/hdfs-site.xml
@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+<property>
+ <name>dfs.replication</name>
+ <value>1</value>
+ </property>
+
+<property>
+ <name>dfs.name.dir</name>
+ <value>file:///var/lib/hadoop/hdfs/namenode</value>
+ </property>
+
+ <property>
+ <name>dfs.data.dir</name>
+ <value>file:///var/lib/hadoop/hdfs/datanode</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/install.sh
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/mesos/install.sh b/tool/mesos/docker/mesos/install.sh
new file mode 100644
index 0000000..4b6ae79
--- /dev/null
+++ b/tool/mesos/docker/mesos/install.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+
+source /root/.bashrc
+#download
+cd /opt
+wget -c http://archive.apache.org/dist/mesos/0.22.0/mesos-0.22.0.tar.gz
+wget https://www.comp.nus.edu.sg/~dinhtta/files/mesos_patch
+tar -zxvf mesos-0.22.0.tar.gz
+
+#patch and install mesos
+cd /opt/mesos-0.22.0
+patch -p5 < ../mesos_patch
+mkdir build; cd build
+../configure
+make
+sudo make install
+
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/mapred-site.xml
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/mesos/mapred-site.xml b/tool/mesos/docker/mesos/mapred-site.xml
new file mode 100644
index 0000000..42bb81c
--- /dev/null
+++ b/tool/mesos/docker/mesos/mapred-site.xml
@@ -0,0 +1,22 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+ <property>
+ <name>mapreduce.framework.name</name>
+ <value>yarn</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/mesos/yarn-site.xml
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/mesos/yarn-site.xml b/tool/mesos/docker/mesos/yarn-site.xml
new file mode 100644
index 0000000..c137af9
--- /dev/null
+++ b/tool/mesos/docker/mesos/yarn-site.xml
@@ -0,0 +1,32 @@
+<?xml version="1.0"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<configuration>
+
+<!-- Site specific YARN configuration properties -->
+ <property>
+ <name>yarn.nodemanager.aux-services</name>
+ <value>mapreduce_shuffle</value>
+ </property>
+
+ <property>
+ <name>yarn.resourcemanager.hostname</name>
+ <value>node0</value>
+ </property>
+
+ <property>
+ <name>yarn.resourcemanager.bind-host</name>
+ <value>0.0.0.0</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/.bashrc
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/singa/.bashrc b/tool/mesos/docker/singa/.bashrc
new file mode 100644
index 0000000..cf0712f
--- /dev/null
+++ b/tool/mesos/docker/singa/.bashrc
@@ -0,0 +1,30 @@
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+
+export LIBRARY_PATH=/opt/OpenBLAS/lib:$LIBRARY_PATH
+export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:/usr/local/lib:$LD_LIBRARY_PATH
+export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
+export PATH=$PATH:/opt/jdk1.8.0_60/bin:/opt/bin
+export JAVA_HOME=/opt/jdk1.8.0_60
+export SINGA_HOME=/root/incubator-singa
+export PATH=$PATH:$SINGA_HOME/bin
+alias ls="ls --color=always"
+# some more ls aliases
+alias ll="ls -alF"
+alias la="ls -A"
+alias l="ls -CF"
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/Dockerfile
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/singa/Dockerfile b/tool/mesos/docker/singa/Dockerfile
new file mode 100644
index 0000000..0fc16bc
--- /dev/null
+++ b/tool/mesos/docker/singa/Dockerfile
@@ -0,0 +1,34 @@
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+
+#build singa from scratch
+FROM ubuntu:14.04
+MAINTAINER Anh Dinh<ug...@gmail.com>
+RUN rm /bin/sh && ln -s /bin/bash /bin/sh
+
+RUN apt-get update && apt-get -y install g++-4.8 build-essential git vim wget zip automake libtool man python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev openssh-server supervisor
+
+RUN cd /opt && wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz && cd /opt && tar -zxvf jdk-8u60-linux-x64.tar.gz && rm -rf jdk-8u60-linux-x64.tar.gz
+
+COPY incubator-singa/tool/docker/singa/ssh.conf /etc/supervisor/conf.d/
+COPY incubator-singa/tool/docker/singa/.bashrc /root/.bashrc
+COPY incubator-singa/tool/docker/singa/.vimrc /root/.vimrc
+ADD incubator-singa /root/incubator-singa
+
+RUN cd && source /root/.bashrc && cd incubator-singa && ./thirdparty/install.sh all && cd thirdparty/ && wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz && tar -zxvf protobuf-2.5.0.tar.gz && cd protobuf-2.5.0 && ./configure && make clean && make && make install && cd ../../ && cp thirdparty/install.sh . && rm -rf thirdparty/* && mv install.sh thirdparty/ && ./thirdparty/install.sh zookeeper && ./autogen.sh && ./configure && make && make install && cd examples/cifar10 && mv Makefile.example Makefile && make download && make create && service ssh start
+
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/Dockerfile_gpu
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/singa/Dockerfile_gpu b/tool/mesos/docker/singa/Dockerfile_gpu
new file mode 100644
index 0000000..da57479
--- /dev/null
+++ b/tool/mesos/docker/singa/Dockerfile_gpu
@@ -0,0 +1,80 @@
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+
+FROM singa/base
+MAINTAINER NVIDIA CORPORATION <di...@nvidia.com>
+
+LABEL com.nvidia.volumes.needed="nvidia_driver"
+
+ENV NVIDIA_GPGKEY_SUM bd841d59a27a406e513db7d405550894188a4c1cd96bf8aa4f82f1b39e0b5c1c
+ENV NVIDIA_GPGKEY_FPR 889bee522da690103c4b085ed88c3d385c37d3be
+
+RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/GPGKEY && \
+ apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +2 > cudasign.pub && \
+ echo "$NVIDIA_GPGKEY_SUM cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && \
+ echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64 /" > /etc/apt/sources.list.d/cuda.list
+
+ENV CUDA_VERSION 7.0
+LABEL com.nvidia.cuda.version="7.0"
+
+ENV CUDA_PKG_VERSION 7-0=7.0-28
+RUN apt-get update && apt-get install -y --no-install-recommends --force-yes curl \
+ cuda-nvrtc-$CUDA_PKG_VERSION \
+ cuda-cusolver-$CUDA_PKG_VERSION \
+ cuda-cublas-$CUDA_PKG_VERSION \
+ cuda-cufft-$CUDA_PKG_VERSION \
+ cuda-curand-$CUDA_PKG_VERSION \
+ cuda-cusparse-$CUDA_PKG_VERSION \
+ cuda-npp-$CUDA_PKG_VERSION \
+ cuda-cudart-$CUDA_PKG_VERSION && \
+ ln -s cuda-$CUDA_VERSION /usr/local/cuda
+
+RUN apt-get install -y --no-install-recommends --force-yes \
+ cuda-core-$CUDA_PKG_VERSION \
+ cuda-misc-headers-$CUDA_PKG_VERSION \
+ cuda-command-line-tools-$CUDA_PKG_VERSION \
+ cuda-license-$CUDA_PKG_VERSION \
+ cuda-nvrtc-dev-$CUDA_PKG_VERSION \
+ cuda-cusolver-dev-$CUDA_PKG_VERSION \
+ cuda-cublas-dev-$CUDA_PKG_VERSION \
+ cuda-cufft-dev-$CUDA_PKG_VERSION \
+ cuda-curand-dev-$CUDA_PKG_VERSION \
+ cuda-cusparse-dev-$CUDA_PKG_VERSION \
+ cuda-npp-dev-$CUDA_PKG_VERSION \
+ cuda-cudart-dev-$CUDA_PKG_VERSION \
+ cuda-driver-dev-$CUDA_PKG_VERSION
+
+RUN echo "/usr/local/cuda/lib" >> /etc/ld.so.conf.d/cuda.conf && \
+ echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf && \
+ ldconfig
+
+ENV CUDNN_DOWNLOAD_SUM 98679d5ec039acfd4d81b8bfdc6a6352d6439e921523ff9909d364e706275c2b
+
+RUN curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v3/cudnn-7.0-linux-x64-v3.0-prod.tgz -O && \
+ echo "$CUDNN_DOWNLOAD_SUM cudnn-7.0-linux-x64-v3.0-prod.tgz" | sha256sum -c --strict - && \
+ tar -xzf cudnn-7.0-linux-x64-v3.0-prod.tgz -C /usr/local && \
+ rm cudnn-7.0-linux-x64-v3.0-prod.tgz && \
+ ldconfig
+
+RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
+ echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
+
+ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
+ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH}
+
+RUN cd && source /root/.bashrc && cd incubator-singa && ./configure --enable-cuda --with-cuda=/usr/local/cuda --enable-cudnn --with-cudnn=/usr/local/cuda && make && make install
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/c3ce75e9/tool/mesos/docker/singa/ssh.conf
----------------------------------------------------------------------
diff --git a/tool/mesos/docker/singa/ssh.conf b/tool/mesos/docker/singa/ssh.conf
new file mode 100644
index 0000000..ac04d7e
--- /dev/null
+++ b/tool/mesos/docker/singa/ssh.conf
@@ -0,0 +1,5 @@
+[supervisord]
+nodaemon=true
+
+[program:ssh]
+command=/usr/sbin/sshd -D