You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@apex.apache.org by th...@apache.org on 2018/05/10 04:13:04 UTC
[apex-core] branch master updated: APEXCORE-811 Added Docker build
files This is essentially migration of Dockerfile and related files from
repository: https://github.com/chinmaykolhatkar/docker-pool to Apache
apex-core repository.
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/apex-core.git
The following commit(s) were added to refs/heads/master by this push:
new 77136a5 APEXCORE-811 Added Docker build files This is essentially migration of Dockerfile and related files from repository: https://github.com/chinmaykolhatkar/docker-pool to Apache apex-core repository.
77136a5 is described below
commit 77136a52c8dc9b15f774ad9e0ee7b27878c8a48e
Author: Chinmay Kolhatkar <ch...@apache.org>
AuthorDate: Sat May 5 19:47:12 2018 +0530
APEXCORE-811 Added Docker build files
This is essentially migration of Dockerfile and related files from repository: https://github.com/chinmaykolhatkar/docker-pool to Apache apex-core repository.
The contributions to the previous repository were done by following people from Apache Apex community:
Chinmay Kolhatkar <ch...@apache.org>
Thomas Weise <th...@apache.org>
---
docker-sandbox/build.sh | 25 ++++++++++
docker-sandbox/ubuntu/Dockerfile | 35 ++++++++++++++
docker-sandbox/ubuntu/README.md | 37 +++++++++++++++
docker-sandbox/ubuntu/app/init.sh | 50 ++++++++++++++++++++
docker-sandbox/ubuntu/app/setup.sh | 95 ++++++++++++++++++++++++++++++++++++++
5 files changed, 242 insertions(+)
diff --git a/docker-sandbox/build.sh b/docker-sandbox/build.sh
new file mode 100755
index 0000000..155d3dd
--- /dev/null
+++ b/docker-sandbox/build.sh
@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Build ubuntu 16.04 docker image
+pushd ubuntu
+docker build -t apacheapex/sandbox:ubuntu-16.04 .
+popd
+
diff --git a/docker-sandbox/ubuntu/Dockerfile b/docker-sandbox/ubuntu/Dockerfile
new file mode 100644
index 0000000..4ea4c8b
--- /dev/null
+++ b/docker-sandbox/ubuntu/Dockerfile
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# How to run this script from same folder:
+# docker build .
+
+FROM ubuntu:16.04
+MAINTAINER Chinmay Kolhatkar <ch...@apache.org>
+MAINTAINER Thomas Weise <th...@apache.org>
+
+COPY app/ /app/
+RUN /app/setup.sh
+
+USER apex
+WORKDIR /home/apex
+EXPOSE 50070 8088
+
+ENTRYPOINT ["/app/init.sh"]
+
diff --git a/docker-sandbox/ubuntu/README.md b/docker-sandbox/ubuntu/README.md
new file mode 100644
index 0000000..7cc2ccb
--- /dev/null
+++ b/docker-sandbox/ubuntu/README.md
@@ -0,0 +1,37 @@
+#### [Apache Apex](http://apex.apache.org/) test sandbox
+This repository contains a ready to use [Apache Apex](http://apex.apache.org/) test sandbox docker image in which hadoop and apex is preinstalled and running. All one need to do is launch this docker image and start playing with apex.
+#### Basic
+##### Run apex docker container
+This will create and start a docker container from the docker image.
+```
+docker run -it --name=apex-sandbox apacheapex/sandbox
+```
+##### Start apex cli after launching docker container
+```
+apex@62550653e2d8:~$ apex
+Apex CLI 3.5.0 15.06.2016 @ 08:20:44 UTC rev: 85a2bdd branch: 85a2bdd9bfce49a904b45a4d0d015434d1a89216
+apex>
+```
+#### Advanced
+##### Linux user information
+Username: apex
+
+Password: apex
+
+**NOTE**: User "apex" is added in sudoers list and is provided with root privileges.
+
+##### Run docker container with host directory mounted
+This will create and start a docker container and docker images while mounting local filesystem directory as a mount point inside docker container.
+```
+docker run -it --name=apex-sandbox -v /local/path/to/mount:/mount_location apacheapex/sandbox
+```
+##### Start already created docker container
+```
+docker start -i apex-sandbox
+```
+##### Hadoop and YARN WebUI from local machine
+Following command will map yarn and hadoop ports exposed inside docker container to be mapped to ports of host machine.
+```
+docker run -it --name=apex-sandbox -p 50070:50070 -p 8088:8088 apacheapex/sandbox
+```
+After docker has started, one can point host machine's browser to *localhost:50070* and *localhost:8088* to see hadoop and yarn WebUI respectively.
diff --git a/docker-sandbox/ubuntu/app/init.sh b/docker-sandbox/ubuntu/app/init.sh
new file mode 100755
index 0000000..b28a912
--- /dev/null
+++ b/docker-sandbox/ubuntu/app/init.sh
@@ -0,0 +1,50 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+. /lib/lsb/init-functions
+
+set -e
+
+echo -n "* Starting hadoop services. This might take few seconds."
+echo -n "."; sudo /etc/init.d/hadoop-hdfs-namenode restart >/dev/null \
+ && echo -n "."; sudo /etc/init.d/hadoop-hdfs-datanode restart >/dev/null \
+ && echo -n "."; sudo /etc/init.d/hadoop-yarn-resourcemanager restart >/dev/null \
+ && echo -n "."; sudo /etc/init.d/hadoop-yarn-nodemanager restart >/dev/null \
+ && echo -n "."; sudo /etc/init.d/hadoop-yarn-timelineserver restart >/dev/null \
+ && echo -n "."; sudo /etc/init.d/ssh start >/dev/null
+log_end_msg $?
+
+echo
+echo "====================================="
+echo " Welcome to Apache Apex Test Sandbox "
+echo "====================================="
+echo "This docker image uses bigtop package of hadoop and apex."
+echo "This image provides a ready to use environment for quickly launching apex application."
+echo "Currently running docker container has hadoop services initialized and started."
+echo
+echo "Just type \"apex\" on command line to get apex cli console. See man page of apex for details."
+echo "Enjoy Apexing!!!"
+echo
+echo "=====Information about Container====="
+echo "IPv4 Address: $(hostname -i)"
+echo "Hostname: $(hostname)"
+echo
+
+/bin/bash
diff --git a/docker-sandbox/ubuntu/app/setup.sh b/docker-sandbox/ubuntu/app/setup.sh
new file mode 100755
index 0000000..b5beb41
--- /dev/null
+++ b/docker-sandbox/ubuntu/app/setup.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Install softwares
+apt-get update -y
+apt-get -y -o APT::Immediate-Configure=false install wget
+wget -O- http://archive.apache.org/dist/bigtop/bigtop-1.2.1/repos/GPG-KEY-bigtop | apt-key add -
+wget -O /etc/apt/sources.list.d/bigtop-1.2.1.list http://archive.apache.org/dist/bigtop/bigtop-1.2.1/repos/ubuntu16.04/bigtop.list
+apt-get update -y
+
+apt-get install -y -q --no-install-recommends openjdk-8-jre-headless vim screen curl sudo unzip man openssh-server unzip hadoop\*
+
+# Download atrato apex cli package temporarily to build the docker image. Later this should be replaced with
+# the one built from dist folder. Captured in APEXCORE-813.
+wget https://github.com/atrato/apex-cli-package/releases/download/v3.7.0/apex-cli-package-3.7.0-bin.zip
+unzip apex-cli-package-3.7.0-bin.zip
+pushd apex-cli-package-3.7.0
+mkdir -p /usr/lib/apex
+mv * /usr/lib/apex
+popd
+rm -rf apex-cli-package-3.7.0
+rm apex-cli-package-3.7.0-bin.zip
+ln -s /usr/lib/apex/bin/apex /usr/bin/apex
+
+# Autodetect JAVA_HOME if not defined
+. /usr/lib/bigtop-utils/bigtop-detect-javahome
+
+## turn off YARN nodemanager vmem check
+sed -i 's#</configuration>##' /etc/hadoop/conf/yarn-site.xml
+cat >> /etc/hadoop/conf/yarn-site.xml << EOF
+ <property>
+ <name>yarn.nodemanager.vmem-check-enabled</name>
+ <value>false</value>
+ </property>
+</configuration>
+EOF
+
+## enable WebHDFS and append
+sed -i 's#</configuration>##' /etc/hadoop/conf/hdfs-site.xml
+cat >> /etc/hadoop/conf/hdfs-site.xml << EOF
+ <property>
+ <name>dfs.webhdfs.enabled</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>dfs.support.append</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>dfs.support.broken.append</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>dfs.permissions.enabled</name>
+ <value>false</value>
+ </property>
+</configuration>
+EOF
+
+## format NameNode
+/etc/init.d/hadoop-hdfs-namenode init
+## start HDFS
+for i in hadoop-hdfs-namenode hadoop-hdfs-datanode ; do service $i start ; done
+## initialize HDFS
+/usr/lib/hadoop/libexec/init-hdfs.sh
+## stop HDFS
+for i in hadoop-hdfs-namenode hadoop-hdfs-datanode ; do service $i stop ; done
+## clean up
+apt-get autoclean
+rm -rf /var/lib/apt/lists/*
+rm -rf /var/log/hadoop-hdfs/*
+
+# Creating user
+echo 'root:sc@mb0t' |chpasswd
+useradd apex -s /bin/bash -U -G sudo -p apex -m
+echo "apex:apex" |chpasswd
+echo 'apex ALL=(ALL) NOPASSWD: /etc/init.d/hadoop*' >> /etc/sudoers
+echo 'apex ALL=(ALL) NOPASSWD: /etc/init.d/ssh*' >> /etc/sudoers
--
To stop receiving notification emails like this one, please contact
thw@apache.org.