You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by sh...@apache.org on 2020/02/28 10:16:27 UTC
[kylin] branch master updated: KYLIN-4388 Refine the Dockerfile
This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/master by this push:
new 1a29468 KYLIN-4388 Refine the Dockerfile
1a29468 is described below
commit 1a29468854a1ddd46ac41642d08cd6ef49359a39
Author: shaofengshi <sh...@apache.org>
AuthorDate: Thu Feb 27 14:24:48 2020 +0800
KYLIN-4388 Refine the Dockerfile
---
docker/Dockerfile | 123 +++++---------------------
docker/Dockerfile_dev | 43 +++++++++
docker/{Dockerfile => Dockerfile_hadoop} | 23 +----
docker/README.md | 145 +++++++++++++++++++++++++++++++
docker/build_image.sh | 19 +---
docker/entrypoint.sh | 5 ++
docker/run_container.sh | 2 +-
docker/use-guide.md | 136 -----------------------------
8 files changed, 218 insertions(+), 278 deletions(-)
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 0655bce..4ca399f 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -15,110 +15,27 @@
# limitations under the License.
#
-# Docker image for apache kylin
-FROM centos:6.9
+# Docker image for apache kylin, based on the Hadoop image
+FROM hadoop2.7-all-in-one
+
+ENV KYLIN_VERSION 3.0.1
+ENV KYLIN_HOME /home/admin/apache-kylin-$KYLIN_VERSION-bin-hbase1x
+
+# Download released Kylin
+RUN wget https://archive.apache.org/dist/kylin/apache-kylin-$KYLIN_VERSION/apache-kylin-$KYLIN_VERSION-bin-hbase1x.tar.gz \
+ && tar -zxvf /home/admin/apache-kylin-$KYLIN_VERSION-bin-hbase1x.tar.gz \
+ && rm -f /home/admin/apache-kylin-$KYLIN_VERSION-bin-hbase1x.tar.gz
+
+RUN echo "kylin.engine.spark-conf.spark.executor.memory=1G" >> $KYLIN_HOME/conf/kylin.properties \
+ && echo "kylin.engine.spark-conf-mergedict.spark.executor.memory=1.5G" >> $KYLIN_HOME/conf/kylin.properties \
+ && echo "kylin.engine.livy-conf.livy-url=http://127.0.0.1:8998" >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.livy-conf.livy-key.file=hdfs://localhost:9000/kylin/livy/kylin-job-$KYLIN_VERSION.jar >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.livy-conf.livy-arr.jars=hdfs://localhost:9000/kylin/livy/hbase-client-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-common-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop2-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-server-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/htrace-core-*-incubating.jar,hdfs://localhost:9000/kylin/livy/metrics- [...]
+ && echo kylin.source.hive.quote-enabled=false >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.spark-conf.spark.eventLog.dir=hdfs://localhost:9000/kylin/spark-history >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs://localhost:9000/kylin/spark-history >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.source.hive.redistribute-flat-table=false >> $KYLIN_HOME/conf/kylin.properties
-MAINTAINER weibin0516 <co...@126.com>
-
-ENV HIVE_VERSION 1.2.1
-ENV HADOOP_VERSION 2.7.0
-ENV HBASE_VERSION 1.1.2
-ENV SPARK_VERSION 2.3.1
-ENV ZK_VERSION 3.4.6
-ENV KAFKA_VERSION 1.1.1
-ENV LIVY_VERSION 0.6.0
-ENV KYLIN_VERSION 3.0.0-alpha2
-
-ENV JAVA_HOME /home/admin/jdk1.8.0_141
-ENV MVN_HOME /home/admin/apache-maven-3.6.1
-ENV HADOOP_HOME /home/admin/hadoop-$HADOOP_VERSION
-ENV HIVE_HOME /home/admin/apache-hive-$HIVE_VERSION-bin
-ENV HADOOP_CONF $HADOOP_HOME/etc/hadoop
-ENV HADOOP_CONF_DIR $HADOOP_HOME/etc/hadoop
-ENV HBASE_HOME /home/admin/hbase-$HBASE_VERSION
-ENV SPARK_HOME /home/admin/spark-$SPARK_VERSION-bin-hadoop2.6
-ENV SPARK_CONF_DIR /home/admin/spark-$SPARK_VERSION-bin-hadoop2.6/conf
-ENV ZK_HOME /home/admin/zookeeper-$ZK_VERSION
-ENV KAFKA_HOME /home/admin/kafka_2.11-$KAFKA_VERSION
-ENV LIVY_HOME /home/admin/apache-livy-$LIVY_VERSION-incubating-bin
-ENV PATH $PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:$MVN_HOME/bin:spark-$SPARK_VERSION-bin-hadoop2.6/bin:$KAFKA_HOME/bin
-
-USER root
-
-WORKDIR /home/admin
-
-# install tools
-RUN yum -y install lsof.x86_64 wget.x86_64 tar.x86_64 git.x86_64 mysql-server.x86_64 mysql.x86_64 unzip.x86_64
-
-# install mvn
-RUN wget https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz \
- && tar -zxvf apache-maven-3.6.1-bin.tar.gz \
- && rm -f apache-maven-3.6.1-bin.tar.gz
-COPY conf/maven/settings.xml $MVN_HOME/conf/settings.xml
-
-# install npm
-RUN curl -sL https://rpm.nodesource.com/setup_8.x | bash - \
- && yum install -y nodejs
-
-# setup jdk
-RUN wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz" \
- && tar -zxvf /home/admin/jdk-8u141-linux-x64.tar.gz \
- && rm -f /home/admin/jdk-8u141-linux-x64.tar.gz
-
-# setup hadoop
-RUN wget https://archive.apache.org/dist/hadoop/core/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz \
- && tar -zxvf /home/admin/hadoop-$HADOOP_VERSION.tar.gz \
- && rm -f /home/admin/hadoop-$HADOOP_VERSION.tar.gz \
- && mkdir -p /data/hadoop
-COPY conf/hadoop/* $HADOOP_CONF/
-
-# setup hbase
-RUN wget https://archive.apache.org/dist/hbase/$HBASE_VERSION/hbase-$HBASE_VERSION-bin.tar.gz \
- && tar -zxvf /home/admin/hbase-$HBASE_VERSION-bin.tar.gz \
- && rm -f /home/admin/hbase-$HBASE_VERSION-bin.tar.gz \
- && mkdir -p /data/hbase \
- && mkdir -p /data/zookeeper
-COPY conf/hbase/hbase-site.xml $HBASE_HOME/conf
-
-# setup hive
-RUN wget https://archive.apache.org/dist/hive/hive-$HIVE_VERSION/apache-hive-$HIVE_VERSION-bin.tar.gz \
- && tar -zxvf /home/admin/apache-hive-$HIVE_VERSION-bin.tar.gz \
- && rm -f /home/admin/apache-hive-$HIVE_VERSION-bin.tar.gz \
- && wget -P $HIVE_HOME/lib https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.24/mysql-connector-java-5.1.24.jar
-COPY conf/hive/hive-site.xml $HIVE_HOME/conf
-
-# setup spark
-RUN wget https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.6.tgz \
- && tar -zxvf /home/admin/spark-$SPARK_VERSION-bin-hadoop2.6.tgz \
- && rm -f /home/admin/spark-$SPARK_VERSION-bin-hadoop2.6.tgz \
- && cp $HIVE_HOME/conf/hive-site.xml $SPARK_HOME/conf \
- && cp $SPARK_HOME/yarn/*.jar $HADOOP_HOME/share/hadoop/yarn/lib
-RUN cp $HIVE_HOME/lib/mysql-connector-java-5.1.24.jar $SPARK_HOME/jars
-RUN echo spark.sql.catalogImplementation=hive > $SPARK_HOME/conf/spark-defaults.conf
-
-# setup kafka
-RUN wget https://archive.apache.org/dist/kafka/$KAFKA_VERSION/kafka_2.11-$KAFKA_VERSION.tgz \
- && tar -zxvf /home/admin/kafka_2.11-$KAFKA_VERSION.tgz \
- && rm -f /home/admin/kafka_2.11-$KAFKA_VERSION.tgz
-
-# setup livy
-RUN wget https://www.apache.org/dist/incubator/livy/$LIVY_VERSION-incubating/apache-livy-$LIVY_VERSION-incubating-bin.zip \
- && unzip /home/admin/apache-livy-$LIVY_VERSION-incubating-bin.zip \
- && rm -f /home/admin/apache-livy-$LIVY_VERSION-incubating-bin.zip
-
-# copy kylin source code
-COPY ./kylin /home/admin/kylin_sourcecode
-
-RUN sed -i -e 's#<!--<version>3.4.1</version>-->#<version>3.4.1</version>#g' /home/admin/kylin_sourcecode/pom.xml \
- && echo "kylin.engine.spark-conf.spark.executor.memory=1G" >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo "kylin.engine.spark-conf-mergedict.spark.executor.memory=1.5G" >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo "kylin.engine.livy-conf.livy-url=http://127.0.0.1:8998" >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.livy-conf.livy-key.file=hdfs://localhost:9000/kylin/livy/kylin-job-$KYLIN_VERSION.jar >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.livy-conf.livy-arr.jars=hdfs://localhost:9000/kylin/livy/hbase-client-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-common-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop2-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-server-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/htrace-core-*-incubating.jar,hdfs://localhost:9000/kylin/livy/metrics- [...]
- && echo kylin.source.hive.quote-enabled=false >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.spark-conf.spark.eventLog.dir=hdfs://localhost:9000/kylin/spark-history >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs://localhost:9000/kylin/spark-history >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.source.hive.redistribute-flat-table=false >> /home/admin/kylin_sourcecode/build/conf/kylin.properties
COPY ./entrypoint.sh /home/admin/entrypoint.sh
RUN chmod u+x /home/admin/entrypoint.sh
diff --git a/docker/Dockerfile_dev b/docker/Dockerfile_dev
new file mode 100644
index 0000000..132973d
--- /dev/null
+++ b/docker/Dockerfile_dev
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Docker image for apache kylin, based on the Hadoop image
+FROM hadoop2.7-all-in-one
+
+## for dev
+ENV KYLIN_VERSION 3.1.0-SNAPSHOT
+ENV KYLIN_HOME /home/admin/apache-kylin-$KYLIN_VERSION-bin
+# Copy compiled Kylin from local
+COPY apache-kylin-$KYLIN_VERSION-bin.tar.gz /home/admin/
+RUN tar -zxvf /home/admin/apache-kylin-$KYLIN_VERSION-bin.tar.gz \
+ && rm -f /home/admin/apache-kylin-$KYLIN_VERSION-bin.tar.gz
+
+RUN echo "kylin.engine.spark-conf.spark.executor.memory=1G" >> $KYLIN_HOME/conf/kylin.properties \
+ && echo "kylin.engine.spark-conf-mergedict.spark.executor.memory=1.5G" >> $KYLIN_HOME/conf/kylin.properties \
+ && echo "kylin.engine.livy-conf.livy-url=http://127.0.0.1:8998" >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.livy-conf.livy-key.file=hdfs://localhost:9000/kylin/livy/kylin-job-$KYLIN_VERSION.jar >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.livy-conf.livy-arr.jars=hdfs://localhost:9000/kylin/livy/hbase-client-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-common-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop2-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-server-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/htrace-core-*-incubating.jar,hdfs://localhost:9000/kylin/livy/metrics- [...]
+ && echo kylin.source.hive.quote-enabled=false >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.spark-conf.spark.eventLog.dir=hdfs://localhost:9000/kylin/spark-history >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs://localhost:9000/kylin/spark-history >> $KYLIN_HOME/conf/kylin.properties \
+ && echo kylin.source.hive.redistribute-flat-table=false >> $KYLIN_HOME/conf/kylin.properties
+
+
+COPY ./entrypoint.sh /home/admin/entrypoint.sh
+RUN chmod u+x /home/admin/entrypoint.sh
+
+ENTRYPOINT ["/home/admin/entrypoint.sh"]
diff --git a/docker/Dockerfile b/docker/Dockerfile_hadoop
similarity index 69%
copy from docker/Dockerfile
copy to docker/Dockerfile_hadoop
index 0655bce..a0312f4 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile_hadoop
@@ -15,11 +15,9 @@
# limitations under the License.
#
-# Docker image for apache kylin
+# Docker image with Hadoop/Hive/HBase/Spark/ZK/Kafka/Livy installed
FROM centos:6.9
-MAINTAINER weibin0516 <co...@126.com>
-
ENV HIVE_VERSION 1.2.1
ENV HADOOP_VERSION 2.7.0
ENV HBASE_VERSION 1.1.2
@@ -27,7 +25,6 @@ ENV SPARK_VERSION 2.3.1
ENV ZK_VERSION 3.4.6
ENV KAFKA_VERSION 1.1.1
ENV LIVY_VERSION 0.6.0
-ENV KYLIN_VERSION 3.0.0-alpha2
ENV JAVA_HOME /home/admin/jdk1.8.0_141
ENV MVN_HOME /home/admin/apache-maven-3.6.1
@@ -106,21 +103,3 @@ RUN wget https://www.apache.org/dist/incubator/livy/$LIVY_VERSION-incubating/apa
&& unzip /home/admin/apache-livy-$LIVY_VERSION-incubating-bin.zip \
&& rm -f /home/admin/apache-livy-$LIVY_VERSION-incubating-bin.zip
-# copy kylin source code
-COPY ./kylin /home/admin/kylin_sourcecode
-
-RUN sed -i -e 's#<!--<version>3.4.1</version>-->#<version>3.4.1</version>#g' /home/admin/kylin_sourcecode/pom.xml \
- && echo "kylin.engine.spark-conf.spark.executor.memory=1G" >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo "kylin.engine.spark-conf-mergedict.spark.executor.memory=1.5G" >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo "kylin.engine.livy-conf.livy-url=http://127.0.0.1:8998" >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.livy-conf.livy-key.file=hdfs://localhost:9000/kylin/livy/kylin-job-$KYLIN_VERSION.jar >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.livy-conf.livy-arr.jars=hdfs://localhost:9000/kylin/livy/hbase-client-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-common-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-hadoop2-compat-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/hbase-server-$HBASE_VERSION.jar,hdfs://localhost:9000/kylin/livy/htrace-core-*-incubating.jar,hdfs://localhost:9000/kylin/livy/metrics- [...]
- && echo kylin.source.hive.quote-enabled=false >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.spark-conf.spark.eventLog.dir=hdfs://localhost:9000/kylin/spark-history >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs://localhost:9000/kylin/spark-history >> /home/admin/kylin_sourcecode/build/conf/kylin.properties \
- && echo kylin.source.hive.redistribute-flat-table=false >> /home/admin/kylin_sourcecode/build/conf/kylin.properties
-
-COPY ./entrypoint.sh /home/admin/entrypoint.sh
-RUN chmod u+x /home/admin/entrypoint.sh
-
-ENTRYPOINT ["/home/admin/entrypoint.sh"]
diff --git a/docker/README.md b/docker/README.md
new file mode 100644
index 0000000..7959939
--- /dev/null
+++ b/docker/README.md
@@ -0,0 +1,145 @@
+
+In order to allow users to easily try Kylin, and to facilitate developers to verify and debug after modifying the source code. We provide the all-in-one Kylin docker image. In this image, each service that Kylin relies on is properly installed and deployed, including:
+
+- Jdk 1.8
+- Hadoop 2.7.0
+- Hive 1.2.1
+- Hbase 1.1.2
+- Spark 2.3.1
+- Zookeeper 3.4.6
+- Kafka 1.1.1
+- MySQL 5.1.73
+- Maven 3.6.1
+
+## Quickly try Kylin with pre-built images
+
+We have pushed the Kylin images to the [docker hub](https://hub.docker.com/r/apachekylin/apache-kylin-standalone). You do not need to build the image locally, just pull the image from remote (you can browse docker hub to check the available versions):
+
+```
+docker pull apachekylin/apache-kylin-standalone:3.0.1
+```
+
+After the pull is successful, execute "sh run_container.sh" or the following command to start the container:
+
+```
+docker run -d \
+-m 8G \
+-p 7070:7070 \
+-p 8088:8088 \
+-p 50070:50070 \
+-p 8032:8032 \
+-p 8042:8042 \
+-p 16010:16010 \
+apache-kylin-standalone
+```
+
+The following services are automatically started when the container starts:
+
+- NameNode, DataNode
+- ResourceManager, NodeManager
+- HBase
+- Kafka
+- Kylin
+
+and run automatically `$KYLIN_HOME/bin/sample.sh `, create a kylin_streaming_topic topic in Kafka and continue to send data to this topic. This is to let the users start the container and then experience the batch and streaming way to build the cube and query.
+
+After the container is started, we can enter the container through the `docker exec` command. Of course, since we have mapped the specified port in the container to the local port, we can open the pages of each service directly in the native browser, such as:
+
+- Kylin Web UI: [http://127.0.0.1:7070/kylin/login](http://127.0.0.1:7070/kylin/login)
+- HDFS NameNode Web UI: [http://127.0.0.1:50070](http://127.0.0.1:50070/)
+- YARN ResourceManager Web UI: [http://127.0.0.1:8088](http://127.0.0.1:8088/)
+- HBase Web UI: [http://127.0.0.1:16010](http://127.0.0.1:16010/)
+
+In the container, the relevant environment variables are as follows:
+
+```
+JAVA_HOME=/home/admin/jdk1.8.0_141
+HADOOP_HOME=/home/admin/hadoop-2.7.0
+KAFKA_HOME=/home/admin/kafka_2.11-1.1.1
+SPARK_HOME=/home/admin/spark-2.3.1-bin-hadoop2.6
+HBASE_HOME=/home/admin/hbase-1.1.2
+HIVE_HOME=/home/admin/apache-hive-1.2.1-bin
+```
+
+After about 1 to 2 minutes, all the services should be started. At the Kylin login page (http://127.0.0.1:7070/kylin), enter ADMIN:KYLIN to login, select the "learn_kylin" project. In the "Model" tab, you should be able to see two sample cubes: "kylin_sales_cube" and "kylin_streaming_cube". If they don't appear, go to the "System" tab, and then click "Reload metadata", they should be loaded.
+
+In the "Model" tab, you can click "Build" to build the two sample cubes. After the cubes be built, try some queries in the "Insight" tab.
+
+If you want to login into the Docker container, run "docker ps" to get the container id:
+
+```
+> docker ps
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+c15d10ff6bf1 apachekylin/apache-kylin-standalone:3.0.1 "/home/admin/entrypo…" 55 minutes ago Up 55 minutes 0.0.0.0:7070->7070/tcp, 0.0.0.0:8032->8032/tcp, 0.0.0.0:8042->8042/tcp, 0.0.0.0:8088->8088/tcp, 0.0.0.0:50070->50070/tcp, 0.0.0.0:16010->16010/tcp romantic_moser
+```
+
+Then run "docker -it <container id> bash" to login it with bash:
+
+```
+> docker exec -it c15d10ff6bf1 bash
+[root@c15d10ff6bf1 admin]# ls
+apache-hive-1.2.1-bin apache-maven-3.6.1 first_run hbase-1.1.2 kafka_2.11-1.1.1
+apache-kylin-3.0.0-alpha2-bin-hbase1x entrypoint.sh hadoop-2.7.0 jdk1.8.0_141 spark-2.3.1-bin-hadoop2.6
+```
+
+## Build Docker image in local
+
+You can build the docker image by yourself with the provided Dockerfile. Here we separate the scripts into several files:
+
+- Dockerfile_hadoop: build a Hadoop image with Hadoop/Hive/HBase/Spark/Kafka and other components installed;
+- Dockerfile: based on the Hadoop image, download Kylin from apache website and then start all services.
+- Dockerfile_dev: similar with "Dockerfile", instead of downloading the released version, it copies local built Kylin package to the image.
+
+Others:
+- conf/: the Hadoop/HBase/Hive/Maven configuration files for this docker; Will copy them into the image on 'docker build';
+- entrypoint.sh: the entrypoint script, which will start all the services;
+
+The build is very simple:
+
+```
+./build_image.sh
+```
+The script will build the Hadoop image first, and then build Kylin image based on it. Depends on the network bandwidth, the first time may take a while.
+
+## Customize the Docker image
+
+You can customize these scripts and Dockerfile to make your image.
+
+For example, if you made some code change in Kylin, you can make a new binary package in local with:
+
+```
+./build/scripts/package.sh
+```
+
+The new package is generated in "dist/" folder; Copy it to the "docker" folder:
+
+```
+cp ./dist/apache-kylin-3.1.0-SNAPSHOT-bin.tar.gz ./docker
+```
+
+Use the "Dockerfile_dev" file to build:
+
+```
+docker build -f Dockerfile_dev -t apache-kylin-standalone:test .
+
+```
+
+## Build Docker image for your Hadoop environment
+
+You can run Kylin in Docker with your Hadoop cluster. In this case, you need to build a customized image:
+
+- Use the same version Hadoop components as your cluster;
+- Use your cluster's configuration files (copy to conf/);
+- Modify the "entrypoint.sh", only start Kylin, no need to start other Hadoop services;
+
+
+## Container resource recommendation
+
+In order to allow Kylin to build the cube smoothly, the memory resource we configured for Yarn NodeManager is 6G, plus the memory occupied by each service, please ensure that the memory of the container is not less than 8G, so as to avoid errors due to insufficient memory.
+
+For the resource setting method for the container, please refer to:
+
+- Mac user: <https://docs.docker.com/docker-for-mac/#advanced>
+- Linux user: <https://docs.docker.com/config/containers/resource_constraints/#memory>
+
+---
diff --git a/docker/build_image.sh b/docker/build_image.sh
index cc214f0..19fbec5 100644
--- a/docker/build_image.sh
+++ b/docker/build_image.sh
@@ -17,24 +17,11 @@
# limitations under the License.
#
-echo "start build kylin image base on current source code"
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd ${DIR}
echo "build image in dir "${DIR}
-rm -rf ./kylin
-mkdir -p ./kylin
-echo "start copy kylin source code"
-
-for file in `ls ../`
-do
- if [ docker != $file ]
- then
- cp -r ../$file ./kylin/
- fi
-done
-
-echo "finish copy kylin source code"
-
-docker build -t apache-kylin-standalone .
+echo "start build Hadoop docker image"
+docker build -f Dockerfile_hadoop -t hadoop2.7-all-in-one .
+docker build -f Dockerfile -t apache-kylin-standalone .
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 97c0be8..edeefec 100644
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -61,6 +61,11 @@ $LIVY_HOME/bin/livy-server start
# prepare kafka topic and data
$KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic kylin_streaming_topic
+nohup $KYLIN_HOME/bin/kylin.sh org.apache.kylin.source.kafka.util.KafkaSampleProducer --topic kylin_streaming_topic --broker localhost:9092 < /dev/null 2>&1 > /tmp/kafka-sample.log &
+# create sample cube
+sh $KYLIN_HOME/bin/sample.sh
+# start kylin
+$KYLIN_HOME/bin/kylin.sh start
while :
do
diff --git a/docker/run_container.sh b/docker/run_container.sh
old mode 100644
new mode 100755
index 5213c55..8f03d75
--- a/docker/run_container.sh
+++ b/docker/run_container.sh
@@ -22,6 +22,6 @@ docker run -d \
-p 50070:50070 \
-p 8032:8032 \
-p 8042:8042 \
--p 60010:60010 \
+-p 16010:16010 \
-p 8998:8998 \
apache-kylin-standalone
\ No newline at end of file
diff --git a/docker/use-guide.md b/docker/use-guide.md
deleted file mode 100644
index fae12fc..0000000
--- a/docker/use-guide.md
+++ /dev/null
@@ -1,136 +0,0 @@
-
-In order to allow users to easily try Kylin, and to facilitate developers to verify and debug after modifying the source code. We provide Kylin's docker image. In this image, each service that Kylin relies on is properly installed and deployed, including:
-
-- Jdk 1.8
-- Hadoop 2.7.0
-- Hive 1.2.1
-- Hbase 1.1.2
-- Spark 2.3.1
-- Zookeeper 3.4.6
-- Kafka 1.1.1
-- Mysql
-- Maven 3.6.1
-
-## Quickly try Kylin
-
-We have pushed the Kylin image for the user to the docker hub. Users do not need to build the image locally, just execute the following command to pull the image from the docker hub:
-
-```
-docker pull apachekylin/apache-kylin-standalone:3.0.0-alpha2
-```
-
-After the pull is successful, execute the following command to start the container:
-
-```
-docker run -d \
--m 8G \
--p 7070:7070 \
--p 8088:8088 \
--p 50070:50070 \
--p 8032:8032 \
--p 8042:8042 \
--p 60010:60010 \
-apachekylin/apache-kylin-standalone:3.0.0-alpha2
-```
-
-The following services are automatically started when the container starts:
-
-- NameNode, DataNode
-- ResourceManager, NodeManager
-- HBase
-- Kafka
-- Kylin
-
-and run automatically `$KYLIN_HOME/bin/sample.sh `, create a kylin_streaming_topic topic in Kafka and continue to send data to this topic. This is to let the users start the container and then experience the batch and streaming way to build the cube and query.
-
-After the container is started, we can enter the container through the `docker exec` command. Of course, since we have mapped the specified port in the container to the local port, we can open the pages of each service directly in the native browser, such as:
-
-- Kylin Web UI: [http://127.0.0.1:7070/kylin/login](http://127.0.0.1:7070/kylin/login)
-- Hdfs NameNode Web UI: [http://127.0.0.1:50070](http://127.0.0.1:50070/)
-- Yarn ResourceManager Web UI: [http://127.0.0.1:8088](http://127.0.0.1:8088/)
-- HBase Web UI: [http://127.0.0.1:60010](http://127.0.0.1:60010/)
-
-In the container, the relevant environment variables are as follows:
-
-```
-JAVA_HOME=/home/admin/jdk1.8.0_141
-HADOOP_HOME=/home/admin/hadoop-2.7.0
-KAFKA_HOME=/home/admin/kafka_2.11-1.1.1
-SPARK_HOME=/home/admin/spark-2.3.1-bin-hadoop2.6
-HBASE_HOME=/home/admin/hbase-1.1.2
-HIVE_HOME=/home/admin/apache-hive-1.2.1-bin
-KYLIN_HOME=/home/admin/apache-kylin-3.0.0-alpha2-bin-hbase1x
-```
-
-## Build image to verify source code modifications
-
-The docker image can also be used when developers have modified the source code and want to package, deploy, and verify the source code. First, we go to the docker directory under the root directory of the source and execute the script below to build the image and copy the source into the image.:
-
-```
-#!/usr/bin/env bash
-
-echo "start build kylin image base on current source code"
-
-rm -rf ./kylin
-mkdir -p ./kylin
-
-echo "start copy kylin source code"
-
-for file in `ls ../../kylin/`
-do
- if [ docker != $file ]
- then
- cp -r ../../kylin/$file ./kylin/
- fi
-done
-
-echo "finish copy kylin source code"
-
-docker build -t apache-kylin-standalone .⏎
-```
-
-Due to need to download and deploy various binary packages over the network, the entire build process can last for tens of minutes, depending on the network.
-
-When the image build is complete, execute the following command to start the container:
-
-```
-docker run -d \
--m 8G \
--p 7070:7070 \
--p 8088:8088 \
--p 50070:50070 \
--p 8032:8032 \
--p 8042:8042 \
--p 60010:60010 \
-apache-kylin-standalone
-```
-
-When the container starts, execute the docker exec command to enter the container. The source code is stored in the container dir `/home/admin/kylin_sourcecode`, execute the following command to package the source code:
-
-```
-cd /home/admin/kylin_sourcecod
-build/script/package.sh
-```
-
-After the package is complete, an binary package ending in `.tar.gz` will be generated in the `/home/admin/kylin_sourcecode/dist` directory, such as `apache-kylin-3.0.0-alpha2-bin-hbase1x.tar.gz`. We can use this binary package to deploy and launch Kylin services such as:
-
-```
-cp /home/admin/kylin_sourcecode/dist/apache-kylin-3.0.0-alpha2-bin-hbase1x.tar.gz /home/admin
-tar -zxvf /home/admin/apache-kylin-3.0.0-alpha2-bin-hbase1x.tar.gz
-/home/admin/apache-kylin-3.0.0-alpha2-bin-hbase1x/kylin.sh start
-```
-
-We can also open pages for services such as Hdfs, Yarn, HBase, and Kylin in the browser of this machine.
-
-## Container resource recommendation
-
-In order to allow Kylin to build the cube smoothly, the memory resource we configured for Yarn NodeManager is 6G, plus the memory occupied by each service, please ensure that the memory of the container is not less than 8G, so as to avoid errors due to insufficient memory.
-
-For the resource setting method for the container, please refer to:
-
-- Mac user: <https://docs.docker.com/docker-for-mac/#advanced>
-- Linux user: <https://docs.docker.com/config/containers/resource_constraints/#memory>
-
----
-
-For old docker image, please check the github page [kylin-docker](https://github.com/Kyligence/kylin-docker/).
\ No newline at end of file