You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/08/05 13:56:30 UTC
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #132: [ISSUE-48][FEATURE] Add Uniffle Dockerfile
jerqi commented on code in PR #132:
URL: https://github.com/apache/incubator-uniffle/pull/132#discussion_r938839086
##########
deploy/kubernetes/docker/build.sh:
##########
@@ -0,0 +1,78 @@
+#!/bin/bash
Review Comment:
We need an Apache license.
##########
deploy/kubernetes/docker/build.sh:
##########
@@ -0,0 +1,78 @@
+#!/bin/bash
+
+function exit_with_usage() {
+ set +x
+ echo "./build.sh - Tool for building docker images of Remote Shuffle Service"
+ echo ""
+ echo "Usage:"
+ echo "+------------------------------------------------------------------------------------------------------+"
+ echo "| ./build.sh [--hadoop-version <hadoop version>] [--registry <registry url>] |"
+ echo "+------------------------------------------------------------------------------------------------------+"
+ exit 1
+}
+
+UNKNOWN_REGISTRY="UNKNOWN_REGISTRY"
+REGISTRY=$UNKNOWN_REGISTRY
+
+HADOOP_VERSION=3.2.3
+while (( "$#" )); do
+ case $1 in
+ --registry)
+ REGISTRY="$2"
+ shift
+ ;;
+ --hadoop-version)
+ HADOOP_VERSION="$2"
+ shift
+ ;;
+ --help)
+ exit_with_usage
+ ;;
+ --*)
+ echo "Error: $1 is not supported"
+ exit_with_usage
+ ;;
+ -*)
+ break
+ ;;
+ *)
+ echo "Error: $1 is not supported"
+ exit_with_usage
+ ;;
+ esac
+ shift
+done
+
+if [ "$REGISTRY" == $UNKNOWN_REGISTRY ]; \
+ then echo "please set registry url"; exit; \
+fi
+
+HADOOP_FILE=hadoop-${HADOOP_VERSION}.tar.gz
+HADOOP_URL=https://archive.apache.org/dist/hadoop/core/hadoop-${HADOOP_VERSION}/${HADOOP_FILE}
+echo "HADOOP_URL is $HADOOP_URL"
+if [ ! -e "$HADOOP_FILE" ]; \
+ then wget "$HADOOP_URL"; \
+ else echo "${HADOOP_FILE} has been downloaded"; \
+fi
+
+RSS_DIR=../../..
+cd $RSS_DIR || exit
+RSS_VERSION=$(mvn help:evaluate -Dexpression=project.version 2>/dev/null | grep -v "INFO" | grep -v "WARNING" | tail -n 1)
+RSS_FILE=rss-${RSS_VERSION}.tgz
+if [ ! -e "$RSS_FILE" ]; \
+ then sh build_distribution.sh \
+ else echo "$RSS_FILE has been built"; \
+fi
+cd "$OLDPWD" || exit
+cp "$RSS_DIR/$RSS_FILE" .
+
+IMAGE=$REGISTRY/rss-server:$RSS_VERSION
+
+echo "building image: $IMAGE"
+docker build -t "$IMAGE" \
+ --build-arg RSS_VERSION="$RSS_VERSION" \
+ --build-arg HADOOP_VERSION="$HADOOP_VERSION" \
+ -f Dockerfile --no-cache .
+
+echo "pushing image: $IMAGE"
+docker push "$IMAGE"
Review Comment:
We need a new line at the end of the file.
##########
deploy/kubernetes/docker/start.sh:
##########
@@ -0,0 +1,40 @@
+#!/bin/bash
Review Comment:
We need an Apache license.
##########
.gitignore:
##########
@@ -21,4 +21,5 @@ metastore_db/
derby.log
dependency-reduced-pom.xml
rss-*.tgz
-
+hadoop-*.tar.gz
Review Comment:
Could we remove this two files?
##########
deploy/kubernetes/docker/build.sh:
##########
@@ -0,0 +1,78 @@
+#!/bin/bash
+
+function exit_with_usage() {
+ set +x
+ echo "./build.sh - Tool for building docker images of Remote Shuffle Service"
+ echo ""
+ echo "Usage:"
+ echo "+------------------------------------------------------------------------------------------------------+"
+ echo "| ./build.sh [--hadoop-version <hadoop version>] [--registry <registry url>] |"
+ echo "+------------------------------------------------------------------------------------------------------+"
+ exit 1
+}
+
+UNKNOWN_REGISTRY="UNKNOWN_REGISTRY"
+REGISTRY=$UNKNOWN_REGISTRY
+
+HADOOP_VERSION=3.2.3
Review Comment:
Should we use Hadoop version 2.8.5? Because we use the version in other modules of Uniffle, we should keep consistent with them.
##########
deploy/kubernetes/docker/hadoopconfig/.gitkeep:
##########
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Review Comment:
Could we remove this file? Why do we add this file?
##########
deploy/kubernetes/docker/Dockerfile:
##########
@@ -0,0 +1,29 @@
+FROM openjdk:8-jdk-slim
+
+ARG HADOOP_VERSION
+ARG RSS_VERSION
+
+RUN apt-get update && apt-get install -y zlib1g zlib1g-dev lzop lsof netcat dnsutils less procps iputils-ping \
+ && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+RUN useradd -ms /bin/bash rssadmin
+RUN mkdir -p /data/rssadmin/
+RUN chown -R rssadmin:rssadmin /data
+USER rssadmin
+
+COPY rss-${RSS_VERSION}.tgz /data/rssadmin
+RUN tar -xvf /data/rssadmin/rss-${RSS_VERSION}.tgz -C /data/rssadmin
+RUN mv /data/rssadmin/rss-${RSS_VERSION} /data/rssadmin/rss
+RUN rm -rf /data/rssadmin/rss-${RSS_VERSION}.tgz
+
+COPY start.sh /data/rssadmin/rss/bin
+
+COPY hadoop-${HADOOP_VERSION}.tar.gz /data/rssadmin
+RUN tar -zxvf /data/rssadmin/hadoop-${HADOOP_VERSION}.tar.gz -C /data/rssadmin
+RUN mv /data/rssadmin/hadoop-${HADOOP_VERSION} /data/rssadmin/hadoop
+RUN rm -rf /data/rssadmin/hadoop-${HADOOP_VERSION}.tar.gz
+COPY hadoopconfig/ /data/rssadmin/hadoop/etc/hadoop
+
+ENV RSS_VERSION ${RSS_VERSION}
+ENV HADOOP_VERSION ${HADOOP_VERSION}
+CMD bash /data/rssadmin/rss/bin/start.sh
Review Comment:
We need a new line at the end of the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org