You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/08/07 02:18:54 UTC
[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #132: [ISSUE-48][FEATURE] Add Uniffle Dockerfile
zuston commented on code in PR #132:
URL: https://github.com/apache/incubator-uniffle/pull/132#discussion_r939596513
##########
deploy/kubernetes/docker/Dockerfile:
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jdk-slim
+
+ARG HADOOP_VERSION
+ARG RSS_VERSION
+
+RUN apt-get update && apt-get install -y zlib1g zlib1g-dev lzop lsof netcat dnsutils less procps iputils-ping \
+ && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+RUN useradd -ms /bin/bash rssadmin
+RUN mkdir -p /data/rssadmin/
+RUN chown -R rssadmin:rssadmin /data
+USER rssadmin
+
+COPY rss-${RSS_VERSION}.tgz /data/rssadmin
+RUN tar -xvf /data/rssadmin/rss-${RSS_VERSION}.tgz -C /data/rssadmin
+RUN mv /data/rssadmin/rss-${RSS_VERSION} /data/rssadmin/rss
+RUN rm -rf /data/rssadmin/rss-${RSS_VERSION}.tgz
+
+COPY start.sh /data/rssadmin/rss/bin
+
+COPY hadoop-${HADOOP_VERSION}.tar.gz /data/rssadmin
+RUN tar -zxvf /data/rssadmin/hadoop-${HADOOP_VERSION}.tar.gz -C /data/rssadmin
+RUN mv /data/rssadmin/hadoop-${HADOOP_VERSION} /data/rssadmin/hadoop
+RUN rm -rf /data/rssadmin/hadoop-${HADOOP_VERSION}.tar.gz
+COPY hadoopconfig/ /data/rssadmin/hadoop/etc/hadoop
Review Comment:
What files in hadoopconfig folder? core-site/hdfs-site.xml?
##########
deploy/kubernetes/docker/Dockerfile:
##########
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jdk-slim
+
+ARG HADOOP_VERSION
+ARG RSS_VERSION
+
+RUN apt-get update && apt-get install -y zlib1g zlib1g-dev lzop lsof netcat dnsutils less procps iputils-ping \
Review Comment:
Do we need to consider kerberos? If yes, we need to setup kerberos client.
Maybe we shouldn't consider this in early version.
##########
deploy/kubernetes/docker/build.sh:
##########
@@ -0,0 +1,95 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+function exit_with_usage() {
+ set +x
+ echo "./build.sh - Tool for building docker images of Remote Shuffle Service"
+ echo ""
+ echo "Usage:"
+ echo "+------------------------------------------------------------------------------------------------------+"
+ echo "| ./build.sh [--hadoop-version <hadoop version>] [--registry <registry url>] |"
+ echo "+------------------------------------------------------------------------------------------------------+"
+ exit 1
+}
+
+UNKNOWN_REGISTRY="UNKNOWN_REGISTRY"
+REGISTRY=$UNKNOWN_REGISTRY
+
+HADOOP_VERSION=2.8.5
+while (( "$#" )); do
+ case $1 in
+ --registry)
+ REGISTRY="$2"
+ shift
+ ;;
+ --hadoop-version)
+ HADOOP_VERSION="$2"
+ shift
+ ;;
+ --help)
+ exit_with_usage
+ ;;
+ --*)
+ echo "Error: $1 is not supported"
+ exit_with_usage
+ ;;
+ -*)
+ break
+ ;;
+ *)
+ echo "Error: $1 is not supported"
+ exit_with_usage
+ ;;
+ esac
+ shift
+done
+
+if [ "$REGISTRY" == $UNKNOWN_REGISTRY ]; \
+ then echo "please set registry url"; exit; \
+fi
+
+HADOOP_FILE=hadoop-${HADOOP_VERSION}.tar.gz
+HADOOP_URL=https://archive.apache.org/dist/hadoop/core/hadoop-${HADOOP_VERSION}/${HADOOP_FILE}
+echo "HADOOP_URL is $HADOOP_URL"
+if [ ! -e "$HADOOP_FILE" ]; \
+ then wget "$HADOOP_URL"; \
+ else echo "${HADOOP_FILE} has been downloaded"; \
+fi
+
+RSS_DIR=../../..
+cd $RSS_DIR || exit
+RSS_VERSION=$(mvn help:evaluate -Dexpression=project.version 2>/dev/null | grep -v "INFO" | grep -v "WARNING" | tail -n 1)
+RSS_FILE=rss-${RSS_VERSION}.tgz
+if [ ! -e "$RSS_FILE" ]; \
+ then sh build_distribution.sh \
+ else echo "$RSS_FILE has been built"; \
+fi
+cd "$OLDPWD" || exit
+cp "$RSS_DIR/$RSS_FILE" .
+
+IMAGE=$REGISTRY/rss-server:$RSS_VERSION
+
+echo "building image: $IMAGE"
+docker build -t "$IMAGE" \
Review Comment:
Using LABEL to add more metadata, like build-time/author/branch/commit-id?
##########
deploy/kubernetes/docker/start.sh:
##########
@@ -0,0 +1,57 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+basedir='/data/rssadmin/rss'
+cd $basedir || exit
+
+coordinator_conf=$basedir'/conf/coordinator.conf'
+echo "coordinator_conf: $coordinator_conf"
+server_conf=$basedir'/conf/server.conf'
+echo "server_conf: $server_conf"
+
+
+if [ "$SERVICE_NAME" == "coordinator" ];then
+
+ bash ${basedir}/bin/start-coordinator.sh &
Review Comment:
Directly exit when executing startup script failed.
`bash ${basedir}/bin/start-coordinator.sh || exit 1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org