You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by gr...@apache.org on 2020/03/09 19:52:34 UTC
[kudu] branch master updated (fb0411f -> ef8696a)
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from fb0411f [tserver] report the newly bootstrapped tablet after OpenTablet completes
new e18f32f [docker] Fix docker build
new c1b7d76 [docker] Reduce docker image size
new ef8696a [docker] Use the buildkit when building images
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docker/Dockerfile | 37 ++++++++++++++++++++++---------------
docker/bootstrap-dev-env.sh | 28 ++++++++++++++++++----------
docker/bootstrap-python-env.sh | 2 +-
docker/bootstrap-runtime-env.sh | 2 --
docker/docker-build.sh | 9 ++++++++-
5 files changed, 49 insertions(+), 29 deletions(-)
[kudu] 03/03: [docker] Use the buildkit when building images
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit ef8696ac436776cf52181e9d94afb3c9608c9efa
Author: Grant Henke <gr...@apache.org>
AuthorDate: Mon Mar 9 09:57:45 2020 -0500
[docker] Use the buildkit when building images
The docker buildkit is an opt in build engine that is faster and offers
more advanced image building features.
https://docs.docker.com/develop/develop-images/build_enhancements/
This patch sets `DOCKER_BUILDKIT=1` in the `docker-build.sh` file
to enable the buildkit. Follow on patches will use some of the
new features as well.
Change-Id: Ib1bff68d65d0d4039efccabf7e1288163aafee0c
Reviewed-on: http://gerrit.cloudera.org:8080/15390
Tested-by: Grant Henke <gr...@apache.org>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
docker/docker-build.sh | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/docker/docker-build.sh b/docker/docker-build.sh
index f7ada2c..0eb8f53 100755
--- a/docker/docker-build.sh
+++ b/docker/docker-build.sh
@@ -104,6 +104,13 @@ TAG_LATEST=${TAG_LATEST:=1}
TAG_HASH=${TAG_HASH:=0}
DOCKER_CACHE_FROM=${DOCKER_CACHE_FROM:=""}
+# Enabled the docker buildkit so we can use advanced features
+# like skipping unused stages and mounting scripts that don't
+# need to remain in the image along with an improvement on
+# performance, storage management, feature functionality, and security.
+# https://docs.docker.com/develop/develop-images/build_enhancements/
+export DOCKER_BUILDKIT=1
+
VERSION=$(cat "$ROOT/version.txt")
VCS_REF=$(git rev-parse --short HEAD || echo "")
[kudu] 02/03: [docker] Reduce docker image size
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit c1b7d76bbe91fa9d024358c2923051fb9df91a5c
Author: Grant Henke <gr...@apache.org>
AuthorDate: Mon Mar 9 09:50:37 2020 -0500
[docker] Reduce docker image size
This patch reduces the Docker `kudu` image size by ~60 MB
(295MB -> 235MB).
It does this by moving the chown calls to before the files are
copied and using the `—chown` flag when copying files.
Before this change the chown command would effecively
double the size due to how aufs layers work.
I also removed some optional and unused packages from the
runtime and dev images.
Change-Id: I88238679912149d00ae3edc6a9d2d568f13aafca
Reviewed-on: http://gerrit.cloudera.org:8080/15389
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
docker/Dockerfile | 23 +++++++++++++----------
docker/bootstrap-dev-env.sh | 28 ++++++++++++++++++----------
docker/bootstrap-runtime-env.sh | 2 --
3 files changed, 31 insertions(+), 22 deletions(-)
diff --git a/docker/Dockerfile b/docker/Dockerfile
index fe4b3ac..185ff3d 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -280,24 +280,27 @@ ARG GID=1000
ARG INSTALL_DIR="/opt/kudu"
ARG DATA_DIR="/var/lib/kudu"
+# Setup the kudu user and create the neccessary directories.
+# We do this before copying any files othwerwise the image size is doubled by the chown change.
+RUN groupadd -g ${GID} kudu || groupmod -n kudu $(getent group ${GID} | cut -d: -f1) \
+ && useradd --shell /bin/bash -u ${UID} -g ${GID} -m kudu \
+ && mkdir -p ${INSTALL_DIR} && chown -R kudu:kudu ${INSTALL_DIR} \
+ && mkdir -p ${DATA_DIR} && chown -R kudu:kudu ${DATA_DIR}
+
# Copy the binaries.
WORKDIR $INSTALL_DIR/bin
-COPY --from=build /kudu/build/latest/bin/kudu ./
+COPY --chown=kudu:kudu --from=build /kudu/build/latest/bin/kudu ./
# Add to the binaries to the path.
ENV PATH=$INSTALL_DIR/bin/:$PATH
-WORKDIR $INSTALL_DIR
# Copy the web files.
-COPY --from=build /kudu/www ./www
-COPY ./docker/kudu-entrypoint.sh /
+WORKDIR $INSTALL_DIR
+COPY --chown=kudu:kudu --from=build /kudu/www ./www
-# Setup the kudu user and create the neccessary directories.
-RUN groupadd -g ${GID} kudu || groupmod -n kudu $(getent group ${GID} | cut -d: -f1) \
- && useradd --shell /bin/bash -u ${UID} -g ${GID} -m kudu \
- && chown -R kudu:kudu ${INSTALL_DIR} \
- && mkdir -p ${DATA_DIR} && chown -R kudu:kudu ${DATA_DIR}
-USER kudu
+# Copy the entrypoint script.
+COPY --chown=kudu:kudu ./docker/kudu-entrypoint.sh /
+USER kudu
# Add the entrypoint.
ENTRYPOINT ["/kudu-entrypoint.sh"]
CMD ["help"]
diff --git a/docker/bootstrap-dev-env.sh b/docker/bootstrap-dev-env.sh
index 62754ba..9ea5aad 100755
--- a/docker/bootstrap-dev-env.sh
+++ b/docker/bootstrap-dev-env.sh
@@ -49,7 +49,6 @@ if [[ -f "/usr/bin/yum" ]]; then
krb5-workstation \
libtool \
make \
- nscd \
ntp \
openssl-devel \
patch \
@@ -62,12 +61,13 @@ if [[ -f "/usr/bin/yum" ]]; then
wget
# Install docs build libraries.
- yum install -y \
- doxygen \
- gem \
- graphviz \
- ruby-devel \
- zlib-devel
+ # Note: Uncomment to include in your dev images. These are excluded to reduce image size and build time.
+ # yum install -y \
+ # doxygen \
+ # gem \
+ # graphviz \
+ # ruby-devel \
+ # zlib-devel
# To build on a version older than 7.0, the Red Hat Developer Toolset
# must be installed (in order to have access to a C++11 capable compiler).
@@ -133,7 +133,6 @@ elif [[ -f "/usr/bin/apt-get" ]]; then
libtool \
lsb-release \
make \
- nscd \
ntp \
openssl \
patch \
@@ -141,9 +140,9 @@ elif [[ -f "/usr/bin/apt-get" ]]; then
python \
rsync \
unzip \
- vim-common
+ vim-common \
+ wget
- # Install docs build libraries.
apt-get install -y --no-install-recommends \
doxygen \
gem \
@@ -151,6 +150,15 @@ elif [[ -f "/usr/bin/apt-get" ]]; then
ruby-dev \
xsltproc \
zlib1g-dev
+ # Install docs build libraries.
+ # Note: Uncomment to include in your dev images. These are excluded to reduce image size and build time.
+ # apt-get install -y --no-install-recommends \
+ # doxygen \
+ # gem \
+ # graphviz \
+ # ruby-dev \
+ # xsltproc \
+ # zlib1g-dev
# Reduce the image size by cleaning up after the install.
apt-get clean
diff --git a/docker/bootstrap-runtime-env.sh b/docker/bootstrap-runtime-env.sh
index e8353c6..654ec4d 100755
--- a/docker/bootstrap-runtime-env.sh
+++ b/docker/bootstrap-runtime-env.sh
@@ -37,7 +37,6 @@ if [[ -f "/usr/bin/yum" ]]; then
cyrus-sasl-plain \
krb5-server \
krb5-workstation \
- nscd \
openssl
# Reduce the image size by cleaning up after the install.
@@ -60,7 +59,6 @@ elif [[ -f "/usr/bin/apt-get" ]]; then
libsasl2-2 \
libsasl2-modules \
libsasl2-modules-gssapi-mit \
- nscd \
openssl
# Reduce the image size by cleaning up after the install.
[kudu] 01/03: [docker] Fix docker build
Posted by gr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
granthenke pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e18f32f46a3c27e269346b47f48c30033f69382f
Author: Grant Henke <gr...@apache.org>
AuthorDate: Mon Mar 9 09:39:29 2020 -0500
[docker] Fix docker build
This patch fixes the docker build by moving the `java` directory
`COPY` above the C++ build. This is neccessary now that the
subprocessor depends on the Java build.
I also added a missing label, `vcs-ref`, back to the kudu-python
image and made the kudu-python image a default target given
we want to publish it with each release.
Change-Id: Ifbf32dc0c803b3a1ab44724d45dd1e64728d6133
Reviewed-on: http://gerrit.cloudera.org:8080/15388
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
docker/Dockerfile | 14 +++++++++-----
docker/bootstrap-python-env.sh | 2 +-
docker/docker-build.sh | 2 +-
3 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 444dd32..fe4b3ac 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -137,8 +137,8 @@ LABEL name="Apache Kudu Thirdparty" \
#
# ---- Build ----
# Builds an image that has the Kudu source code pre-built.
-# This is useful for generating a runtime image, but can also be a
-# useful base development image.
+# This is useful for generating a small runtime image,
+# but can also be a useful base development image.
#
FROM thirdparty AS build
@@ -159,9 +159,13 @@ COPY ./build-support build-support
COPY ./docs/support docs/support
COPY ./cmake_modules cmake_modules
COPY ./examples/cpp examples/cpp
-COPY ./java/kudu-hive/ java/kudu-hive/
COPY ./src src
COPY ./CMakeLists.txt ./version.txt ./
+
+# Copy the java build source.
+# Some parts of the C++ build depend on Java code.
+COPY ./java /kudu/java
+
# Build the c++ code.
WORKDIR /kudu/build/$BUILD_TYPE
# Ensure we don't rebuild thirdparty. Instead let docker handle caching.
@@ -183,8 +187,6 @@ RUN ../../build-support/enable_devtoolset.sh \
# Strip the client libraries to reduce the images size
&& if [[ "$STRIP" == "1" ]]; then find "/usr/local" -name "libkudu*" -type f -exec strip {} \;; fi
-# Copy the java build source.
-COPY ./java /kudu/java
# Build the java code.
WORKDIR /kudu/java
RUN ./gradlew jar
@@ -248,6 +250,7 @@ RUN pip install -r requirements.txt \
ARG DOCKERFILE
ARG MAINTAINER
ARG URL
+ARG VCS_REF
ARG VCS_TYPE
ARG VCS_URL
ARG VERSION
@@ -258,6 +261,7 @@ LABEL org.label-schema.name="Apache Kudu Python Client" \
org.label-schema.dockerfile=$DOCKERFILE \
org.label-schema.maintainer=$MAINTAINER \
org.label-schema.url=$URL \
+ org.label-schema.vcs-ref=$VCS_REF \
org.label-schema.vcs-type=$VCS_TYPE \
org.label-schema.vcs-url=$VCS_URL \
org.label-schema.version=$VERSION
diff --git a/docker/bootstrap-python-env.sh b/docker/bootstrap-python-env.sh
index ac7f14b..f1f1c29 100755
--- a/docker/bootstrap-python-env.sh
+++ b/docker/bootstrap-python-env.sh
@@ -18,7 +18,7 @@
# specific language governing permissions and limitations
# under the License.
#
-# This script handles bootstrapping a base OS for
+# This script handles bootstrapping python for
# the Apache Kudu base docker images.
#
##########################################################
diff --git a/docker/docker-build.sh b/docker/docker-build.sh
index 2e5c9c6..f7ada2c 100755
--- a/docker/docker-build.sh
+++ b/docker/docker-build.sh
@@ -97,7 +97,7 @@ ROOT=$(cd $(dirname "$BASH_SOURCE")/.. ; pwd)
# ubuntu:bionic
DEFAULT_OS="ubuntu:xenial"
BASES=${BASES:="$DEFAULT_OS"}
-TARGETS=${TARGETS:="kudu"}
+TARGETS=${TARGETS:="kudu,kudu-python"}
REPOSITORY=${REPOSITORY:="apache/kudu"}
PUBLISH=${PUBLISH:=0}
TAG_LATEST=${TAG_LATEST:=1}