You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by mo...@apache.org on 2021/03/11 16:21:30 UTC
[zeppelin] branch master updated: [ZEPPELIN-5167] Set HOME env
variable to in Dockerfile, so Helium can install package successfully
This is an automated email from the ASF dual-hosted git repository.
moon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new f1db722 [ZEPPELIN-5167] Set HOME env variable to in Dockerfile, so Helium can install package successfully
f1db722 is described below
commit f1db72297b373c856dcb149c2c27dcc3a845959c
Author: Lee moon soo <le...@gmail.com>
AuthorDate: Tue Mar 9 08:45:53 2021 -0800
[ZEPPELIN-5167] Set HOME env variable to in Dockerfile, so Helium can install package successfully
### What is this PR for?
Without HOME env variable set in Docker image, installing a helium visualization package fails with error.
```
INFO [2021-03-04 06:40:51,674] ({qtp418304857-31} YarnTaskExecutor.java[execute]:54) - Running 'yarn config set cache-folder /opt/zeppelin/local-repo/helium-bundle/yarn-cache --registry=https://registry.npmjs.org/' in /opt/zeppelin/local-repo/helium-bundle
ERROR [2021-03-04 06:40:51,793] ({Thread-24} InputStreamHandler.java[log]:60) - /opt/zeppelin/local-repo/helium-bundle/node/yarn/dist/node_modules/mkdirp/index.js:90
ERROR [2021-03-04 06:40:51,795] ({Thread-24} InputStreamHandler.java[log]:60) - throw err0;
ERROR [2021-03-04 06:40:51,795] ({Thread-24} InputStreamHandler.java[log]:60) - ^
ERROR [2021-03-04 06:40:51,795] ({Thread-24} InputStreamHandler.java[log]:60) -
ERROR [2021-03-04 06:40:51,796] ({Thread-24} InputStreamHandler.java[log]:60) - Error: EACCES: permission denied, mkdir '/.cache'
ERROR [2021-03-04 06:40:51,796] ({Thread-24} InputStreamHandler.java[log]:60) - at Error (native)
ERROR [2021-03-04 06:40:51,796] ({Thread-24} InputStreamHandler.java[log]:60) - at Object.fs.mkdirSync (fs.js:922:18)
ERROR [2021-03-04 06:40:51,797] ({Thread-24} InputStreamHandler.java[log]:60) - at sync (/opt/zeppelin/local-repo/helium-bundle/node/yarn/dist/node_modules/mkdirp/index.js:71:13)
ERROR [2021-03-04 06:40:51,797] ({Thread-24} InputStreamHandler.java[log]:60) - at Function.sync (/opt/zeppelin/local-repo/helium-bundle/node/yarn/dist/node_modules/mkdirp/index.js:77:24)
ERROR [2021-03-04 06:40:51,797] ({Thread-24} InputStreamHandler.java[log]:60) - at Object.<anonymous> (/opt/zeppelin/local-repo/helium-bundle/node/yarn/dist/bin/yarn.js:30:8)
ERROR [2021-03-04 06:40:51,797] ({Thread-24} InputStreamHandler.java[log]:60) - at Module._compile (module.js:570:32)
ERROR [2021-03-04 06:40:51,798] ({Thread-24} InputStreamHandler.java[log]:60) - at Object.Module._extensions..js (module.js:579:10)
ERROR [2021-03-04 06:40:51,799] ({Thread-24} InputStreamHandler.java[log]:60) - at Module.load (module.js:487:32)
ERROR [2021-03-04 06:40:51,801] ({Thread-24} InputStreamHandler.java[log]:60) - at tryModuleLoad (module.js:446:12)
ERROR [2021-03-04 06:40:51,801] ({Thread-24} InputStreamHandler.java[log]:60) - at Function.Module._load (module.js:438:3)
```
This error can be solved by manually adding HOME env variable when starting docker container, like
```
docker run -p8080:8080 --rm -e HOME=/opt/zeppelin -e ZEPPELIN_HELIUM_REGISTRY=helium,https://s3.amazonaws.com/helium-package/helium.json apache/zeppelin:0.9.0
```
However, the error message is not very intuitive and it may take lots of time to find a solution.
So, I think it make sense to set the HOME env variable in Dockerfile.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-5167
### How should this be tested?
Reproduce the problem by running Zeppelin from the latest docker image without the HOME env variable.
And try to install a visualization package from Helium menu
```
docker run -p8080:8080 --rm -e ZEPPELIN_HELIUM_REGISTRY=helium,https://s3.amazonaws.com/helium-package/helium.json apache/zeppelin:0.9.0
```
And then test with HOME environment variable, and verify visualization package is installed without error.
```
docker run -p8080:8080 --rm -e HOME=/opt/zeppelin -e ZEPPELIN_HELIUM_REGISTRY=helium,https://s3.amazonaws.com/helium-package/helium.json apache/zeppelin:0.9.0
```
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Lee moon soo <le...@gmail.com>
Closes #4070 from Leemoonsoo/ZEPPELIN-5167 and squashes the following commits:
bc9cc8dcb [Lee moon soo] add HOME env var to zeppelin server Dockerfile
8a296cf8d [Lee moon soo] Z_HOME -> ZEPPELIN_HOME
d9a9ed245 [Lee moon soo] set HOME env variable
---
bin/interpreter.sh | 2 +-
bin/zeppelin.sh | 2 +-
docs/quickstart/kubernetes.md | 2 +-
scripts/docker/zeppelin-interpreter/Dockerfile | 20 +++++++-------
scripts/docker/zeppelin-server/Dockerfile | 37 +++++++++++++-------------
scripts/docker/zeppelin/bin/Dockerfile | 21 ++++++++-------
6 files changed, 43 insertions(+), 41 deletions(-)
diff --git a/bin/interpreter.sh b/bin/interpreter.sh
index 1280ac0..bccebe6 100755
--- a/bin/interpreter.sh
+++ b/bin/interpreter.sh
@@ -39,7 +39,7 @@ if [ -f /proc/self/cgroup ] && [ -n "$(command -v getent)" ]; then
# If there is no passwd entry for the container UID, attempt to create one
if [ -z "$uidentry" ] ; then
if [ -w /etc/passwd ] ; then
- echo "zeppelin:x:$myuid:$mygid:anonymous uid:$Z_HOME:/bin/false" >> /etc/passwd
+ echo "zeppelin:x:$myuid:$mygid:anonymous uid:$ZEPPELIN_HOME:/bin/false" >> /etc/passwd
else
echo "Container ENTRYPOINT failed to add passwd entry for anonymous UID"
fi
diff --git a/bin/zeppelin.sh b/bin/zeppelin.sh
index d8cf00b..10f50c2 100755
--- a/bin/zeppelin.sh
+++ b/bin/zeppelin.sh
@@ -34,7 +34,7 @@ if [ -f /proc/self/cgroup ] && [ -n "$(command -v getent)" ]; then
# If there is no passwd entry for the container UID, attempt to create one
if [ -z "$uidentry" ] ; then
if [ -w /etc/passwd ] ; then
- echo "zeppelin:x:$myuid:$mygid:anonymous uid:$Z_HOME:/bin/false" >> /etc/passwd
+ echo "zeppelin:x:$myuid:$mygid:anonymous uid:$ZEPPELIN_HOME:/bin/false" >> /etc/passwd
else
echo "Container ENTRYPOINT failed to add passwd entry for anonymous UID"
fi
diff --git a/docs/quickstart/kubernetes.md b/docs/quickstart/kubernetes.md
index 8df27a4..da14665 100644
--- a/docs/quickstart/kubernetes.md
+++ b/docs/quickstart/kubernetes.md
@@ -143,7 +143,7 @@ $ mv zeppelin-distribution/target/zeppelin-*.tar.gz scripts/docker/zeppelin/bin/
# wget -O /tmp/zeppelin-${Z_VERSION}-bin-all.tgz http://archive.apache.org/dist/zeppelin/zeppelin-${Z_VERSION}/zeppelin-${Z_VERSION}-bin-all.tgz && \
# tar -zxvf /tmp/zeppelin-${Z_VERSION}-bin-all.tgz && \
# rm -rf /tmp/zeppelin-${Z_VERSION}-bin-all.tgz && \
-# mv /zeppelin-${Z_VERSION}-bin-all ${Z_HOME}
+# mv /zeppelin-${Z_VERSION}-bin-all ${ZEPPELIN_HOME}
# Add following lines right after the commented line above
ADD zeppelin-${Z_VERSION}.tar.gz /
diff --git a/scripts/docker/zeppelin-interpreter/Dockerfile b/scripts/docker/zeppelin-interpreter/Dockerfile
index 0c9d956..0ee0e19 100644
--- a/scripts/docker/zeppelin-interpreter/Dockerfile
+++ b/scripts/docker/zeppelin-interpreter/Dockerfile
@@ -22,7 +22,7 @@ LABEL maintainer="Apache Software Foundation <de...@zeppelin.apache.org>"
ARG version="0.9.0-SNAPSHOT"
ENV VERSION="${version}" \
- Z_HOME="/opt/zeppelin"
+ ZEPPELIN_HOME="/opt/zeppelin"
RUN set -ex && \
apt-get -y update && \
@@ -32,17 +32,17 @@ RUN set -ex && \
apt-get autoclean && \
apt-get clean
-COPY --from=zeppelin-distribution /opt/zeppelin/bin ${Z_HOME}/bin
-COPY log4j.properties ${Z_HOME}/conf/
-COPY log4j_yarn_cluster.properties ${Z_HOME}/conf/
+COPY --from=zeppelin-distribution /opt/zeppelin/bin ${ZEPPELIN_HOME}/bin
+COPY log4j.properties ${ZEPPELIN_HOME}/conf/
+COPY log4j_yarn_cluster.properties ${ZEPPELIN_HOME}/conf/
# Decide:
## 1) Build a huge image with all interpreters (default)
-COPY --from=zeppelin-distribution /opt/zeppelin/interpreter ${Z_HOME}/interpreter
+COPY --from=zeppelin-distribution /opt/zeppelin/interpreter ${ZEPPELIN_HOME}/interpreter
## 2) Build an image with only some interpreters
#### Copy interpreter-shaded JAR, needed for all interpreters
-### COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar ${Z_HOME}/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar
+### COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar ${ZEPPELIN_HOME}/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar
#### Copy specific interpreters, replace "${interpreter_name}" with your interpreter. Of course you can repeat the line with defferent interpreter
-### COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/${interpreter_name} ${Z_HOME}/interpreter/${interpreter_name}
+### COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/${interpreter_name} ${ZEPPELIN_HOME}/interpreter/${interpreter_name}
# Decide: Install conda to manage python and R packages. Maybe adjust the packages in pip_packages.txt or conda_packages.txt
@@ -73,12 +73,12 @@ RUN set -ex && \
# chmod -R ug+rwX /opt/conda
ENV PATH /opt/conda/bin:$PATH
-RUN mkdir -p "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/local-repo" && \
+RUN mkdir -p "${ZEPPELIN_HOME}/logs" "${ZEPPELIN_HOME}/run" "${ZEPPELIN_HOME}/local-repo" && \
# Allow process to edit /etc/passwd, to create a user entry for zeppelin
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
# Give access to some specific folders
- chmod -R 775 "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/local-repo"
+ chmod -R 775 "${ZEPPELIN_HOME}/logs" "${ZEPPELIN_HOME}/run" "${ZEPPELIN_HOME}/local-repo"
USER 1000
ENTRYPOINT [ "/usr/bin/tini", "--" ]
-WORKDIR ${Z_HOME}
+WORKDIR ${ZEPPELIN_HOME}
diff --git a/scripts/docker/zeppelin-server/Dockerfile b/scripts/docker/zeppelin-server/Dockerfile
index ec470e4..9faf92c 100644
--- a/scripts/docker/zeppelin-server/Dockerfile
+++ b/scripts/docker/zeppelin-server/Dockerfile
@@ -41,43 +41,44 @@ ENV LANG=en_US.UTF-8 \
LC_ALL=en_US.UTF-8 \
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
VERSION="${version}" \
- Z_HOME="/opt/zeppelin" \
+ HOME="/opt/zeppelin" \
+ ZEPPELIN_HOME="/opt/zeppelin" \
ZEPPELIN_ADDR="0.0.0.0" \
ZEPPELIN_WAR_TEMPDIR="/tmp/webapps"
# Copy Zeppelin related files
-COPY --from=zeppelin-distribution /opt/zeppelin/zeppelin-web-${VERSION}.war ${Z_HOME}/
-COPY --from=zeppelin-distribution /opt/zeppelin/zeppelin-web-angular-${VERSION}.war ${Z_HOME}/
-COPY --from=zeppelin-distribution /opt/zeppelin/conf ${Z_HOME}/conf
-COPY --from=zeppelin-distribution /opt/zeppelin/bin ${Z_HOME}/bin
-COPY --from=zeppelin-distribution /opt/zeppelin/lib ${Z_HOME}/lib
-COPY --from=zeppelin-distribution /opt/zeppelin/plugins ${Z_HOME}/plugins
-COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar ${Z_HOME}/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar
+COPY --from=zeppelin-distribution /opt/zeppelin/zeppelin-web-${VERSION}.war ${ZEPPELIN_HOME}/
+COPY --from=zeppelin-distribution /opt/zeppelin/zeppelin-web-angular-${VERSION}.war ${ZEPPELIN_HOME}/
+COPY --from=zeppelin-distribution /opt/zeppelin/conf ${ZEPPELIN_HOME}/conf
+COPY --from=zeppelin-distribution /opt/zeppelin/bin ${ZEPPELIN_HOME}/bin
+COPY --from=zeppelin-distribution /opt/zeppelin/lib ${ZEPPELIN_HOME}/lib
+COPY --from=zeppelin-distribution /opt/zeppelin/plugins ${ZEPPELIN_HOME}/plugins
+COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar ${ZEPPELIN_HOME}/interpreter/zeppelin-interpreter-shaded-${VERSION}.jar
# copy example notebooks
-COPY --from=zeppelin-distribution /opt/zeppelin/notebook ${Z_HOME}/notebook
+COPY --from=zeppelin-distribution /opt/zeppelin/notebook ${ZEPPELIN_HOME}/notebook
# copy k8s files
-COPY --from=zeppelin-distribution /opt/zeppelin/k8s ${Z_HOME}/k8s
+COPY --from=zeppelin-distribution /opt/zeppelin/k8s ${ZEPPELIN_HOME}/k8s
# Decide
## 1) Copy and activate all interpreters (default)
-COPY --from=interpreter-settings /opt/zeppelin/interpreter ${Z_HOME}/interpreter
+COPY --from=interpreter-settings /opt/zeppelin/interpreter ${ZEPPELIN_HOME}/interpreter
## 2) Copy and activate only a specific set of interpreter
-# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/spark/interpreter-setting.json ${Z_HOME}/interpreter/spark/interpreter-setting.json
-# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/jdbc/interpreter-setting.json ${Z_HOME}/interpreter/jdbc/interpreter-setting.json
-# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/md/interpreter-setting.json ${Z_HOME}/interpreter/md/interpreter-setting.json
+# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/spark/interpreter-setting.json ${ZEPPELIN_HOME}/interpreter/spark/interpreter-setting.json
+# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/jdbc/interpreter-setting.json ${ZEPPELIN_HOME}/interpreter/jdbc/interpreter-setting.json
+# COPY --from=zeppelin-distribution /opt/zeppelin/interpreter/md/interpreter-setting.json ${ZEPPELIN_HOME}/interpreter/md/interpreter-setting.json
-COPY log4j.properties ${Z_HOME}/conf/
+COPY log4j.properties ${ZEPPELIN_HOME}/conf/
-RUN mkdir -p "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/notebook" "${Z_HOME}/local-repo" && \
+RUN mkdir -p "${ZEPPELIN_HOME}/logs" "${ZEPPELIN_HOME}/run" "${ZEPPELIN_HOME}/notebook" "${ZEPPELIN_HOME}/local-repo" && \
# Allow process to edit /etc/passwd, to create a user entry for zeppelin
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
# Give access to some specific folders
- chmod -R 775 "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/conf" "${Z_HOME}/notebook" "${Z_HOME}/local-repo"
+ chmod -R 775 "${ZEPPELIN_HOME}/logs" "${ZEPPELIN_HOME}/run" "${ZEPPELIN_HOME}/conf" "${ZEPPELIN_HOME}/notebook" "${ZEPPELIN_HOME}/local-repo"
USER 1000
EXPOSE 8080
ENTRYPOINT [ "/usr/bin/tini", "--" ]
-WORKDIR ${Z_HOME}
+WORKDIR ${ZEPPELIN_HOME}
CMD ["bin/zeppelin.sh"]
diff --git a/scripts/docker/zeppelin/bin/Dockerfile b/scripts/docker/zeppelin/bin/Dockerfile
index 30edc6d..c8171ad 100644
--- a/scripts/docker/zeppelin/bin/Dockerfile
+++ b/scripts/docker/zeppelin/bin/Dockerfile
@@ -20,7 +20,8 @@ LABEL maintainer="Apache Software Foundation <de...@zeppelin.apache.org>"
ENV Z_VERSION="0.9.0"
ENV LOG_TAG="[ZEPPELIN_${Z_VERSION}]:" \
- Z_HOME="/opt/zeppelin" \
+ ZEPPELIN_HOME="/opt/zeppelin" \
+ HOME="/opt/zeppelin" \
LANG=en_US.UTF-8 \
LC_ALL=en_US.UTF-8 \
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
@@ -61,26 +62,26 @@ RUN set -ex && \
ENV PATH /opt/conda/bin:$PATH
RUN echo "$LOG_TAG Download Zeppelin binary" && \
- mkdir -p ${Z_HOME} && \
+ mkdir -p ${ZEPPELIN_HOME} && \
wget -nv -O /tmp/zeppelin-${Z_VERSION}-bin-all.tgz https://archive.apache.org/dist/zeppelin/zeppelin-${Z_VERSION}/zeppelin-${Z_VERSION}-bin-all.tgz && \
- tar --strip-components=1 -zxvf /tmp/zeppelin-${Z_VERSION}-bin-all.tgz -C ${Z_HOME} && \
+ tar --strip-components=1 -zxvf /tmp/zeppelin-${Z_VERSION}-bin-all.tgz -C ${ZEPPELIN_HOME} && \
rm -f /tmp/zeppelin-${Z_VERSION}-bin-all.tgz && \
- chown -R root:root ${Z_HOME} && \
- mkdir -p ${Z_HOME}/logs ${Z_HOME}/run ${Z_HOME}/webapps && \
+ chown -R root:root ${ZEPPELIN_HOME} && \
+ mkdir -p ${ZEPPELIN_HOME}/logs ${ZEPPELIN_HOME}/run ${ZEPPELIN_HOME}/webapps && \
# Allow process to edit /etc/passwd, to create a user entry for zeppelin
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
# Give access to some specific folders
- chmod -R 775 "${Z_HOME}/logs" "${Z_HOME}/run" "${Z_HOME}/notebook" "${Z_HOME}/conf" && \
+ chmod -R 775 "${ZEPPELIN_HOME}/logs" "${ZEPPELIN_HOME}/run" "${ZEPPELIN_HOME}/notebook" "${ZEPPELIN_HOME}/conf" && \
# Allow process to create new folders (e.g. webapps)
- chmod 775 ${Z_HOME}
+ chmod 775 ${ZEPPELIN_HOME}
-COPY log4j.properties ${Z_HOME}/conf/
-COPY log4j_docker.properties ${Z_HOME}/conf/
+COPY log4j.properties ${ZEPPELIN_HOME}/conf/
+COPY log4j_docker.properties ${ZEPPELIN_HOME}/conf/
USER 1000
EXPOSE 8080
ENTRYPOINT [ "/usr/bin/tini", "--" ]
-WORKDIR ${Z_HOME}
+WORKDIR ${ZEPPELIN_HOME}
CMD ["bin/zeppelin.sh"]