You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by li...@apache.org on 2020/05/18 15:45:11 UTC
[submarine] branch master updated: SUBMARINE-493. Integrating
jupyter notebook into Submarine
This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 9c2cdf8 SUBMARINE-493. Integrating jupyter notebook into Submarine
9c2cdf8 is described below
commit 9c2cdf83beaec7f3ad24066b3a83121ef0e0b370
Author: Ryan Lo <lo...@gmail.com>
AuthorDate: Mon May 18 09:52:35 2020 +0800
SUBMARINE-493. Integrating jupyter notebook into Submarine
### What is this PR for?
Deploy jupyter notebook with submarine-server
### What type of PR is it?
[Feature]
### Todos
### What is the Jira issue?
[SUBMARINE-493](https://issues.apache.org/jira/browse/SUBMARINE-493)
### How should this be tested?
[passed CI](https://travis-ci.org/github/lowc1012/submarine/builds/688019593)
### Screenshots (if appropriate)
<img width="1273" alt="screenshot1" src="https://user-images.githubusercontent.com/52355146/82148056-a20baa80-9884-11ea-8c77-30fe8ed741dd.png">
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Ryan Lo <lo...@gmail.com>
Closes #290 from lowc1012/SUBMARINE-493 and squashes the following commits:
fdcba8d [Ryan Lo] SUBMARINE-493. Change the path of jupyter docker-images
2c027d4 [Ryan Lo] SUBMARINE-493. typo fixed
71dce2c [Ryan Lo] SUBMARINE-493. Integrating jupyter notebook into Submarine
---
dev-support/docker-images/jupyter/Dockerfile | 83 ++++++++++++++++++++++
dev-support/docker-images/jupyter/build.sh | 33 +++++++++
submarine-cloud/hack/deploy-submarine.sh | 6 ++
submarine-cloud/hack/integration-test.sh | 1 +
submarine-cloud/hack/kind-cluster-build.sh | 3 +
.../manifests/submarine-cluster/jupyter.yaml | 53 ++++++++++++++
.../submarine-cluster/submarine-server.yaml | 4 ++
7 files changed, 183 insertions(+)
diff --git a/dev-support/docker-images/jupyter/Dockerfile b/dev-support/docker-images/jupyter/Dockerfile
new file mode 100644
index 0000000..69eb76e
--- /dev/null
+++ b/dev-support/docker-images/jupyter/Dockerfile
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE=tensorflow/tensorflow:2.1.0-py3-jupyter
+
+FROM $BASE_IMAGE
+
+ARG NB_USER="jovyan"
+ARG NB_UID="1000"
+ARG NB_PREFIX="/jupyter/"
+ARG NB_PORT=8888
+
+USER root
+ENV NB_USER $NB_USER
+ENV NB_UID $NB_UID
+ENV NB_GID $NB_GID
+ENV NB_PREFIX $NB_PREFIX
+ENV NB_PORT $NB_PORT
+
+ENV PATH=$HOME/.local/bin:$PATH
+ENV HOME=/home/$NB_USER
+
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
+ apt-transport-https \
+ build-essential \
+ curl \
+ wget \
+ vim \
+ bzip2 \
+ ca-certificates \
+ sudo \
+ locales \
+ fonts-liberation \
+ run-one \
+ python3-pip \
+ python3-dev \
+ python3-setuptools && \
+ apt-get clean && rm -rf /var/lib/apt/lists/*
+
+RUN echo "$LOG_TAG Set locale" && \
+ echo "LC_ALL=en_US.UTF-8" >> /etc/environment && \
+ echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && \
+ echo "LANG=en_US.UTF-8" > /etc/locale.conf && \
+ locale-gen en_US.UTF-8
+
+ENV LANG=en_US.UTF-8 \
+ LC_ALL=en_US.UTF-8
+
+# Create NB_USER user with UID=1000 and in the 'users' group
+RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
+ chown -R ${NB_USER}:users /usr/local/bin && \
+ mkdir -p $HOME && \
+ chown -R ${NB_USER}:users ${HOME}
+
+
+# Add Tini
+ENV TINI_VERSION v0.19.0
+ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
+RUN mv /tini /usr/local/bin/tini && chmod +x /usr/local/bin/tini
+
+# Install python package
+RUN pip uninstall -y enum34
+RUN pip3 --no-cache-dir install \
+ jupyterhub \
+ jupyterlab
+
+# Configure container startup
+EXPOSE $NB_PORT
+USER $NB_UID
+ENTRYPOINT ["tini", "-g", "--"]
+CMD ["sh","-c", "jupyter notebook --notebook-dir=/home/${NB_USER} --ip=0.0.0.0 --no-browser --allow-root --port=${NB_PORT} --NotebookApp.token='' --NotebookApp.password='' --NotebookApp.allow_origin='*' --NotebookApp.base_url=${NB_PREFIX}"]
diff --git a/dev-support/docker-images/jupyter/build.sh b/dev-support/docker-images/jupyter/build.sh
new file mode 100755
index 0000000..19dc61d
--- /dev/null
+++ b/dev-support/docker-images/jupyter/build.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eo pipefail
+set -x
+
+TF_JUPYTER_IMAGE="apache/submarine:tf2.1.0-jupyter"
+
+if [ -L ${BASH_SOURCE-$0} ]; then
+ PWD=$(dirname $(readlink "${BASH_SOURCE-$0}"))
+else
+ PWD=$(dirname ${BASH_SOURCE-$0})
+fi
+export CURRENT_PATH=$(cd "${PWD}">/dev/null; pwd)
+SUBMARINE_HOME=${CURRENT_PATH}/../../..
+
+# build image
+echo "Start building the ${TF_JUPYTER_IMAGE} docker image ..."
+cd ${CURRENT_PATH}
+docker build -t ${TF_JUPYTER_IMAGE} .
diff --git a/submarine-cloud/hack/deploy-submarine.sh b/submarine-cloud/hack/deploy-submarine.sh
index 62deb44..edb59d3 100755
--- a/submarine-cloud/hack/deploy-submarine.sh
+++ b/submarine-cloud/hack/deploy-submarine.sh
@@ -21,6 +21,7 @@ ROOT=$(unset CDPATH && cd $(dirname "${BASH_SOURCE[0]}")/.. && pwd)
cd $ROOT
SUBMARINE_HOME=${ROOT}/..
SUBMARINE_VERSION="0.4.0-SNAPSHOT"
+TF_JUPYTER_IMAGE="apache/submarine:tf2.1.0-jupyter";
source $ROOT/hack/lib.sh
@@ -70,6 +71,11 @@ function install_submarine() {
fi
$KIND_BIN load docker-image apache/submarine:database-${SUBMARINE_VERSION}
+ if ! docker inspect ${TF_JUPYTER_IMAGE} >/dev/null ; then
+ docker pull ${TF_JUPYTER_IMAGE}
+ fi
+ $KIND_BIN load docker-image ${TF_JUPYTER_IMAGE} >/dev/null
+
if ! docker inspect apache/submarine:server-${SUBMARINE_VERSION} >/dev/null ; then
docker pull apache/submarine:server-${SUBMARINE_VERSION}
fi
diff --git a/submarine-cloud/hack/integration-test.sh b/submarine-cloud/hack/integration-test.sh
index db39469..af109e3 100755
--- a/submarine-cloud/hack/integration-test.sh
+++ b/submarine-cloud/hack/integration-test.sh
@@ -75,6 +75,7 @@ function update_docker_images() {
$SUBMARINE_HOME/dev-support/docker-images/database/build.sh
$SUBMARINE_HOME/dev-support/docker-images/operator/build.sh
$SUBMARINE_HOME/dev-support/docker-images/submarine/build.sh
+ $SUBMARINE_HOME/dev-support/docker-images/jupyter/build.sh
docker images
}
diff --git a/submarine-cloud/hack/kind-cluster-build.sh b/submarine-cloud/hack/kind-cluster-build.sh
index b88ff7b..ca61972 100755
--- a/submarine-cloud/hack/kind-cluster-build.sh
+++ b/submarine-cloud/hack/kind-cluster-build.sh
@@ -136,9 +136,12 @@ EOF
for ((k=1;k<=${volumeNum};k++))
do
mkdir -p ${data_dir}/worker${i}/vol${k}
+ mkdir -p ${data_dir}/worker${i}/submarine-jupyter
cat <<EOF >> ${configFile}
- containerPath: /mnt/disks/vol${k}
hostPath: ${data_dir}/worker${i}/vol${k}
+ - containerPath: /mnt/disks/submarine-jupyter
+ hostPath: ${data_dir}/worker${i}/submarine-jupyter
EOF
done
done
diff --git a/submarine-cloud/manifests/submarine-cluster/jupyter.yaml b/submarine-cloud/manifests/submarine-cluster/jupyter.yaml
new file mode 100644
index 0000000..1015c68
--- /dev/null
+++ b/submarine-cloud/manifests/submarine-cluster/jupyter.yaml
@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+ name: jupyter-notebook
+ labels:
+ app: jupyter
+spec:
+ selector:
+ matchLabels:
+ app: jupyter
+ replicas: 1
+ template:
+ metadata:
+ labels:
+ app: jupyter
+ spec:
+ containers:
+ - name: jupyter
+ image: apache/submarine:tf2.1.0-jupyter
+ imagePullPolicy: IfNotPresent
+ volumeMounts:
+ - mountPath: /home/jovyan
+ name: jupyter-storage
+ volumes:
+ - name: jupyter-storage
+ hostPath:
+ path: /mnt/disks/submarine-jupyter
+ type: DirectoryOrCreate
+---
+kind: Service
+apiVersion: v1
+metadata:
+ name: jupyter-svc
+spec:
+ selector:
+ app: jupyter
+ ports:
+ - port: 8888
diff --git a/submarine-cloud/manifests/submarine-cluster/submarine-server.yaml b/submarine-cloud/manifests/submarine-cluster/submarine-server.yaml
index 76f1d73..5d7f9a3 100644
--- a/submarine-cloud/manifests/submarine-cluster/submarine-server.yaml
+++ b/submarine-cloud/manifests/submarine-cluster/submarine-server.yaml
@@ -108,6 +108,10 @@ spec:
backend:
serviceName: submarine-svc
servicePort: 8080
+ - path: /jupyter
+ backend:
+ serviceName: jupyter-svc
+ servicePort: 8888
---
# You can also access the submarine workbench via port-forward
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org