You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by ch...@apache.org on 2022/07/21 14:17:43 UTC
[submarine] branch master updated: SUBMARINE-1292. Add submarine-cloud-v3 e2e test
This is an automated email from the ASF dual-hosted git repository.
chishengliu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new e4df9741 SUBMARINE-1292. Add submarine-cloud-v3 e2e test
e4df9741 is described below
commit e4df9741a547dd8aae3058863f35529c88fa62ca
Author: joshvictor1024 <jo...@gmail.com>
AuthorDate: Thu Jul 14 09:09:31 2022 +0800
SUBMARINE-1292. Add submarine-cloud-v3 e2e test
### What is this PR for?
<!-- A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://submarine.apache.org/contribution/contributions.html
-->
Add end-to-end test to submarine-cloud-v3
### What type of PR is it?
Feature
### Todos
* [x] - prepare to run in-cluster
* [x] - write e2e test and add to Github Action
* [x] - add new development doc
### What is the Jira issue?
<!-- * Open an issue on Jira https://issues.apache.org/jira/browse/SUBMARINE/
* Put link here, and add [SUBMARINE-*Jira number*] in PR title, eg. `SUBMARINE-23. PR title`
-->
https://issues.apache.org/jira/browse/SUBMARINE-1292
### How should this be tested?
<!--
* First time? Setup Travis CI as described on https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
-->
Follow `submarine-cloud-v3/docs/developer-guide.md`
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? Yes
Author: joshvictor1024 <jo...@gmail.com>
Signed-off-by: Chi-Sheng Liu <ch...@apache.org>
Closes #978 from joshvictor1024/SUBMARINE-1292 and squashes the following commits:
eba20cab [joshvictor1024] change operator image user and group name
35a7c9de [joshvictor1024] add e2e to GA
8bda1ef3 [joshvictor1024] add e2e test
ad0eea60 [joshvictor1024] prepare to run in-cluster
5bd50968 [joshvictor1024] add rbac rules
ae4e3f6f [joshvictor1024] fix no regular resync
---
.../scripts/build-image-locally-v3.sh | 41 ++--
.github/workflows/master.yml | 91 +++++++++
.../docker-images/operator-v3}/Dockerfile | 47 ++---
dev-support/docker-images/operator-v3/build.sh | 38 ++++
submarine-cloud-v3/Dockerfile | 26 ++-
submarine-cloud-v3/Makefile | 2 +-
.../artifacts/submarine-virtualservice.yaml | 2 +-
.../config/default/kustomization.yaml | 23 ++-
.../patches/psp-clustertype-kubernetes-patch.yaml} | 38 ++--
.../patches/psp-clustertype-openshift-patch.yaml} | 38 ++--
submarine-cloud-v3/config/manager/manager.yaml | 7 +-
submarine-cloud-v3/config/rbac/role.yaml | 97 +++-------
.../controllers/submarine_controller.go | 30 +--
.../controllers/submarine_controller_test.go | 210 +++++++++++++++++++++
submarine-cloud-v3/controllers/suite_test.go | 32 +++-
submarine-cloud-v3/docs/developer-guide.md | 58 ++++--
16 files changed, 568 insertions(+), 212 deletions(-)
diff --git a/submarine-cloud-v3/Dockerfile b/.github/scripts/build-image-locally-v3.sh
old mode 100644
new mode 100755
similarity index 50%
copy from submarine-cloud-v3/Dockerfile
copy to .github/scripts/build-image-locally-v3.sh
index 7c009d21..81203af7
--- a/submarine-cloud-v3/Dockerfile
+++ b/.github/scripts/build-image-locally-v3.sh
@@ -1,3 +1,4 @@
+#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
@@ -15,30 +16,20 @@
# limitations under the License.
#
-# Build the manager binary
-FROM golang:1.17 as builder
+SUBMARINE_VERSION="0.8.0-SNAPSHOT"
+FOLDER_LIST=("database" "mlflow" "submarine" "operator-v3")
+IMAGE_LIST=(
+ "apache/submarine:database-${SUBMARINE_VERSION}"
+ "apache/submarine:mlflow-${SUBMARINE_VERSION}"
+ "apache/submarine:server-${SUBMARINE_VERSION}"
+ "apache/submarine:operator-${SUBMARINE_VERSION}"
+)
-WORKDIR /workspace
-# Copy the Go Modules manifests
-COPY go.mod go.mod
-COPY go.sum go.sum
-# cache deps before building and copying source so that we don't need to re-download as much
-# and so that source changes don't invalidate our downloaded layer
-RUN go mod download
+for i in "${!IMAGE_LIST[@]}"
+do
+ echo "Build Image ${IMAGE_LIST[i]}"
+ echo "Execute ./dev-support/docker-images/${FOLDER_LIST[i]}/build.sh"
+ ./dev-support/docker-images/"${FOLDER_LIST[i]}"/build.sh
+ kind load docker-image "${IMAGE_LIST[i]}"
+done
-# Copy the go source
-COPY main.go main.go
-COPY api/ api/
-COPY controllers/ controllers/
-
-# Build
-RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
-
-# Use distroless as minimal base image to package the manager binary
-# Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:nonroot
-WORKDIR /
-COPY --from=builder /workspace/manager .
-USER 65532:65532
-
-ENTRYPOINT ["/manager"]
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index dcaafecc..9d74e5b4 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -132,6 +132,97 @@ jobs:
#Never cache local artifacts
rm -rf ~/.m2/repository/org/apache/submarine
if: always()
+ submarine-operator-v3-e2e-test:
+ needs: generate-k8s-versions-array
+ runs-on: ubuntu-latest
+ timeout-minutes: 60
+ strategy:
+ matrix:
+ k8s-version: ${{fromJSON(needs.generate-k8s-versions-array.outputs.matrix)}}
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ fetch-depth: 50
+ - name: Set up JDK 1.8
+ uses: actions/setup-java@v1
+ with:
+ java-version: "1.8"
+ - name: Set up Maven 3.6.3
+ uses: stCarolas/setup-maven@v4
+ with:
+ maven-version: 3.6.3
+ - name: Setup Golang 1.17.2
+ uses: actions/setup-go@v2
+ with:
+ go-version: "1.17.2"
+ - uses: actions/cache@v2
+ with:
+ path: ~/.m2/repository
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ${{ runner.os }}-maven-
+ - name: Check version
+ run: |
+ mvn --version
+ java -version
+ go version
+ helm version
+ kind version
+ - name: Create kind cluster
+ run: kind create cluster --config ./.github/config/kind-config-kind.yaml --wait 3m --image kindest/node:${{ matrix.k8s-version }}
+ - name: Download Istio 1.13.0
+ run: |
+ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.13.0 sh -
+ cd istio-1.13.0
+ echo "$PWD/bin" >> $GITHUB_PATH
+ - name: Install Istio
+ run: istioctl install -y --set values.global.proxy.resources.requests.cpu=10m
+ - name: Show K8s cluster information
+ run: |
+ kubectl cluster-info
+ kubectl version
+ kubectl get pods -n kube-system
+ echo "current-context:" $(kubectl config current-context)
+ echo "environment-kubeconfig:" ${KUBECONFIG}
+ - name: Build Image locally
+ run: .github/scripts/build-image-locally-v3.sh
+ - name: Install Golang Dependencies
+ working-directory: submarine-cloud-v3
+ run: go mod vendor
+ - name: Install Submarine operator
+ working-directory: submarine-cloud-v3
+ # run: make deploy
+ # GA env variable VERSION overrides the variable in makefile,
+ # which results in the wrong image name.
+ # Running commands manually instead.
+ run: |
+ cd config/manager
+ kustomize edit set image controller=apache/submarine:operator-0.8.0-SNAPSHOT
+ kustomize build ../default | kubectl apply -f -
+ - name: Install Helm Dependencies
+ run: |
+ sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
+ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+ helm install --wait --set dev=true --set storageClass.provisioner=rancher.io/local-path --set storageClass.volumeBindingMode=WaitForFirstConsumer submarine ./helm-charts/submarine -n submarine-cloud-v3-system
+ - name: Run end-to-end test
+ working-directory: submarine-cloud-v3
+ run: go test ./controllers/ -v -ginkgo.v
+ - name: Failure status
+ run: |
+ kubectl describe nodes
+ kubectl get all -A
+ kubectl get events -A --sort-by='{.lastTimestamp}'
+ kubectl logs -l control-plane=controller-manager -n submarine-cloud-v3-system
+ kubectl logs -l app=notebook-controller -n submarine-cloud-v3-system
+ kubectl logs -l control-plane=kubeflow-training-operator -n submarine-cloud-v3-system
+ kubectl describe submarine -A
+ if: ${{ failure() }}
+ - name: Delete temporary build artifacts before caching
+ run: |
+ #Never cache local artifacts
+ rm -rf ~/.m2/repository/org/apache/submarine
+ if: always()
submarine-e2e:
runs-on: ubuntu-latest
timeout-minutes: 60
diff --git a/submarine-cloud-v3/Dockerfile b/dev-support/docker-images/operator-v3/Dockerfile
similarity index 54%
copy from submarine-cloud-v3/Dockerfile
copy to dev-support/docker-images/operator-v3/Dockerfile
index 7c009d21..87f12256 100644
--- a/submarine-cloud-v3/Dockerfile
+++ b/dev-support/docker-images/operator-v3/Dockerfile
@@ -15,30 +15,31 @@
# limitations under the License.
#
+# Adapted from submarine-cloud-v3/Dockerfile and dev-support/docker-images/operator/Dockerfile
+
# Build the manager binary
-FROM golang:1.17 as builder
-
-WORKDIR /workspace
-# Copy the Go Modules manifests
-COPY go.mod go.mod
-COPY go.sum go.sum
-# cache deps before building and copying source so that we don't need to re-download as much
-# and so that source changes don't invalidate our downloaded layer
-RUN go mod download
-
-# Copy the go source
-COPY main.go main.go
-COPY api/ api/
-COPY controllers/ controllers/
-
-# Build
+FROM golang:1.17.2 as build-image
+MAINTAINER Apache Software Foundation <de...@submarine.apache.org>
+
+ADD tmp/submarine-cloud-v3 /usr/src
+
+WORKDIR /usr/src
+
+# use CGO_ENABLED=0 to support alpine image
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
-# Use distroless as minimal base image to package the manager binary
-# Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:nonroot
-WORKDIR /
-COPY --from=builder /workspace/manager .
-USER 65532:65532
+# use alpine to support shell params
+FROM alpine
+MAINTAINER Apache Software Foundation <de...@submarine.apache.org>
+
+WORKDIR /usr/src
+COPY tmp/submarine-cloud-v3/artifacts/ /usr/src/artifacts/
+COPY --from=build-image /usr/src/manager /usr/src/manager
+
+# manager is run as non-root user
+# See submarine-cloud-v3/config/manager/manager.yaml
+RUN addgroup -g 1000 -S submarine
+RUN adduser -u 1000 -S submarine -G submarine
+USER submarine
-ENTRYPOINT ["/manager"]
+CMD /usr/src/manager
diff --git a/dev-support/docker-images/operator-v3/build.sh b/dev-support/docker-images/operator-v3/build.sh
new file mode 100755
index 00000000..eca60d94
--- /dev/null
+++ b/dev-support/docker-images/operator-v3/build.sh
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euxo pipefail
+
+SUBMARINE_VERSION=0.8.0-SNAPSHOT
+SUBMARINE_IMAGE_NAME="apache/submarine:operator-${SUBMARINE_VERSION}"
+
+if [ -L ${BASH_SOURCE-$0} ]; then
+ PWD=$(dirname $(readlink "${BASH_SOURCE-$0}"))
+else
+ PWD=$(dirname ${BASH_SOURCE-$0})
+fi
+export CURRENT_PATH=$(cd "${PWD}">/dev/null; pwd)
+export SUBMARINE_HOME=${CURRENT_PATH}/../../..
+
+mkdir -p "${CURRENT_PATH}/tmp"
+cp -r ${SUBMARINE_HOME}/submarine-cloud-v3/ "${CURRENT_PATH}/tmp/submarine-cloud-v3/"
+
+cd ${CURRENT_PATH}
+echo "Start building the ${SUBMARINE_IMAGE_NAME} docker image ..."
+docker build -t ${SUBMARINE_IMAGE_NAME} .
+
+# clean temp file
+rm -rf "${CURRENT_PATH}/tmp"
diff --git a/submarine-cloud-v3/Dockerfile b/submarine-cloud-v3/Dockerfile
index 7c009d21..6efbde21 100644
--- a/submarine-cloud-v3/Dockerfile
+++ b/submarine-cloud-v3/Dockerfile
@@ -16,7 +16,8 @@
#
# Build the manager binary
-FROM golang:1.17 as builder
+FROM golang:1.17.2 as builder
+MAINTAINER Apache Software Foundation <de...@submarine.apache.org>
WORKDIR /workspace
# Copy the Go Modules manifests
@@ -32,13 +33,22 @@ COPY api/ api/
COPY controllers/ controllers/
# Build
+# use CGO_ENABLED=0 to support alpine image
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
-# Use distroless as minimal base image to package the manager binary
-# Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:nonroot
-WORKDIR /
-COPY --from=builder /workspace/manager .
-USER 65532:65532
+# use alpine to support shell params
+FROM alpine
+MAINTAINER Apache Software Foundation <de...@submarine.apache.org>
-ENTRYPOINT ["/manager"]
+WORKDIR /usr/src
+COPY artifacts/ /usr/src/artifacts/
+COPY --from=builder /workspace/manager /usr/src/manager
+
+# manager is run as non-root user
+# See config/manager/manager.yaml
+RUN addgroup -g 1000 -S submarine
+RUN adduser -u 1000 -S submarine -G submarine
+USER submarine
+
+# CMD ls
+CMD /usr/src/manager
diff --git a/submarine-cloud-v3/Makefile b/submarine-cloud-v3/Makefile
index 99c91a64..7f4b6ad2 100644
--- a/submarine-cloud-v3/Makefile
+++ b/submarine-cloud-v3/Makefile
@@ -134,7 +134,7 @@ run: manifests generate fmt vet ## Run a controller from your host.
go run ./main.go
.PHONY: docker-build
-docker-build: test ## Build docker image with the manager.
+docker-build: ## Build docker image with the manager.
docker build -t ${IMG} .
.PHONY: docker-push
diff --git a/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml b/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
index 961ab006..6405ddea 100644
--- a/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
@@ -24,7 +24,7 @@ spec:
hosts:
- "*"
gateways:
- - submarine/submarine-gateway
+ - submarine-cloud-v3-system/submarine-gateway
http:
- match:
- uri:
diff --git a/submarine-cloud-v3/config/default/kustomization.yaml b/submarine-cloud-v3/config/default/kustomization.yaml
index 816b193c..9bc434b2 100644
--- a/submarine-cloud-v3/config/default/kustomization.yaml
+++ b/submarine-cloud-v3/config/default/kustomization.yaml
@@ -41,11 +41,32 @@ bases:
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
#- ../prometheus
+patchesJson6902:
+# [PSP] To create a PodSecurityPolicy, uncomment all the sections with [PSP]
+# ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
+# This configuration enables the database/minio/server to set securityContext.runAsUser
+# [PSP] To create a PodSecurityPolicy, you must set the value of cluster type as "kubernetes" or "openshift"
+# To set it to "kubernetes", uncomment the following sections
+- path: patches/psp-clustertype-kubernetes-patch.yaml
+ target:
+ group: rbac.authorization.k8s.io
+ version: v1
+ kind: ClusterRole
+ name: manager-role
+# [PSP] To create a PodSecurityPolicy, you must set the value of cluster type as "kubernetes" or "openshift"
+# To set it to "openshift", uncomment the following sections
+# - path: patches/psp-clustertype-openshift-patch.yaml
+# target:
+# group: rbac.authorization.k8s.io
+# version: v1
+# kind: ClusterRole
+# name: manager-role
+
patchesStrategicMerge:
# Protect the /metrics endpoint by putting it behind auth.
# If you want your controller-manager to expose the /metrics
# endpoint w/o any authn/z, please comment the following line.
-- manager_auth_proxy_patch.yaml
+# - manager_auth_proxy_patch.yaml
# Mount the controller config file for loading manager configurations
# through a ComponentConfig type
diff --git a/submarine-cloud-v3/Dockerfile b/submarine-cloud-v3/config/default/patches/psp-clustertype-kubernetes-patch.yaml
similarity index 50%
copy from submarine-cloud-v3/Dockerfile
copy to submarine-cloud-v3/config/default/patches/psp-clustertype-kubernetes-patch.yaml
index 7c009d21..50679094 100644
--- a/submarine-cloud-v3/Dockerfile
+++ b/submarine-cloud-v3/config/default/patches/psp-clustertype-kubernetes-patch.yaml
@@ -15,30 +15,14 @@
# limitations under the License.
#
-# Build the manager binary
-FROM golang:1.17 as builder
-
-WORKDIR /workspace
-# Copy the Go Modules manifests
-COPY go.mod go.mod
-COPY go.sum go.sum
-# cache deps before building and copying source so that we don't need to re-download as much
-# and so that source changes don't invalidate our downloaded layer
-RUN go mod download
-
-# Copy the go source
-COPY main.go main.go
-COPY api/ api/
-COPY controllers/ controllers/
-
-# Build
-RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
-
-# Use distroless as minimal base image to package the manager binary
-# Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:nonroot
-WORKDIR /
-COPY --from=builder /workspace/manager .
-USER 65532:65532
-
-ENTRYPOINT ["/manager"]
+- op: add
+ path: /rules/-
+ value:
+ apiGroups:
+ - policy
+ resources:
+ - podsecuritypolicies
+ verbs:
+ - use
+ resourceNames:
+ - submarine-anyuid
diff --git a/submarine-cloud-v3/Dockerfile b/submarine-cloud-v3/config/default/patches/psp-clustertype-openshift-patch.yaml
similarity index 50%
copy from submarine-cloud-v3/Dockerfile
copy to submarine-cloud-v3/config/default/patches/psp-clustertype-openshift-patch.yaml
index 7c009d21..3abd126e 100644
--- a/submarine-cloud-v3/Dockerfile
+++ b/submarine-cloud-v3/config/default/patches/psp-clustertype-openshift-patch.yaml
@@ -15,30 +15,14 @@
# limitations under the License.
#
-# Build the manager binary
-FROM golang:1.17 as builder
-
-WORKDIR /workspace
-# Copy the Go Modules manifests
-COPY go.mod go.mod
-COPY go.sum go.sum
-# cache deps before building and copying source so that we don't need to re-download as much
-# and so that source changes don't invalidate our downloaded layer
-RUN go mod download
-
-# Copy the go source
-COPY main.go main.go
-COPY api/ api/
-COPY controllers/ controllers/
-
-# Build
-RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
-
-# Use distroless as minimal base image to package the manager binary
-# Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:nonroot
-WORKDIR /
-COPY --from=builder /workspace/manager .
-USER 65532:65532
-
-ENTRYPOINT ["/manager"]
+- op: add
+ path: /rules/-
+ value:
+ apiGroups:
+ - security.openshift.io
+ resources:
+ - securitycontextconstraints
+ verbs:
+ - use
+ resourceNames:
+ - anyuid
diff --git a/submarine-cloud-v3/config/manager/manager.yaml b/submarine-cloud-v3/config/manager/manager.yaml
index a0e1d4a0..b91ea946 100644
--- a/submarine-cloud-v3/config/manager/manager.yaml
+++ b/submarine-cloud-v3/config/manager/manager.yaml
@@ -20,6 +20,7 @@ kind: Namespace
metadata:
labels:
control-plane: controller-manager
+ istio-injection: enabled
name: system
---
apiVersion: apps/v1
@@ -43,11 +44,13 @@ spec:
spec:
securityContext:
runAsNonRoot: true
+ runAsUser: 1000
containers:
- command:
- - /manager
+ - /usr/src/manager
args:
- - --leader-elect
+ - -clustertype=kubernetes
+ - -createpsp=true
image: controller:latest
name: manager
securityContext:
diff --git a/submarine-cloud-v3/config/rbac/role.yaml b/submarine-cloud-v3/config/rbac/role.yaml
index 7384f94b..cdb8c456 100644
--- a/submarine-cloud-v3/config/rbac/role.yaml
+++ b/submarine-cloud-v3/config/rbac/role.yaml
@@ -22,6 +22,16 @@ metadata:
creationTimestamp: null
name: manager-role
rules:
+- apiGroups:
+ - ""
+ resources:
+ - configmaps
+ - jobs
+ - namespaces
+ - pods
+ - secrets
+ verbs:
+ - '*'
- apiGroups:
- ""
resources:
@@ -30,101 +40,54 @@ rules:
- create
- patch
- apiGroups:
- - apps
+ - apiextensions.k8s.io
resources:
- - deployments
+ - customresourcedefinitions
verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
+ - '*'
- apiGroups:
- apps
resources:
+ - deployments
- statefulsets
verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
+ - '*'
- apiGroups:
- - ""
+ - apps
resources:
- - persistentvolumeclaims
+ - replicasets
verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
+ - '*'
- apiGroups:
- ""
resources:
+ - persistentvolumeclaims
- serviceaccounts
+ - services
verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
+ - '*'
- apiGroups:
- - ""
+ - kubeflow.org
resources:
- - services
+ - notebooks
+ - pytorchjobs
+ - tfjobs
+ - xgboostjobs
verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
+ - '*'
- apiGroups:
- networking.istio.io
resources:
- virtualservices
verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
+ - '*'
- apiGroups:
- - rbac
+ - rbac.authorization.k8s.io
resources:
- rolebindings
- verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
-- apiGroups:
- - rbac
- resources:
- roles
verbs:
- - create
- - delete
- - get
- - list
- - patch
- - update
- - watch
+ - '*'
- apiGroups:
- submarine.apache.org
resources:
diff --git a/submarine-cloud-v3/controllers/submarine_controller.go b/submarine-cloud-v3/controllers/submarine_controller.go
index fa334c8f..4c1d29bf 100644
--- a/submarine-cloud-v3/controllers/submarine_controller.go
+++ b/submarine-cloud-v3/controllers/submarine_controller.go
@@ -20,6 +20,7 @@ import (
"context"
"encoding/json"
"fmt"
+ "time"
"github.com/go-logr/logr"
@@ -119,23 +120,27 @@ type SubmarineReconciler struct {
// On change, run `make manifest` to update config/rbac/role.yaml
// Reference: https://github.com/apache/submarine/blob/master/submarine-cloud-v3/config/rbac/role.yaml
-// Submarine resource
+// Submarine resources
//+kubebuilder:rbac:groups=submarine.apache.org,resources=submarines,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=submarine.apache.org,resources=submarines/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=submarine.apache.org,resources=submarines/finalizers,verbs=update
// Event
//+kubebuilder:rbac:groups="",resources=events,verbs=create;patch
+//+kubebuilder:rbac:groups="",resources=pods;secrets;configmaps;namespaces;jobs,verbs="*"
+//+kubebuilder:rbac:groups="apiextensions.k8s.io",resources=customresourcedefinitions,verbs="*"
-// Other resources
-//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
-//+kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
-//+kubebuilder:rbac:groups=core,resources=serviceaccounts,verbs=get;list;watch;create;update;patch;delete
-//+kubebuilder:rbac:groups=core,resources=persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete
-//+kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
-//+kubebuilder:rbac:groups=rbac,resources=roles,verbs=get;list;watch;create;update;patch;delete
-//+kubebuilder:rbac:groups=rbac,resources=rolebindings,verbs=get;list;watch;create;update;patch;delete
-//+kubebuilder:rbac:groups=networking.istio.io,resources=virtualservices,verbs=get;list;watch;create;update;patch;delete
+// k8s resources
+//+kubebuilder:rbac:groups=apps,resources=deployments;statefulsets,verbs="*"
+//+kubebuilder:rbac:groups=apps,resources=replicasets,verbs="*"
+//+kubebuilder:rbac:groups=core,resources=persistentvolumeclaims;serviceaccounts;services,verbs="*"
+//+kubebuilder:rbac:groups=rbac.authorization.k8s.io,resources=roles;rolebindings,verbs="*"
+
+// kubeflow resources
+//+kubebuilder:rbac:groups=kubeflow.org,resources=notebooks;pytorchjobs;tfjobs;xgboostjobs,verbs="*"
+
+// Istio resources
+//+kubebuilder:rbac:groups=networking.istio.io,resources=virtualservices,verbs="*"
// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
@@ -222,7 +227,10 @@ func (r *SubmarineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (
return ctrl.Result{}, err
}
- return ctrl.Result{}, nil
+ // Re-run Reconcile regularly
+ result := ctrl.Result{}
+ result.RequeueAfter = time.Second * 30 // default resync period
+ return result, nil
}
func (r *SubmarineReconciler) updateSubmarineStatus(ctx context.Context, submarine, submarineCopy *submarineapacheorgv1alpha1.Submarine) error {
diff --git a/submarine-cloud-v3/controllers/submarine_controller_test.go b/submarine-cloud-v3/controllers/submarine_controller_test.go
new file mode 100644
index 00000000..741e3e87
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_controller_test.go
@@ -0,0 +1,210 @@
+/*
+Copyright 2022.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controllers
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "path/filepath"
+ "time"
+
+ . "github.com/onsi/ginkgo"
+ . "github.com/onsi/gomega"
+ "github.com/pkg/errors"
+ corev1 "k8s.io/api/core/v1"
+ apierrors "k8s.io/apimachinery/pkg/api/errors"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/apimachinery/pkg/types"
+ "k8s.io/apimachinery/pkg/util/yaml"
+ "sigs.k8s.io/controller-runtime/pkg/client"
+
+ submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
+)
+
+var _ = Describe("Submarine controller", func() {
+
+ // Define utility constants and variables
+ const (
+ submarineNamespace = "submarine-test-submit-custom-resource" // The namespace where the Submarine CR is created
+
+ createNsTimeout = time.Second * 10
+ createNsInterval = time.Second * 2
+ createSubmarineTimeout = time.Second * 1200
+ createSubmarineInterval = time.Second * 10
+ deleteSubmarineTimeout = time.Second * 600
+ deleteSubmarineInterval = time.Second * 10
+ deleteNsTimeout = time.Second * 120
+ deleteNsInterval = time.Second * 2
+ )
+ var (
+ // The name of Submarine is specified in the YAML file.
+ // Storing name to call k8sClient.Get with NamespacedName
+ submarineName string
+
+ ctx = context.Background()
+ )
+
+ Context("Create a test namespace", func() {
+ It("Should create a test namespace", func() {
+ msg := fmt.Sprintf("Creating the test namespace %s", submarineNamespace)
+ By(msg)
+
+ ns := &corev1.Namespace{
+ ObjectMeta: metav1.ObjectMeta{
+ Name: submarineNamespace,
+ Labels: map[string]string{
+ "istio-injection": "enabled",
+ },
+ },
+ }
+ Expect(k8sClient.Create(ctx, ns)).Should(Succeed())
+
+ // We'll need to retry getting this newly created namespace, given that creation may not immediately happen.
+ createdNs := &corev1.Namespace{} // stub
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx, types.NamespacedName{Name: submarineNamespace, Namespace: "default"}, createdNs)
+ if err != nil {
+ return false
+ }
+ return true
+ }, createNsTimeout, createNsInterval).Should(BeTrue())
+
+ // The namespace should have Istio label
+ Expect(createdNs.Labels["istio-injection"]).To(Equal("enabled"))
+ })
+ })
+
+ Context("Create Submarine", func() {
+ It("Should create a Submarine and it should become RUNNING", func() {
+ By("Creating a new Submarine")
+ submarine, err := MakeSubmarineFromYaml("../config/samples/_v1alpha1_submarine.yaml")
+ Expect(err).To(BeNil())
+
+ // The name of Submarine is specified in the YAML file.
+ // Storing name to call k8sClient.Get with NamespacedName
+ submarineName = submarine.Name
+
+ // Create Submarine in our namespace
+ submarine.Namespace = submarineNamespace
+ Expect(k8sClient.Create(ctx, submarine)).Should(Succeed())
+
+ // We'll need to retry getting this newly created Submarine, given that creation may not immediately happen.
+ createdSubmarine := &submarineapacheorgv1alpha1.Submarine{} // stub
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx, types.NamespacedName{Name: submarineName, Namespace: submarineNamespace}, createdSubmarine)
+ if err != nil {
+ return false
+ }
+ return true
+ }, createNsTimeout, createNsInterval).Should(BeTrue())
+
+ // Wait for Submarine to be in RUNNING state
+ msg := fmt.Sprintf("Waiting until Submarine %s/%s become RUNNING", submarineName, submarineNamespace)
+ By(msg)
+ Eventually(func() bool {
+ err = k8sClient.Get(ctx, types.NamespacedName{Name: submarineName, Namespace: submarineNamespace}, createdSubmarine)
+ Expect(err).To(BeNil())
+
+ state := createdSubmarine.Status.SubmarineState.State
+ Expect(state).ToNot(Equal(submarineapacheorgv1alpha1.FailedState))
+ if createdSubmarine.Status.SubmarineState.State == submarineapacheorgv1alpha1.RunningState {
+ return true
+ }
+ return false
+ }, createSubmarineTimeout, createSubmarineInterval).Should(BeTrue())
+ })
+ })
+
+ Context("Delete Submarine", func() {
+ It("Should delete the Submarine", func() {
+ Expect(submarineName).ToNot(BeNil())
+
+ msg := fmt.Sprintf("Deleting Submarine %s/%s", submarineName, submarineNamespace)
+ By(msg)
+
+ createdSubmarine := &submarineapacheorgv1alpha1.Submarine{} // stub
+ err := k8sClient.Get(ctx, types.NamespacedName{Name: submarineName, Namespace: submarineNamespace}, createdSubmarine)
+ Expect(err).To(BeNil())
+
+ foreground := metav1.DeletePropagationForeground
+ err = k8sClient.Delete(ctx, createdSubmarine, &client.DeleteOptions{
+ PropagationPolicy: &foreground,
+ })
+ Expect(err).To(BeNil())
+
+ // Wait for Submarine to be deleted entirely
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx, types.NamespacedName{Name: submarineName, Namespace: submarineNamespace}, createdSubmarine)
+ if apierrors.IsNotFound(err) {
+ return true
+ }
+ Expect(err).To(BeNil())
+ return false
+ }, deleteSubmarineTimeout, deleteSubmarineInterval).Should(BeTrue())
+ })
+ })
+
+ Context("Delete the test namespace", func() {
+ It("Should delete the test namespace", func() {
+ msg := fmt.Sprintf("Deleting the test namespace %s", submarineNamespace)
+ By(msg)
+
+ createdNs := &corev1.Namespace{} // stub
+ Expect(k8sClient.Get(ctx, types.NamespacedName{Name: submarineNamespace, Namespace: "default"}, createdNs)).Should(Succeed())
+ Expect(k8sClient.Delete(ctx, createdNs)).Should(Succeed())
+
+ // Wait for submarine to be deleted entirely
+
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx, types.NamespacedName{Name: submarineNamespace, Namespace: "default"}, createdNs)
+ if apierrors.IsNotFound(err) {
+ return true
+ }
+ Expect(err).To(BeNil())
+ return false
+ }, deleteNsTimeout, deleteNsInterval).Should(BeTrue())
+ })
+ })
+})
+
+func MakeSubmarineFromYaml(pathToYaml string) (*submarineapacheorgv1alpha1.Submarine, error) {
+ manifest, err := PathToOSFile(pathToYaml)
+ if err != nil {
+ return nil, err
+ }
+ tmp := submarineapacheorgv1alpha1.Submarine{}
+ if err := yaml.NewYAMLOrJSONDecoder(manifest, 100).Decode(&tmp); err != nil {
+ return nil, errors.Wrap(err, fmt.Sprintf("failed to decode file %s", pathToYaml))
+ }
+ return &tmp, err
+}
+
+// PathToOSFile gets the absolute path from relative path.
+func PathToOSFile(relativePath string) (*os.File, error) {
+ path, err := filepath.Abs(relativePath)
+ if err != nil {
+ return nil, errors.Wrap(err, fmt.Sprintf("failed generate absolute file path of %s", relativePath))
+ }
+
+ manifest, err := os.Open(path)
+ if err != nil {
+ return nil, errors.Wrap(err, fmt.Sprintf("failed to open file %s", path))
+ }
+
+ return manifest, nil
+}
diff --git a/submarine-cloud-v3/controllers/suite_test.go b/submarine-cloud-v3/controllers/suite_test.go
index b73422a3..39b4f485 100644
--- a/submarine-cloud-v3/controllers/suite_test.go
+++ b/submarine-cloud-v3/controllers/suite_test.go
@@ -17,12 +17,15 @@ limitations under the License.
package controllers
import (
+ "context"
"path/filepath"
"testing"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
- "k8s.io/client-go/kubernetes/scheme"
+
+ "k8s.io/apimachinery/pkg/runtime"
+ clientgoscheme "k8s.io/client-go/kubernetes/scheme"
"k8s.io/client-go/rest"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/envtest"
@@ -30,6 +33,8 @@ import (
logf "sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
+ istioscheme "istio.io/client-go/pkg/clientset/versioned/scheme"
+
submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
//+kubebuilder:scaffold:imports
)
@@ -40,6 +45,9 @@ import (
var cfg *rest.Config
var k8sClient client.Client
var testEnv *envtest.Environment
+var scheme *runtime.Scheme
+var ctx context.Context
+var cancel context.CancelFunc
func TestAPIs(t *testing.T) {
RegisterFailHandler(Fail)
@@ -50,30 +58,44 @@ func TestAPIs(t *testing.T) {
}
var _ = BeforeSuite(func() {
+ // Setup ctx and cancel, which are used in BeforeSuite and AfterSuite first,
+ // in case BeforeSuite fails
+ ctx, cancel = context.WithCancel(context.TODO())
+
logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))
By("bootstrapping test environment")
+ useExistingCluster := true
testEnv = &envtest.Environment{
CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
ErrorIfCRDPathMissing: true,
+ UseExistingCluster: &useExistingCluster,
}
cfg, err := testEnv.Start()
Expect(err).NotTo(HaveOccurred())
Expect(cfg).NotTo(BeNil())
- err = submarineapacheorgv1alpha1.AddToScheme(scheme.Scheme)
+ By("adding required schemes")
+ scheme = runtime.NewScheme()
+ err = clientgoscheme.AddToScheme(scheme)
+ Expect(err).NotTo(HaveOccurred())
+ err = istioscheme.AddToScheme(scheme)
+ Expect(err).NotTo(HaveOccurred())
+ err = submarineapacheorgv1alpha1.AddToScheme(scheme)
Expect(err).NotTo(HaveOccurred())
//+kubebuilder:scaffold:scheme
- k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
+ By("creating k8s client")
+ k8sClient, err = client.New(cfg, client.Options{Scheme: scheme})
Expect(err).NotTo(HaveOccurred())
Expect(k8sClient).NotTo(BeNil())
-
-}, 60)
+}, 60) // timeout, seconds
var _ = AfterSuite(func() {
+ cancel()
+
By("tearing down the test environment")
err := testEnv.Stop()
Expect(err).NotTo(HaveOccurred())
diff --git a/submarine-cloud-v3/docs/developer-guide.md b/submarine-cloud-v3/docs/developer-guide.md
index 8250f315..75badc02 100644
--- a/submarine-cloud-v3/docs/developer-guide.md
+++ b/submarine-cloud-v3/docs/developer-guide.md
@@ -22,16 +22,16 @@ Kubernetes version: `1.21.0`
## Prerequisites
-First finish the prerequisites specified in the [QuickStart](https://submarine.apache.org/docs/next/gettingStarted/quickstart) section on the submarine website. Prepare a minikube cluster with Istio installed. Prepare namespaces `submarine` and `submarine-user-test` and label them `istio-injection=enabled`.
+First finish the prerequisites specified in the [QuickStart](https://submarine.apache.org/docs/next/gettingStarted/quickstart) section on the submarine website. Prepare a minikube cluster with Istio installed. Prepare namespaces `submarine-user-test` and `submarine-cloud-v3-system` and label them `istio-injection=enabled`.
Verify with kubectl:
```bash
$ kubectl get namespace --show-labels
-NAME STATUS AGE LABELS
-istio-system Active 7d8h kubernetes.io/metadata.name=istio-system
-submarine Active 7d8h istio-injection=enabled,kubernetes.io/metadata.name=submarine
-submarine-user-test Active 27h istio-injection=enabled,kubernetes.io/metadata.name=submarine-user-test
+NAME STATUS AGE LABELS
+istio-system Active 7d8h kubernetes.io/metadata.name=istio-system
+submarine-user-test Active 27h istio-injection=enabled,kubernetes.io/metadata.name=submarine-user-test
+submarine-cloud-v3-system Active 27h istio-injection=enabled,kubernetes.io/metadata.name=submarine-submarine-cloud-v3-system
$ kubectl get pod -n istio-system
NAME READY STATUS RESTARTS AGE
@@ -39,16 +39,20 @@ istio-ingressgateway-77968dbd74-wq4vb 1/1 Running 1 7d4h
istiod-699b647f8b-nx9rt 1/1 Running 2 7d4h
```
-Next, install submarine dependencies with helm. `--set dev=true` option will not install the operator deployment to the cluster.
+## Run operator out-of-cluster
+
+Before running submarine operator, install submarine dependencies with helm. `--set dev=true` option will not install the operator deployment to the cluster.
```bash
-helm install --set dev=true submarine ../helm-charts/submarine/ -n submarine
+helm install --set dev=true submarine ../helm-charts/submarine/ -n submarine-cloud-v3-system
```
-## Run operator out-of-cluster
+Now we run the submarine operator.
```bash
# Step1: Apply the submarine CRD.
+# Note that the submarine CRD /helm-charts/submarine/crds/crd.yaml is for submarine-cloud-v2.
+# This step will overwrite it with a CRD generated with controller-gen.
make install
# Step2: Run the operator in a terminal.
@@ -73,31 +77,53 @@ make uninstall
```
## Run operator in-cluster
+
+Note that running `make deploy` `make undeploy` creates and deletes `submarine-cloud-v3-system` respectively. Therefore we will run `helm uninstall` before `make undeploy`.
+
```bash
# Step1: Build the docker image.
eval $(minikube docker-env)
make docker-build
eval $(minikube docker-env -u)
-# Step2: Deploy the operator.
-# A new namespace is created with name submarine-cloud-v3-system, and will be used for the deployment.
+# Step2: Install submarine dependencies with Helm
+# If the namespace submarine-cloud-v3-system doesn't exist yet,
+# running Step3 will create it.
+# However, note that if podSecurityPolicy is enabled,
+# the submarine operator pod will not be permitted until running this
+helm install --set dev=true submarine ../helm-charts/submarine/ -n submarine-cloud-v3-system
+
+# Step3: Deploy the operator.
+# 1) Note that the submarine CRD /helm-charts/submarine/crds/crd.yaml is for submarine-cloud-v2.
+# This step will overwrite it with a CRD generated with controller-gen.
+# 2) A new namespace is created with name submarine-cloud-v3-system, and will be used for the deployment.
+# If such a namespace already exists, this step will overwrite its spec.
+# 3) Other resources are created in this namespaces
make deploy
-# Step3: Verify the operator is up and running.
+# Step4: Verify the operator is up and running.
kubectl get deployment -n submarine-cloud-v3-system
-# Step4: Deploy a submarine.
+# Step5: Deploy a submarine.
kubectl apply -n submarine-user-test -f config/samples/_v1alpha1_submarine.yaml
# You can now view the submarine workbench
-# Step5: Cleanup submarine.
+# Step6: Cleanup submarine.
kubectl delete -n submarine-user-test submarine example-submarine
-# Step6: Cleanup operator.
+# Step7: Cleanup submarine dependencies.
+helm uninstall submarine -n submarine-cloud-v3-system
+
+# Step8: Cleanup operator.
+# Note that this step deletes the namespace submarine-cloud-v3-system
make undeploy
```
+### Installing Helm in a different namespace
+
+By default, the Istio virtual service created by the operator binds to the Istio gateway `submarine-cloud-v3-system/submarine-gateway`, which should be installed in the `submarine-cloud-v3-system` namespace via Helm. If `helm install` is run with `-n <your_namespace>`, edit the spec `spec.gateways` of the virtual service to be `<your_namespace>/submarine-gateway` manually once it's created by the operator.
+
### Rebuild Operator Image
When running operator in-cluster, we need to rebuild the operator image for changes to take effect.
@@ -133,3 +159,7 @@ Steps to add new resource created and controlled by the operator:
4. If needed, import new scheme in `main.go`.
5. If there are new resource types, add RBAC marker comments in `contorllers/submarine_controller.go`.
6. Run `make manifests` to update cluster role rules in `config/rbac/role.yaml`.
+
+## Run Operator End-to-end Tests
+
+Have the submarine operator running in-cluster or out-of-cluster before running `make test`. For verbose mode, instead run `go test ./controllers/ -v -ginkgo.v`, which outputs anything written to `GinkgoWriter` to stdout.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org