You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by pi...@apache.org on 2022/12/14 07:37:08 UTC
[submarine] branch master updated: SUBMARINE-1230. Optimize submarine-operator-v3
This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 4d234b01 SUBMARINE-1230. Optimize submarine-operator-v3
4d234b01 is described below
commit 4d234b01e7383a22b5e0ded0d58966e13c8eb16a
Author: cdmikechen <cd...@apache.org>
AuthorDate: Sat Dec 10 22:09:21 2022 +0800
SUBMARINE-1230. Optimize submarine-operator-v3
### What is this PR for?
Optimise submarine-operator-v3 code for production deployment.
### What type of PR is it?
Refactoring
### Todos
* [x] - Optimising the parameters of CRD configuration (e.g. add a config to replace image and other parameters related to image, add env in server, add pull secrets in CR, minio secrets, etc.)
* [x] - Move seldon-secret.yaml to operator
* [x] - Support seldon/submarine gateway config
* [x] - Log timestamp format
* [x] - Fix license
* [x] - Adjusting the host list of the gateway
* [x] - Support for updates of some CR configs (e.g. replicas, image ...)
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-1230
### How should this be tested?
submarine-operator-v3 test case in github action
### Screenshots (if appropriate)
NA
### Questions:
* Do the license files need updating? Yes
* Are there breaking changes for older versions? Yes
* Does this need new documentation? Yes
Author: cdmikechen <cd...@apache.org>
Signed-off-by: Kevin <pi...@apache.org>
Closes #1026 from cdmikechen/SUBMARINE-1230 and squashes the following commits:
9aed80f8 [cdmikechen] Add serve and tensorboard test case
c0e5d8ed [cdmikechen] Add server test case
f3147be5 [cdmikechen] Add some test cases
e415052e [cdmikechen] Database Secret
3c3f3b02 [cdmikechen] Change compare
c66024e2 [cdmikechen] Remove seldon secret step
6456794d [cdmikechen] Fix sidecar cni error
8c7f2508 [cdmikechen] Remove istio sidecar and add finalizers
3cac3477 [cdmikechen] submarine istio gateway
764658da [cdmikechen] continue rebase
dae36345 [cdmikechen] Minio secrets and server database check
cbe5f3b4 [cdmikechen] change namespace to submarine-cloud-v3-system
f18008c3 [cdmikechen] Fix server compare and test error
48db1947 [cdmikechen] Fix status update error
b3768823 [cdmikechen] add compare vs secret server
f73c9dda [cdmikechen] fix vs test
8a811d5f [cdmikechen] test for minikube
f2247e2f [cdmikechen] update helm charts crd.yaml
dfa13ff6 [cdmikechen] add seldon secret and handle the case where common is empty
e4bd2e31 [cdmikechen] move minio secret to Secret
ed773fef [cdmikechen] add env in server
4397e172 [cdmikechen] Change License add minio/mc/busybox/mlflow/tensorflow image add pullsecrets Modify the VirtualService policy
---
.github/workflows/master.yml | 2 +-
dev-support/docker-images/operator-v3/Dockerfile | 8 +-
helm-charts/submarine/crds/crd.yaml | 384 +++++++++++++++++----
.../submarine/templates/submarine-operator.yaml | 8 +
submarine-cloud-v3/Dockerfile | 8 +-
.../api/v1alpha1/groupversion_info.go | 29 +-
submarine-cloud-v3/api/v1alpha1/submarine_types.go | 76 ++--
.../api/v1alpha1/zz_generated.deepcopy.go | 78 ++++-
.../artifacts/submarine-database.yaml | 13 +-
submarine-cloud-v3/artifacts/submarine-minio.yaml | 21 +-
submarine-cloud-v3/artifacts/submarine-mlflow.yaml | 24 +-
.../artifacts/submarine-serve.yaml | 6 +-
.../artifacts/submarine-server-rbac.yaml | 12 +
submarine-cloud-v3/artifacts/submarine-server.yaml | 39 ++-
.../artifacts/submarine-tensorboard.yaml | 1 +
.../artifacts/submarine-virtualservice.yaml | 4 +-
.../crd/bases/submarine.apache.org_submarines.yaml | 161 ++++++++-
.../config/samples/_v1alpha1_submarine.yaml | 15 +-
.../controllers/submarine_controller.go | 66 +++-
.../controllers/submarine_controller_test.go | 106 +++++-
.../controllers/submarine_database.go | 110 +++++-
.../controllers/submarine_database_test.go | 45 +++
submarine-cloud-v3/controllers/submarine_minio.go | 80 ++++-
.../controllers/submarine_minio_test.go | 47 +++
submarine-cloud-v3/controllers/submarine_mlflow.go | 108 +++++-
.../controllers/submarine_mlflow_test.go | 69 ++++
.../controllers/submarine_observer_rbac.go | 5 +-
submarine-cloud-v3/controllers/submarine_serve.go | 96 ++++++
.../controllers/submarine_serve_test.go | 47 +++
submarine-cloud-v3/controllers/submarine_server.go | 125 ++++++-
.../controllers/submarine_server_rbac.go | 5 +-
.../controllers/submarine_server_test.go | 74 ++++
.../controllers/submarine_storage_rbac.go | 7 +-
.../controllers/submarine_tensorboard.go | 53 ++-
.../controllers/submarine_tensorboard_test.go | 52 +++
.../controllers/submarine_virtualservice.go | 59 +++-
submarine-cloud-v3/controllers/suite_test.go | 29 +-
.../controllers/{ => util}/parser.go | 29 +-
submarine-cloud-v3/controllers/util/util.go | 151 ++++++++
submarine-cloud-v3/hack/boilerplate.go.txt | 29 +-
submarine-cloud-v3/main.go | 88 +++--
.../submarine/commons/utils/SubmarineConfVars.java | 9 +
.../pysubmarine/submarine/tracking/client.py | 18 +-
submarine-serve/README.md | 25 +-
submarine-serve/installation/install.sh | 35 --
submarine-serve/pom.xml | 1 -
.../submarine/serve/istio/IstioVirtualService.java | 3 +-
.../serve/istio/IstioVirtualServiceSpec.java | 2 +-
.../submarine/serve/utils/IstioConstants.java | 23 +-
.../org/apache/submarine/server/s3/Client.java | 25 +-
.../apache/submarine/server/s3/S3Constants.java | 14 +-
.../submitter/k8s/parser/ExperimentSpecParser.java | 33 ++
website/docs/gettingStarted/quickstart.md | 13 +-
53 files changed, 2195 insertions(+), 375 deletions(-)
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 68bbcd99..1b0745f3 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -207,7 +207,7 @@ jobs:
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
helm dependency update ./helm-charts/submarine
- helm install --wait --set dev=true --set storageClass.provisioner=rancher.io/local-path --set storageClass.volumeBindingMode=WaitForFirstConsumer submarine ./helm-charts/submarine -n submarine-cloud-v3-system
+ helm install --wait --set dev=true --set storageClass.provisioner=rancher.io/local-path --set storageClass.volumeBindingMode=WaitForFirstConsumer --set seldon-core-operator.istio.gateway=submarine-cloud-v3-system/seldon-gateway submarine ./helm-charts/submarine -n submarine-cloud-v3-system
- name: Run end-to-end test
working-directory: submarine-cloud-v3
run: go test ./controllers/ -v -ginkgo.v
diff --git a/dev-support/docker-images/operator-v3/Dockerfile b/dev-support/docker-images/operator-v3/Dockerfile
index 87f12256..dad70c78 100644
--- a/dev-support/docker-images/operator-v3/Dockerfile
+++ b/dev-support/docker-images/operator-v3/Dockerfile
@@ -42,4 +42,10 @@ RUN addgroup -g 1000 -S submarine
RUN adduser -u 1000 -S submarine -G submarine
USER submarine
-CMD /usr/src/manager
+CMD /usr/src/manager -leader-elect=${SUBMARINE_LEADER_ELECT:-false} \
+ -istioenable=${SUBMARINE_ISTIO_ENABLE} \
+ -submarineateway=${SUBMARINE_ISTIO_SUBMARINE_GATEWAY} \
+ -seldonistioenable=${SUBMARINE_ISTIO_SELDON_ENABLE} \
+ -seldongateway=${SUBMARINE_ISTIO_SELDON_GATEWAY} \
+ -clustertype=${SUBMARINE_CLUSTER_TYPE:kubernetes} \
+ -createpsp=${SUBMARINE_POD_SECURITY_POLICY_ENABLE:-true}
diff --git a/helm-charts/submarine/crds/crd.yaml b/helm-charts/submarine/crds/crd.yaml
index 0d08eb1f..85878974 100644
--- a/helm-charts/submarine/crds/crd.yaml
+++ b/helm-charts/submarine/crds/crd.yaml
@@ -37,75 +37,315 @@ spec:
- all
scope: Namespaced
versions:
- - name: v1alpha1
- schema:
- openAPIV3Schema:
- type: object
- required: ["spec"]
- properties:
- spec:
- type: object
- required:
- - version
- properties:
- version:
- description: submarine docker image version
- type: string
- server:
- type: object
- properties:
- image:
- description: Use this to overwrite the image when development
- type: string
- replicas:
- type: integer
- minimum: 1
- database:
- type: object
- properties:
- image:
- description: Use this to overwrite the image when development
- type: string
- storageSize:
- type: string
- mysqlRootPasswordSecret:
- type: string
- tensorboard:
- type: object
- properties:
- enabled:
- type: boolean
- storageSize:
- type: string
- mlflow:
- type: object
- properties:
- enabled:
- type: boolean
- storageSize:
- type: string
- minio:
- type: object
- properties:
- enabled:
- type: boolean
- storageSize:
- type: string
- status:
- type: object
- properties:
- availableServerReplicas:
- type: integer
- availableDatabaseReplicas:
- type: integer
- submarineState:
- type: object
- properties:
- state:
- type: string
- errorMessage:
- type: string
- served: true
- storage: true
- subresources:
- status: {}
+ - name: v1alpha1
+ schema:
+ openAPIV3Schema:
+ description: Submarine is the Schema for the submarines API
+ properties:
+ apiVersion:
+ description: 'APIVersion defines the versioned schema of this representation
+ of an object. Servers should convert recognized schemas to the latest
+ internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+ type: string
+ kind:
+ description: 'Kind is a string value representing the REST resource this
+ object represents. Servers may infer this from the endpoint the client
+ submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+ type: string
+ metadata:
+ type: object
+ spec:
+ description: SubmarineSpec defines the desired state of Submarine
+ properties:
+ common:
+ description: Common is the spec that defines some submarine common
+ configurations
+ properties:
+ image:
+ description: Image is the basic image used in initcontainer in
+ some submarine resources
+ properties:
+ busybox:
+ description: BusyboxImage is the image used by the submarine
+ service resource (server...) to check whether the minio
+ is started in the initcontainer
+ type: string
+ mc:
+ description: McImage is the image used by the submarine service
+ resource (server...) to check whether the minio is started
+ in the initcontainer
+ type: string
+ pullSecrets:
+ description: PullSecrets is a specified array of imagePullSecrets.
+ This secrets must be manually created in the namespace.
+ items:
+ type: string
+ type: array
+ type: object
+ required:
+ - image
+ type: object
+ database:
+ description: Database is the spec that defines the submarine database
+ properties:
+ image:
+ description: Image is the submarine database's docker image
+ type: string
+ mysqlRootPasswordSecret:
+ description: 'MysqlRootPasswordSecret is the mysql root password
+ secret, this secret need password key: MYSQL_ROOT_PASSWORD'
+ type: string
+ storageSize:
+ description: StorageSize is the storage size of the database
+ type: string
+ required:
+ - mysqlRootPasswordSecret
+ - storageSize
+ type: object
+ minio:
+ description: Minio is the spec that defines the submarine minio
+ properties:
+ accessKey:
+ description: AccessKey defines the access_key of minio
+ type: string
+ enabled:
+ description: Enabled defines whether to enable minio or not
+ type: boolean
+ image:
+ description: Image is the submarine minio's docker image
+ type: string
+ secretKey:
+ description: SecretKey defines the secret_key of minio
+ type: string
+ storageSize:
+ description: StorageSize defines the storage size of minio
+ type: string
+ required:
+ - enabled
+ - storageSize
+ type: object
+ mlflow:
+ description: Mlflow is the spec that defines the submarine mlflow
+ properties:
+ enabled:
+ description: Enabled defines whether to enable mlflow or not
+ type: boolean
+ image:
+ description: Image is the submarine mlflow's docker image
+ type: string
+ storageSize:
+ description: StorageSize defines the storage size of mlflow
+ type: string
+ required:
+ - enabled
+ - storageSize
+ type: object
+ server:
+ description: Server is the spec that defines the submarine server
+ properties:
+ env:
+ description: Envs is the extra environments that submarine server
+ requires
+ items:
+ description: EnvVar represents an environment variable present
+ in a Container.
+ properties:
+ name:
+ description: Name of the environment variable. Must be a
+ C_IDENTIFIER.
+ type: string
+ value:
+ description: 'Variable references $(VAR_NAME) are expanded
+ using the previously defined environment variables in
+ the container and any service environment variables. If
+ a variable cannot be resolved, the reference in the input
+ string will be unchanged. Double $$ are reduced to a single
+ $, which allows for escaping the $(VAR_NAME) syntax: i.e.
+ "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)".
+ Escaped references will never be expanded, regardless
+ of whether the variable exists or not. Defaults to "".'
+ type: string
+ valueFrom:
+ description: Source for the environment variable's value.
+ Cannot be used if value is not empty.
+ properties:
+ configMapKeyRef:
+ description: Selects a key of a ConfigMap.
+ properties:
+ key:
+ description: The key to select.
+ type: string
+ name:
+ description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+ TODO: Add other useful fields. apiVersion, kind,
+ uid?'
+ type: string
+ optional:
+ description: Specify whether the ConfigMap or its
+ key must be defined
+ type: boolean
+ required:
+ - key
+ type: object
+ fieldRef:
+ description: 'Selects a field of the pod: supports metadata.name,
+ metadata.namespace, `metadata.labels[''<KEY>'']`,
+ `metadata.annotations[''<KEY>'']`, spec.nodeName,
+ spec.serviceAccountName, status.hostIP, status.podIP,
+ status.podIPs.'
+ properties:
+ apiVersion:
+ description: Version of the schema the FieldPath
+ is written in terms of, defaults to "v1".
+ type: string
+ fieldPath:
+ description: Path of the field to select in the
+ specified API version.
+ type: string
+ required:
+ - fieldPath
+ type: object
+ resourceFieldRef:
+ description: 'Selects a resource of the container: only
+ resources limits and requests (limits.cpu, limits.memory,
+ limits.ephemeral-storage, requests.cpu, requests.memory
+ and requests.ephemeral-storage) are currently supported.'
+ properties:
+ containerName:
+ description: 'Container name: required for volumes,
+ optional for env vars'
+ type: string
+ divisor:
+ anyOf:
+ - type: integer
+ - type: string
+ description: Specifies the output format of the
+ exposed resources, defaults to "1"
+ pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+ x-kubernetes-int-or-string: true
+ resource:
+ description: 'Required: resource to select'
+ type: string
+ required:
+ - resource
+ type: object
+ secretKeyRef:
+ description: Selects a key of a secret in the pod's
+ namespace
+ properties:
+ key:
+ description: The key of the secret to select from. Must
+ be a valid secret key.
+ type: string
+ name:
+ description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+ TODO: Add other useful fields. apiVersion, kind,
+ uid?'
+ type: string
+ optional:
+ description: Specify whether the Secret or its key
+ must be defined
+ type: boolean
+ required:
+ - key
+ type: object
+ type: object
+ required:
+ - name
+ type: object
+ type: array
+ image:
+ description: Image is the submarine server's docker image
+ type: string
+ replicas:
+ description: Replicas is the number of submarine server's replica
+ format: int32
+ minimum: 1
+ type: integer
+ required:
+ - replicas
+ type: object
+ tensorboard:
+ description: Tensorboard is the spec that defines the submarine tensorboard
+ properties:
+ enabled:
+ description: Enabled defines whether to enable tensorboard or
+ not
+ type: boolean
+ image:
+ description: Image is the submarine tensorboard's docker image
+ type: string
+ storageSize:
+ description: StorageSize defines the storage size of tensorboard
+ type: string
+ required:
+ - enabled
+ - storageSize
+ type: object
+ version:
+ description: Version is the submarine docker image version
+ type: string
+ virtualservice:
+ description: Virtualservice is the spec that defines the submarine
+ virtualservice
+ properties:
+ gateways:
+ description: Hosts is the submarine virtualservice's gateways
+ items:
+ type: string
+ type: array
+ hosts:
+ description: Hosts is the submarine virtualservice's destination
+ hosts
+ items:
+ type: string
+ type: array
+ type: object
+ required:
+ - database
+ - minio
+ - mlflow
+ - server
+ - tensorboard
+ - version
+ type: object
+ status:
+ description: SubmarineStatus defines the observed state of Submarine
+ properties:
+ availableDatabaseReplicas:
+ description: AvailableServerReplicas is the current available replicas
+ of submarine database
+ format: int32
+ type: integer
+ availableServerReplicas:
+ description: AvailableServerReplicas is the current available replicas
+ of submarine server
+ format: int32
+ type: integer
+ submarineState:
+ description: SubmarineState tells the overall submarine state.
+ properties:
+ errorMessage:
+ type: string
+ state:
+ description: SubmarineStateType represents the type of the current
+ state of a submarine.
+ type: string
+ required:
+ - state
+ type: object
+ required:
+ - availableDatabaseReplicas
+ - availableServerReplicas
+ type: object
+ type: object
+ served: true
+ storage: true
+ subresources:
+ status: {}
+status:
+ acceptedNames:
+ kind: ""
+ plural: ""
+ conditions: []
+ storedVersions: []
diff --git a/helm-charts/submarine/templates/submarine-operator.yaml b/helm-charts/submarine/templates/submarine-operator.yaml
index a763d4fa..65093d3a 100644
--- a/helm-charts/submarine/templates/submarine-operator.yaml
+++ b/helm-charts/submarine/templates/submarine-operator.yaml
@@ -59,6 +59,14 @@ spec:
{{- else }}
value: "false"
{{- end }}
+ - name: SUBMARINE_ISTIO_ENABLE
+ value: "{{ .Values.istio.enabled }}"
+ - name: SUBMARINE_ISTIO_SUBMARINE_GATEWAY
+ value: "{{ .Release.Namespace }}/submarine-gateway"
+ - name: SUBMARINE_ISTIO_SELDON_ENABLE
+ value: "{{ index .Values "seldon-core-operator" "istio" "enable" | quote | default "true" }}"
+ - name: SUBMARINE_ISTIO_SELDON_GATEWAY
+ value: {{ index .Values "seldon-core-operator" "istio" "gateway" | default "{{ .Release.Namespace }}/seldon-gateway" }}
serviceAccountName: submarine-operator
status: {}
{{ end }}
diff --git a/submarine-cloud-v3/Dockerfile b/submarine-cloud-v3/Dockerfile
index 6efbde21..07879002 100644
--- a/submarine-cloud-v3/Dockerfile
+++ b/submarine-cloud-v3/Dockerfile
@@ -51,4 +51,10 @@ RUN adduser -u 1000 -S submarine -G submarine
USER submarine
# CMD ls
-CMD /usr/src/manager
+CMD /usr/src/manager -leader-elect=${SUBMARINE_LEADER_ELECT:-false} \
+ -istioenable=${SUBMARINE_ISTIO_ENABLE} \
+ -submarineateway=${SUBMARINE_ISTIO_SUBMARINE_GATEWAY} \
+ -seldonistioenable=${SUBMARINE_ISTIO_SELDON_ENABLE} \
+ -seldongateway=${SUBMARINE_ISTIO_SELDON_GATEWAY} \
+ -clustertype=${SUBMARINE_CLUSTER_TYPE:kubernetes} \
+ -createpsp=${SUBMARINE_POD_SECURITY_POLICY_ENABLE:-true}
diff --git a/submarine-cloud-v3/api/v1alpha1/groupversion_info.go b/submarine-cloud-v3/api/v1alpha1/groupversion_info.go
index bc4d119b..4fa738ed 100644
--- a/submarine-cloud-v3/api/v1alpha1/groupversion_info.go
+++ b/submarine-cloud-v3/api/v1alpha1/groupversion_info.go
@@ -1,18 +1,19 @@
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
// Package v1alpha1 contains API Schema definitions for the v1alpha1 API group
//+kubebuilder:object:generate=true
diff --git a/submarine-cloud-v3/api/v1alpha1/submarine_types.go b/submarine-cloud-v3/api/v1alpha1/submarine_types.go
index 5acc791f..3ad43720 100644
--- a/submarine-cloud-v3/api/v1alpha1/submarine_types.go
+++ b/submarine-cloud-v3/api/v1alpha1/submarine_types.go
@@ -1,22 +1,24 @@
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package v1alpha1
import (
+ corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
@@ -47,31 +49,35 @@ type SubmarineSpec struct {
// Database is the spec that defines the submarine database
Database *SubmarineDatabaseSpec `json:"database"`
// Virtualservice is the spec that defines the submarine virtualservice
- Virtualservice *SubmarineVirtualserviceSpec `json:"virtualservice"`
+ Virtualservice *SubmarineVirtualserviceSpec `json:"virtualservice,omitempty"`
// Tensorboard is the spec that defines the submarine tensorboard
Tensorboard *SubmarineTensorboardSpec `json:"tensorboard"`
// Mlflow is the spec that defines the submarine mlflow
Mlflow *SubmarineMlflowSpec `json:"mlflow"`
// Minio is the spec that defines the submarine minio
Minio *SubmarineMinioSpec `json:"minio"`
+ // Common is the spec that defines some submarine common configurations
+ Common *SubmarineCommon `json:"common,omitempty"`
}
// SubmarineServerSpec defines the desired submarine server
type SubmarineServerSpec struct {
// Image is the submarine server's docker image
- Image string `json:"image"`
+ Image string `json:"image,omitempty"`
// Replicas is the number of submarine server's replica
// +kubebuilder:validation:Minimum=1
Replicas *int32 `json:"replicas"`
+ // Envs is the extra environments that submarine server requires
+ Env []corev1.EnvVar `json:"env,omitempty"`
}
-// SubmarineServerSpec defines the desired submarine database
+// SubmarineDatabaseSpec defines the desired submarine database
type SubmarineDatabaseSpec struct {
// Image is the submarine database's docker image
- Image string `json:"image"`
+ Image string `json:"image,omitempty"`
// StorageSize is the storage size of the database
StorageSize string `json:"storageSize"`
- // MysqlRootPasswordSecret is the mysql root password secret
+ // MysqlRootPasswordSecret is the mysql root password secret, this secret need password key: MYSQL_ROOT_PASSWORD
MysqlRootPasswordSecret string `json:"mysqlRootPasswordSecret"`
}
@@ -83,28 +89,54 @@ type SubmarineVirtualserviceSpec struct {
Gateways []string `json:"gateways,omitempty"`
}
-// SubmarineServerSpec defines the desired submarine tensorboard
+// SubmarineTensorboardSpec defines the desired submarine tensorboard
type SubmarineTensorboardSpec struct {
+ // Image is the submarine tensorboard's docker image
+ Image string `json:"image,omitempty"`
// Enabled defines whether to enable tensorboard or not
Enabled *bool `json:"enabled"`
// StorageSize defines the storage size of tensorboard
StorageSize string `json:"storageSize"`
}
-// SubmarineServerSpec defines the desired submarine mlflow
+// SubmarineMlflowSpec defines the desired submarine mlflow
type SubmarineMlflowSpec struct {
+ // Image is the submarine mlflow's docker image
+ Image string `json:"image,omitempty"`
// Enabled defines whether to enable mlflow or not
Enabled *bool `json:"enabled"`
// StorageSize defines the storage size of mlflow
StorageSize string `json:"storageSize"`
}
-// SubmarineServerSpec defines the desired submarine minio
+// SubmarineMinioSpec defines the desired submarine minio
type SubmarineMinioSpec struct {
+ // Image is the submarine minio's docker image
+ Image string `json:"image,omitempty"`
// Enabled defines whether to enable minio or not
Enabled *bool `json:"enabled"`
// StorageSize defines the storage size of minio
StorageSize string `json:"storageSize"`
+ // AccessKey defines the access_key of minio
+ AccessKey string `json:"accessKey,omitempty"`
+ // SecretKey defines the secret_key of minio
+ SecretKey string `json:"secretKey,omitempty"`
+}
+
+// SubmarineCommon defines the observed common configuration
+type SubmarineCommon struct {
+ // Image is the basic image used in initcontainer in some submarine resources
+ Image CommonImage `json:"image"`
+}
+
+// CommonImage defines the observed common image info
+type CommonImage struct {
+ // McImage is the image used by the submarine service resource (server...) to check whether the minio is started in the initcontainer
+ McImage string `json:"mc,omitempty"`
+ // BusyboxImage is the image used by the submarine service resource (server...) to check whether the minio is started in the initcontainer
+ BusyboxImage string `json:"busybox,omitempty"`
+ // PullSecrets is a specified array of imagePullSecrets. This secrets must be manually created in the namespace.
+ PullSecrets []string `json:"pullSecrets,omitempty"`
}
// SubmarineStatus defines the observed state of Submarine
diff --git a/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go b/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go
index bd02163c..c09b18b3 100644
--- a/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go
+++ b/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go
@@ -2,29 +2,51 @@
// +build !ignore_autogenerated
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
// Code generated by controller-gen. DO NOT EDIT.
package v1alpha1
import (
+ "k8s.io/api/core/v1"
runtime "k8s.io/apimachinery/pkg/runtime"
)
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *CommonImage) DeepCopyInto(out *CommonImage) {
+ *out = *in
+ if in.PullSecrets != nil {
+ in, out := &in.PullSecrets, &out.PullSecrets
+ *out = make([]string, len(*in))
+ copy(*out, *in)
+ }
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new CommonImage.
+func (in *CommonImage) DeepCopy() *CommonImage {
+ if in == nil {
+ return nil
+ }
+ out := new(CommonImage)
+ in.DeepCopyInto(out)
+ return out
+}
+
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Submarine) DeepCopyInto(out *Submarine) {
*out = *in
@@ -52,6 +74,22 @@ func (in *Submarine) DeepCopyObject() runtime.Object {
return nil
}
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *SubmarineCommon) DeepCopyInto(out *SubmarineCommon) {
+ *out = *in
+ in.Image.DeepCopyInto(&out.Image)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SubmarineCommon.
+func (in *SubmarineCommon) DeepCopy() *SubmarineCommon {
+ if in == nil {
+ return nil
+ }
+ out := new(SubmarineCommon)
+ in.DeepCopyInto(out)
+ return out
+}
+
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *SubmarineDatabaseSpec) DeepCopyInto(out *SubmarineDatabaseSpec) {
*out = *in
@@ -147,6 +185,13 @@ func (in *SubmarineServerSpec) DeepCopyInto(out *SubmarineServerSpec) {
*out = new(int32)
**out = **in
}
+ if in.Env != nil {
+ in, out := &in.Env, &out.Env
+ *out = make([]v1.EnvVar, len(*in))
+ for i := range *in {
+ (*in)[i].DeepCopyInto(&(*out)[i])
+ }
+ }
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SubmarineServerSpec.
@@ -192,6 +237,11 @@ func (in *SubmarineSpec) DeepCopyInto(out *SubmarineSpec) {
*out = new(SubmarineMinioSpec)
(*in).DeepCopyInto(*out)
}
+ if in.Common != nil {
+ in, out := &in.Common, &out.Common
+ *out = new(SubmarineCommon)
+ (*in).DeepCopyInto(*out)
+ }
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SubmarineSpec.
diff --git a/submarine-cloud-v3/artifacts/submarine-database.yaml b/submarine-cloud-v3/artifacts/submarine-database.yaml
index 55d016bb..f5932df4 100644
--- a/submarine-cloud-v3/artifacts/submarine-database.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-database.yaml
@@ -68,7 +68,10 @@ spec:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
- value: "password"
+ valueFrom:
+ secretKeyRef:
+ name: submarine-database-secret
+ key: MYSQL_ROOT_PASSWORD
volumeMounts:
- mountPath: /var/lib/mysql
name: volume
@@ -80,3 +83,11 @@ spec:
- name: volume
persistentVolumeClaim:
claimName: submarine-database-pvc
+---
+apiVersion: v1
+kind: Secret
+metadata:
+ name: submarine-database-secret
+type: Opaque
+stringData:
+ MYSQL_ROOT_PASSWORD: "password"
diff --git a/submarine-cloud-v3/artifacts/submarine-minio.yaml b/submarine-cloud-v3/artifacts/submarine-minio.yaml
index 8cf62450..50ea488d 100644
--- a/submarine-cloud-v3/artifacts/submarine-minio.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-minio.yaml
@@ -54,6 +54,8 @@ spec:
metadata:
labels:
app: submarine-minio
+ annotations:
+ sidecar.istio.io/inject: "false"
spec:
serviceAccountName: "submarine-storage"
containers:
@@ -65,9 +67,15 @@ spec:
- /data
env:
- name: MINIO_ACCESS_KEY
- value: "submarine_minio"
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_ACCESS_KEY
- name: MINIO_SECRET_KEY
- value: "submarine_minio"
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_SECRET_KEY
ports:
- containerPort: 9000
volumeMounts:
@@ -78,3 +86,12 @@ spec:
- name: "volume"
persistentVolumeClaim:
claimName: "submarine-minio-pvc"
+---
+apiVersion: v1
+kind: Secret
+metadata:
+ name: submarine-minio-secret
+type: Opaque
+stringData:
+ MINIO_ACCESS_KEY: "submarine_minio"
+ MINIO_SECRET_KEY: "submarine_minio"
diff --git a/submarine-cloud-v3/artifacts/submarine-mlflow.yaml b/submarine-cloud-v3/artifacts/submarine-mlflow.yaml
index 71091016..34ab2f08 100644
--- a/submarine-cloud-v3/artifacts/submarine-mlflow.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-mlflow.yaml
@@ -50,6 +50,7 @@ spec:
selector:
matchLabels:
app: submarine-mlflow
+ replicas: 1
template:
metadata:
labels:
@@ -62,13 +63,13 @@ spec:
command: ["sh", "-c",
"until nc -z submarine-database 3306;
do echo waiting for database connection;
- sleep 20; done"]
+ sleep 10; done"]
- name: submarine-mlflow-initcontainer
image: "minio/mc"
command: ["/bin/bash", "-c",
"cnt=0;
- while ! /bin/bash -c 'mc --config-dir /root/.mc config host add minio http://submarine-minio-service:9000
- submarine_minio submarine_minio' 2>&1;
+ while ! /bin/bash -c 'mc --config-dir /tmp/.mc config host add minio http://submarine-minio-service:9000
+ ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}' 2>&1;
do
sleep 15;
((cnt=cnt+1));
@@ -77,14 +78,25 @@ spec:
exit 1;
fi;
done;
- if /bin/bash -c 'mc --config-dir /root/.mc ls minio/mlflow' >/dev/null 2>&1; then
+ if /bin/bash -c 'mc --config-dir /tmp/.mc ls minio/mlflow' >/dev/null 2>&1; then
echo 'Bucket minio/mlflow already exists, skipping creation.';
else
- /bin/bash -c 'mc --config-dir /root/.mc mb minio/mlflow';
+ /bin/bash -c 'mc --config-dir /tmp/.mc mb minio/mlflow';
fi;"]
volumeMounts:
- name: mc-config-vol
- mountPath: /root/.mc
+ mountPath: /tmp/.mc
+ env:
+ - name: MINIO_ACCESS_KEY
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_ACCESS_KEY
+ - name: MINIO_SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_SECRET_KEY
containers:
- name: submarine-mlflow-container
image: apache/submarine:mlflow-0.8.0-SNAPSHOT
diff --git a/submarine-serve/installation/seldon-secret.yaml b/submarine-cloud-v3/artifacts/submarine-serve.yaml
similarity index 86%
rename from submarine-serve/installation/seldon-secret.yaml
rename to submarine-cloud-v3/artifacts/submarine-serve.yaml
index 9d02dfed..4f27f2c1 100644
--- a/submarine-serve/installation/seldon-secret.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-serve.yaml
@@ -6,24 +6,26 @@
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+#
+---
apiVersion: v1
kind: Secret
metadata:
name: submarine-serve-secret
- # namespace: submarine-user-test
type: Opaque
stringData:
RCLONE_CONFIG_S3_TYPE: s3
RCLONE_CONFIG_S3_PROVIDER: minio
RCLONE_CONFIG_S3_ENV_AUTH: "false"
+ # These values are by default the same as the minio envs in submarine-cloud-v3/artifacts/submarine-minio.yaml
RCLONE_CONFIG_S3_ACCESS_KEY_ID: submarine_minio
RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: submarine_minio
RCLONE_CONFIG_S3_ENDPOINT: http://submarine-minio-service:9000
diff --git a/submarine-cloud-v3/artifacts/submarine-server-rbac.yaml b/submarine-cloud-v3/artifacts/submarine-server-rbac.yaml
index 9e9b67e5..c6f8a066 100644
--- a/submarine-cloud-v3/artifacts/submarine-server-rbac.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-server-rbac.yaml
@@ -21,17 +21,29 @@ kind: Role
metadata:
name: "submarine-server"
rules:
+# need to add submarines/finalizers
+# https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#ownerreferencespermissionenforcement
+- apiGroups:
+ - submarine.apache.org
+ resources:
+ - submarines/finalizers
+ verbs:
+ - "update"
- apiGroups:
- kubeflow.org
resources:
- tfjobs
- tfjobs/status
+ - tfjobs/finalizers
- pytorchjobs
- pytorchjobs/status
+ - pytorchjobs/finalizers
- xgboostjobs
- xgboostjobs/status
+ - xgboostjobs/finalizers
- notebooks
- notebooks/status
+ - notebooks/finalizers
verbs:
- get
- list
diff --git a/submarine-cloud-v3/artifacts/submarine-server.yaml b/submarine-cloud-v3/artifacts/submarine-server.yaml
index 85d7b8b0..5e65723e 100644
--- a/submarine-cloud-v3/artifacts/submarine-server.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-server.yaml
@@ -54,16 +54,21 @@ spec:
metadata:
labels:
app: "submarine-server"
-
spec:
serviceAccountName: "submarine-server"
initContainers:
+ - name: check-database-connection
+ image: busybox:1.28
+ command: [ "sh", "-c",
+ "until nc -z submarine-database 3306;
+ do echo waiting for database connection;
+ sleep 10; done" ]
- name: submarine-server-initcontainer
image: "minio/mc"
command: ["/bin/bash", "-c",
"cnt=0;
- while ! /bin/bash -c 'mc --config-dir /root/.mc config host add minio http://submarine-minio-service:9000
- submarine_minio submarine_minio' 2>&1;
+ while ! /bin/bash -c 'mc --config-dir /tmp/.mc config host add minio http://submarine-minio-service:9000
+ ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}' 2>&1;
do
sleep 15;
((cnt=cnt+1));
@@ -72,14 +77,25 @@ spec:
exit 1;
fi;
done;
- if /bin/bash -c 'mc --config-dir /root/.mc ls minio/submarine' >/dev/null 2>&1; then
+ if /bin/bash -c 'mc --config-dir /tmp/.mc ls minio/submarine' >/dev/null 2>&1; then
echo 'Bucket minio/submarine already exists, skipping creation.';
else
- /bin/bash -c 'mc --config-dir /root/.mc mb minio/submarine';
+ /bin/bash -c 'mc --config-dir /tmp/.mc mb minio/submarine';
fi;"]
volumeMounts:
- name: mc-config-vol
- mountPath: /root/.mc
+ mountPath: /tmp/.mc
+ env:
+ - name: MINIO_ACCESS_KEY
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_ACCESS_KEY
+ - name: MINIO_SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_SECRET_KEY
volumes:
- name: mc-config-vol
emptyDir: { }
@@ -92,7 +108,16 @@ spec:
value: "8080"
- name: K8S_APISERVER_URL
value: "kubernetes.default.svc"
-
+ - name: SUBMARINE_S3_ACCESS_KEY_ID
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_ACCESS_KEY
+ - name: SUBMARINE_S3_SECRET_ACCESS_KEY
+ valueFrom:
+ secretKeyRef:
+ name: submarine-minio-secret
+ key: MINIO_SECRET_KEY
image: "apache/submarine:server-0.8.0-SNAPSHOT"
imagePullPolicy: IfNotPresent
ports:
diff --git a/submarine-cloud-v3/artifacts/submarine-tensorboard.yaml b/submarine-cloud-v3/artifacts/submarine-tensorboard.yaml
index b108f8bc..069a2dd9 100644
--- a/submarine-cloud-v3/artifacts/submarine-tensorboard.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-tensorboard.yaml
@@ -49,6 +49,7 @@ spec:
selector:
matchLabels:
app: submarine-tensorboard
+ replicas: 1
template:
metadata:
labels:
diff --git a/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml b/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
index d9c7ad81..10abe051 100644
--- a/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
@@ -21,8 +21,10 @@ kind: VirtualService
metadata:
name: submarine-virtual-service
spec:
+ hosts:
+ - '*'
gateways:
- - submarine-cloud-v3-system/submarine-gateway
+ - submarine/submarine-gateway
http:
- match:
- uri:
diff --git a/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml b/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml
index 0cc6cf9f..1dd70ed1 100644
--- a/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml
+++ b/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml
@@ -52,6 +52,34 @@ spec:
spec:
description: SubmarineSpec defines the desired state of Submarine
properties:
+ common:
+ description: Common is the spec that defines some submarine common
+ configurations
+ properties:
+ image:
+ description: Image is the basic image used in initcontainer in
+ some submarine resources
+ properties:
+ busybox:
+ description: BusyboxImage is the image used by the submarine
+ service resource (server...) to check whether the minio
+ is started in the initcontainer
+ type: string
+ mc:
+ description: McImage is the image used by the submarine service
+ resource (server...) to check whether the minio is started
+ in the initcontainer
+ type: string
+ pullSecrets:
+ description: PullSecrets is a specified array of imagePullSecrets.
+ This secrets must be manually created in the namespace.
+ items:
+ type: string
+ type: array
+ type: object
+ required:
+ - image
+ type: object
database:
description: Database is the spec that defines the submarine database
properties:
@@ -59,23 +87,31 @@ spec:
description: Image is the submarine database's docker image
type: string
mysqlRootPasswordSecret:
- description: MysqlRootPasswordSecret is the mysql root password
- secret
+ description: 'MysqlRootPasswordSecret is the mysql root password
+ secret, this secret need password key: MYSQL_ROOT_PASSWORD'
type: string
storageSize:
description: StorageSize is the storage size of the database
type: string
required:
- - image
- mysqlRootPasswordSecret
- storageSize
type: object
minio:
description: Minio is the spec that defines the submarine minio
properties:
+ accessKey:
+ description: AccessKey defines the access_key of minio
+ type: string
enabled:
description: Enabled defines whether to enable minio or not
type: boolean
+ image:
+ description: Image is the submarine minio's docker image
+ type: string
+ secretKey:
+ description: SecretKey defines the secret_key of minio
+ type: string
storageSize:
description: StorageSize defines the storage size of minio
type: string
@@ -89,6 +125,9 @@ spec:
enabled:
description: Enabled defines whether to enable mlflow or not
type: boolean
+ image:
+ description: Image is the submarine mlflow's docker image
+ type: string
storageSize:
description: StorageSize defines the storage size of mlflow
type: string
@@ -99,6 +138,117 @@ spec:
server:
description: Server is the spec that defines the submarine server
properties:
+ env:
+ description: Envs is the extra environments that submarine server
+ requires
+ items:
+ description: EnvVar represents an environment variable present
+ in a Container.
+ properties:
+ name:
+ description: Name of the environment variable. Must be a
+ C_IDENTIFIER.
+ type: string
+ value:
+ description: 'Variable references $(VAR_NAME) are expanded
+ using the previously defined environment variables in
+ the container and any service environment variables. If
+ a variable cannot be resolved, the reference in the input
+ string will be unchanged. Double $$ are reduced to a single
+ $, which allows for escaping the $(VAR_NAME) syntax: i.e.
+ "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)".
+ Escaped references will never be expanded, regardless
+ of whether the variable exists or not. Defaults to "".'
+ type: string
+ valueFrom:
+ description: Source for the environment variable's value.
+ Cannot be used if value is not empty.
+ properties:
+ configMapKeyRef:
+ description: Selects a key of a ConfigMap.
+ properties:
+ key:
+ description: The key to select.
+ type: string
+ name:
+ description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+ TODO: Add other useful fields. apiVersion, kind,
+ uid?'
+ type: string
+ optional:
+ description: Specify whether the ConfigMap or its
+ key must be defined
+ type: boolean
+ required:
+ - key
+ type: object
+ fieldRef:
+ description: 'Selects a field of the pod: supports metadata.name,
+ metadata.namespace, `metadata.labels[''<KEY>'']`,
+ `metadata.annotations[''<KEY>'']`, spec.nodeName,
+ spec.serviceAccountName, status.hostIP, status.podIP,
+ status.podIPs.'
+ properties:
+ apiVersion:
+ description: Version of the schema the FieldPath
+ is written in terms of, defaults to "v1".
+ type: string
+ fieldPath:
+ description: Path of the field to select in the
+ specified API version.
+ type: string
+ required:
+ - fieldPath
+ type: object
+ resourceFieldRef:
+ description: 'Selects a resource of the container: only
+ resources limits and requests (limits.cpu, limits.memory,
+ limits.ephemeral-storage, requests.cpu, requests.memory
+ and requests.ephemeral-storage) are currently supported.'
+ properties:
+ containerName:
+ description: 'Container name: required for volumes,
+ optional for env vars'
+ type: string
+ divisor:
+ anyOf:
+ - type: integer
+ - type: string
+ description: Specifies the output format of the
+ exposed resources, defaults to "1"
+ pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+ x-kubernetes-int-or-string: true
+ resource:
+ description: 'Required: resource to select'
+ type: string
+ required:
+ - resource
+ type: object
+ secretKeyRef:
+ description: Selects a key of a secret in the pod's
+ namespace
+ properties:
+ key:
+ description: The key of the secret to select from. Must
+ be a valid secret key.
+ type: string
+ name:
+ description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+ TODO: Add other useful fields. apiVersion, kind,
+ uid?'
+ type: string
+ optional:
+ description: Specify whether the Secret or its key
+ must be defined
+ type: boolean
+ required:
+ - key
+ type: object
+ type: object
+ required:
+ - name
+ type: object
+ type: array
image:
description: Image is the submarine server's docker image
type: string
@@ -108,7 +258,6 @@ spec:
minimum: 1
type: integer
required:
- - image
- replicas
type: object
tensorboard:
@@ -118,6 +267,9 @@ spec:
description: Enabled defines whether to enable tensorboard or
not
type: boolean
+ image:
+ description: Image is the submarine tensorboard's docker image
+ type: string
storageSize:
description: StorageSize defines the storage size of tensorboard
type: string
@@ -151,7 +303,6 @@ spec:
- server
- tensorboard
- version
- - virtualservice
type: object
status:
description: SubmarineStatus defines the observed state of Submarine
diff --git a/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml b/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml
index 844f2afb..d8c914f9 100644
--- a/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml
+++ b/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml
@@ -27,12 +27,17 @@ spec:
database:
image: "apache/submarine:database-0.8.0-SNAPSHOT" # overwrite the image when development
storageSize: "1Gi"
- mysqlRootPasswordSecret: "root-pass-secret"
+ # mysqlRootPasswordSecret: ""
virtualservice:
- hosts:
- # - "yourHost" # configure when using custom hosts
- gateways:
- # - "yourNamespace/submarine-gateway" # configure when installing Helm in a different namespace
+ # configure when using custom hosts, default is "*"
+ hosts: []
+ # hosts:
+ # - "yourHost"
+
+ # configure when installing Helm in a different namespace, default is "submarine/submarine-gateway"
+ gateways: []
+ # gateways:
+ # - "yourNamespace/submarine-gateway"
tensorboard:
enabled: true
storageSize: "10Gi"
diff --git a/submarine-cloud-v3/controllers/submarine_controller.go b/submarine-cloud-v3/controllers/submarine_controller.go
index 4c1d29bf..c8e3303b 100644
--- a/submarine-cloud-v3/controllers/submarine_controller.go
+++ b/submarine-cloud-v3/controllers/submarine_controller.go
@@ -1,18 +1,19 @@
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package controllers
@@ -59,6 +60,7 @@ const (
artifactPath = "./artifacts/"
databaseYamlPath = artifactPath + "submarine-database.yaml"
minioYamlPath = artifactPath + "submarine-minio.yaml"
+ serveYamlPath = artifactPath + "submarine-serve.yaml"
mlflowYamlPath = artifactPath + "submarine-mlflow.yaml"
serverYamlPath = artifactPath + "submarine-server.yaml"
tensorboardYamlPath = artifactPath + "submarine-tensorboard.yaml"
@@ -66,6 +68,7 @@ const (
observerRbacYamlPath = artifactPath + "submarine-observer-rbac.yaml"
storageRbacYamlPath = artifactPath + "submarine-storage-rbac.yaml"
virtualServiceYamlPath = artifactPath + "submarine-virtualservice.yaml"
+ istioSidecarUid = 1337
)
// Name of deployments whose replica count and readiness need to be checked
@@ -112,6 +115,11 @@ type SubmarineReconciler struct {
Log logr.Logger
Recorder record.EventRecorder
// Fields required by submarine
+ IstioEnable bool
+ SubmarineGateway string
+ SeldonIstioEnable bool
+ SeldonGateway string
+ Namespace string
ClusterType string
CreatePodSecurityPolicy bool
}
@@ -197,7 +205,8 @@ func (r *SubmarineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (
submarineCopy.Status.State = submarineapacheorgv1alpha1.CreatingState
r.recordSubmarineEvent(submarineCopy)
}
- case submarineapacheorgv1alpha1.CreatingState:
+ // If an event is performed in a failed state, we also need to process it
+ case submarineapacheorgv1alpha1.CreatingState, submarineapacheorgv1alpha1.FailedState:
if err := r.createSubmarine(ctx, submarineCopy); err != nil {
submarineCopy.Status.State = submarineapacheorgv1alpha1.FailedState
submarineCopy.Status.ErrorMessage = err.Error()
@@ -211,6 +220,7 @@ func (r *SubmarineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (
}
if ok {
submarineCopy.Status.State = submarineapacheorgv1alpha1.RunningState
+ submarineCopy.Status.ErrorMessage = ""
r.recordSubmarineEvent(submarineCopy)
}
case submarineapacheorgv1alpha1.RunningState:
@@ -224,7 +234,9 @@ func (r *SubmarineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (
// Update STATUS of Submarine
err = r.updateSubmarineStatus(ctx, submarine, submarineCopy)
if err != nil {
- return ctrl.Result{}, err
+ submarineCopy.Status.State = submarineapacheorgv1alpha1.FailedState
+ submarineCopy.Status.ErrorMessage = err.Error()
+ r.recordSubmarineEvent(submarineCopy)
}
// Re-run Reconcile regularly
@@ -239,10 +251,12 @@ func (r *SubmarineReconciler) updateSubmarineStatus(ctx context.Context, submari
err := r.Get(ctx, types.NamespacedName{Name: serverName, Namespace: submarine.Namespace}, serverDeployment)
if err != nil {
if errors.IsNotFound(err) {
- submarineCopy.Status.AvailableServerReplicas = serverDeployment.Status.AvailableReplicas
+ submarineCopy.Status.AvailableServerReplicas = 0
} else {
return err
}
+ } else {
+ submarineCopy.Status.AvailableServerReplicas = serverDeployment.Status.AvailableReplicas
}
// Update database replicas
@@ -250,10 +264,12 @@ func (r *SubmarineReconciler) updateSubmarineStatus(ctx context.Context, submari
err = r.Get(ctx, types.NamespacedName{Name: databaseName, Namespace: submarine.Namespace}, statefulset)
if err != nil {
if errors.IsNotFound(err) {
- submarineCopy.Status.AvailableDatabaseReplicas = statefulset.Status.ReadyReplicas
+ submarineCopy.Status.AvailableDatabaseReplicas = 0
} else {
return err
}
+ } else {
+ submarineCopy.Status.AvailableDatabaseReplicas = statefulset.Status.ReadyReplicas
}
// Skip update if nothing changed.
@@ -331,6 +347,11 @@ func (r *SubmarineReconciler) createSubmarine(ctx context.Context, submarine *su
return err
}
+ err = r.createSubmarineServe(ctx, submarine)
+ if err != nil && !errors.IsAlreadyExists(err) {
+ return err
+ }
+
return nil
}
@@ -427,3 +448,12 @@ func (r *SubmarineReconciler) SetupWithManager(mgr ctrl.Manager) error {
For(&submarineapacheorgv1alpha1.Submarine{}).
Complete(r)
}
+
+// CreatePullSecrets will convert `submarine.spec.common.image.pullSecrets` to []`corev1.LocalObjectReference`
+func (r *SubmarineReconciler) CreatePullSecrets(pullSecrets *[]string) []corev1.LocalObjectReference {
+ secrets := make([]corev1.LocalObjectReference, 0)
+ for _, secret := range *pullSecrets {
+ secrets = append(secrets, corev1.LocalObjectReference{Name: secret})
+ }
+ return secrets
+}
diff --git a/submarine-cloud-v3/controllers/submarine_controller_test.go b/submarine-cloud-v3/controllers/submarine_controller_test.go
index 536bd98c..a17e56c3 100644
--- a/submarine-cloud-v3/controllers/submarine_controller_test.go
+++ b/submarine-cloud-v3/controllers/submarine_controller_test.go
@@ -1,31 +1,38 @@
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package controllers
import (
"context"
"fmt"
+ appsv1 "k8s.io/api/apps/v1"
+ "k8s.io/apimachinery/pkg/runtime"
+ "k8s.io/apimachinery/pkg/runtime/serializer"
+ clientgoscheme "k8s.io/client-go/kubernetes/scheme"
"os"
"path/filepath"
+ ctrl "sigs.k8s.io/controller-runtime"
"time"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"github.com/pkg/errors"
+ istio "istio.io/client-go/pkg/apis/networking/v1alpha3"
istiov1alpha3 "istio.io/client-go/pkg/apis/networking/v1alpha3"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
@@ -37,6 +44,62 @@ import (
submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
)
+func createScheme() *runtime.Scheme {
+ scheme := runtime.NewScheme()
+ _ = clientgoscheme.AddToScheme(scheme)
+ _ = appsv1.AddToScheme(scheme)
+ _ = corev1.AddToScheme(scheme)
+ _ = submarineapacheorgv1alpha1.AddToScheme(scheme)
+ _ = istio.AddToScheme(scheme)
+ _ = serializer.NewCodecFactory(scheme).UniversalDeserializer().Decode
+ return scheme
+}
+
+// We are mainly testing created resources, so for the time being we have not declared k8s client and recoder
+func createSubmarineReconciler(config ...*SubmarineReconciler) *SubmarineReconciler {
+ namespace := "submarine"
+ istioEnable := false
+ submarineGateway := "submarine/submarine-gateway"
+ seldonIstioEnable := false
+ seldonGateway := "submarine/seldon-gateway"
+ clusterType := "kubernetes"
+ createPodSecurityPolicy := false
+ if len(config) > 0 {
+ if config[0].Namespace != "" {
+ namespace = config[0].Namespace
+ }
+ if &config[0].IstioEnable != nil {
+ istioEnable = config[0].IstioEnable
+ }
+ if config[0].SubmarineGateway != "" {
+ submarineGateway = config[0].SubmarineGateway
+ }
+ if &config[0].SeldonIstioEnable != nil {
+ seldonIstioEnable = config[0].SeldonIstioEnable
+ }
+ if config[0].SeldonGateway != "" {
+ seldonGateway = config[0].SeldonGateway
+ }
+ if config[0].ClusterType != "" {
+ clusterType = config[0].ClusterType
+ }
+ if &config[0].CreatePodSecurityPolicy != nil {
+ createPodSecurityPolicy = config[0].CreatePodSecurityPolicy
+ }
+ }
+ return &SubmarineReconciler{
+ Scheme: createScheme(),
+ Log: ctrl.Log.WithName("submarine-test"),
+ Namespace: namespace,
+ IstioEnable: istioEnable,
+ SubmarineGateway: submarineGateway,
+ SeldonIstioEnable: seldonIstioEnable,
+ SeldonGateway: seldonGateway,
+ ClusterType: clusterType,
+ CreatePodSecurityPolicy: createPodSecurityPolicy,
+ }
+}
+
var _ = Describe("Submarine controller", func() {
// Define utility constants and variables
@@ -217,9 +280,9 @@ var _ = Describe("Submarine controller", func() {
err := k8sClient.Get(ctx, types.NamespacedName{Name: virtualServiceName, Namespace: submarineNamespaceDefaultCR}, createdVirtualService)
Expect(err).To(BeNil())
- // The default value for host is <submarine namespace>.submarine
- Expect(createdVirtualService.Spec.Hosts[0]).To(Equal(submarineNamespaceDefaultCR + ".submarine"))
- // The default value for gateway is submarine-cloud-v3-system/submarine-gateway
+ // The default value for host is *
+ Expect(createdVirtualService.Spec.Hosts[0]).To(Equal("*"))
+ // The default value for gateway is ${namespace}/submarine-gateway
Expect(createdVirtualService.Spec.Gateways[0]).To(Equal("submarine-cloud-v3-system/submarine-gateway"))
})
It(fmt.Sprintf("Hosts and Gateways should have custom values In %s", submarineNamespaceCustomCR), func() {
@@ -338,6 +401,15 @@ func MakeSubmarineFromYaml(pathToYaml string) (*submarineapacheorgv1alpha1.Subma
return &tmp, err
}
+func MakeSubmarineFromYamlByNamespace(pathToYaml string, namespace string) (*submarineapacheorgv1alpha1.Submarine, error) {
+ submarine, err := MakeSubmarineFromYaml(pathToYaml)
+ if err != nil {
+ return nil, err
+ }
+ submarine.Namespace = namespace
+ return submarine, nil
+}
+
// PathToOSFile gets the absolute path from relative path.
func PathToOSFile(relativePath string) (*os.File, error) {
path, err := filepath.Abs(relativePath)
diff --git a/submarine-cloud-v3/controllers/submarine_database.go b/submarine-cloud-v3/controllers/submarine_database.go
index 91e48aa8..9e83f643 100644
--- a/submarine-cloud-v3/controllers/submarine_database.go
+++ b/submarine-cloud-v3/controllers/submarine_database.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineDatabasePersistentVolumeClaim(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.PersistentVolumeClaim {
- pvc, err := ParsePersistentVolumeClaimYaml(databaseYamlPath)
+ pvc, err := util.ParsePersistentVolumeClaimYaml(databaseYamlPath)
if err != nil {
r.Log.Error(err, "ParsePersistentVolumeClaimYaml")
}
@@ -45,8 +46,22 @@ func (r *SubmarineReconciler) newSubmarineDatabasePersistentVolumeClaim(ctx cont
return pvc
}
+// newSubmarineDatabaseSecret is a function to create database secret which stores mysql root password
+func (r *SubmarineReconciler) newSubmarineDatabaseSecret(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Secret {
+ secret, err := util.ParseSecretYaml(databaseYamlPath)
+ if err != nil {
+ r.Log.Error(err, "ParseSecretYaml")
+ }
+ secret.Namespace = submarine.Namespace
+ err = controllerutil.SetControllerReference(submarine, secret, r.Scheme)
+ if err != nil {
+ r.Log.Error(err, "Set Secret ControllerReference")
+ }
+ return secret
+}
+
func (r *SubmarineReconciler) newSubmarineDatabaseStatefulSet(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *appsv1.StatefulSet {
- statefulset, err := ParseStatefulSetYaml(databaseYamlPath)
+ statefulset, err := util.ParseStatefulSetYaml(databaseYamlPath)
if err != nil {
r.Log.Error(err, "ParseStatefulSetYaml")
}
@@ -57,16 +72,28 @@ func (r *SubmarineReconciler) newSubmarineDatabaseStatefulSet(ctx context.Contex
r.Log.Error(err, "Set Stateful Set ControllerReference")
}
+ // database image
databaseImage := submarine.Spec.Database.Image
if databaseImage != "" {
statefulset.Spec.Template.Spec.Containers[0].Image = databaseImage
+ } else {
+ statefulset.Spec.Template.Spec.Containers[0].Image = fmt.Sprintf("apache/submarine:database-%s", submarine.Spec.Version)
+ }
+ // pull secrets
+ pullSecrets := util.GetSubmarineCommonImage(submarine).PullSecrets
+ if pullSecrets != nil {
+ statefulset.Spec.Template.Spec.ImagePullSecrets = r.CreatePullSecrets(&pullSecrets)
+ }
+ // password secret
+ if submarine.Spec.Database.MysqlRootPasswordSecret != "" {
+ statefulset.Spec.Template.Spec.Containers[0].Env[0].ValueFrom.SecretKeyRef.Name = submarine.Spec.Database.MysqlRootPasswordSecret
}
return statefulset
}
func (r *SubmarineReconciler) newSubmarineDatabaseService(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Service {
- service, err := ParseServiceYaml(databaseYamlPath)
+ service, err := util.ParseServiceYaml(databaseYamlPath)
if err != nil {
r.Log.Error(err, "ParseServiceYaml")
}
@@ -106,7 +133,41 @@ func (r *SubmarineReconciler) createSubmarineDatabase(ctx context.Context, subma
return fmt.Errorf(msg)
}
- // Step 2: Create Statefulset
+ // Step 2: Create Secret
+ if submarine.Spec.Database.MysqlRootPasswordSecret == "" {
+ secret := &corev1.Secret{}
+ err = r.Get(ctx, types.NamespacedName{Name: "submarine-database-secret", Namespace: submarine.Namespace}, secret)
+ // If the resource doesn't exist, we'll create it
+ if errors.IsNotFound(err) {
+ secret = r.newSubmarineDatabaseSecret(ctx, submarine)
+ err = r.Create(ctx, secret)
+ r.Log.Info("Create Database Secret", "name", secret.Name)
+ } else {
+ newSecret := r.newSubmarineDatabaseSecret(ctx, submarine)
+ // compare if there are same
+ if !util.CompareSecret(secret, newSecret) {
+ // update meta with uid
+ newSecret.ObjectMeta = secret.ObjectMeta
+ err = r.Update(ctx, newSecret)
+ r.Log.Info("Update Database Secret", "name", secret.Name)
+ }
+ }
+
+ // If an error occurs during Get/Create, we'll requeue the item so we can
+ // attempt processing again later. This could have been caused by a
+ // temporary network failure, or any other transient reason.
+ if err != nil {
+ return err
+ }
+
+ if !metav1.IsControlledBy(secret, submarine) {
+ msg := fmt.Sprintf(MessageResourceExists, secret.Name)
+ r.Recorder.Event(submarine, corev1.EventTypeWarning, ErrResourceExists, msg)
+ return fmt.Errorf(msg)
+ }
+ }
+
+ // Step 3: Create Statefulset
statefulset := &appsv1.StatefulSet{}
err = r.Get(ctx, types.NamespacedName{Name: databaseName, Namespace: submarine.Namespace}, statefulset)
// If the resource doesn't exist, we'll create it
@@ -114,6 +175,15 @@ func (r *SubmarineReconciler) createSubmarineDatabase(ctx context.Context, subma
statefulset = r.newSubmarineDatabaseStatefulSet(ctx, submarine)
err = r.Create(ctx, statefulset)
r.Log.Info("Create StatefulSet", "name", statefulset.Name)
+ } else {
+ newStatefulset := r.newSubmarineDatabaseStatefulSet(ctx, submarine)
+ // compare if there are same
+ if !r.compareDatabaseStatefulset(statefulset, newStatefulset) {
+ // update meta with uid
+ newStatefulset.ObjectMeta = statefulset.ObjectMeta
+ err = r.Update(ctx, newStatefulset)
+ r.Log.Info("Update StatefulSet", "name", newStatefulset.Name)
+ }
}
// If an error occurs during Get/Create, we'll requeue the item so we can
@@ -133,7 +203,7 @@ func (r *SubmarineReconciler) createSubmarineDatabase(ctx context.Context, subma
return err
}
- // Step 3: Create Service
+ // Step 4: Create Service
service := &corev1.Service{}
err = r.Get(ctx, types.NamespacedName{Name: databaseName, Namespace: submarine.Namespace}, service)
// If the resource doesn't exist, we'll create it
@@ -158,3 +228,33 @@ func (r *SubmarineReconciler) createSubmarineDatabase(ctx context.Context, subma
return nil
}
+
+// compareDatabaseStatefulset will determine if two StatefulSets are equal
+func (r *SubmarineReconciler) compareDatabaseStatefulset(oldStatefulset, newStatefulse *appsv1.StatefulSet) bool {
+ // spec.replicas
+ if *oldStatefulset.Spec.Replicas != *newStatefulse.Spec.Replicas {
+ return false
+ }
+
+ if len(oldStatefulset.Spec.Template.Spec.Containers) != 1 {
+ return false
+ }
+ // spec.template.spec.containers[0].env
+ if !util.CompareEnv(oldStatefulset.Spec.Template.Spec.Containers[0].Env,
+ newStatefulse.Spec.Template.Spec.Containers[0].Env) {
+ return false
+ }
+ // spec.template.spec.containers[0].image
+ if oldStatefulset.Spec.Template.Spec.Containers[0].Image !=
+ newStatefulse.Spec.Template.Spec.Containers[0].Image {
+ return false
+ }
+
+ // spec.template.spec.imagePullSecrets
+ if !util.ComparePullSecrets(oldStatefulset.Spec.Template.Spec.ImagePullSecrets,
+ newStatefulse.Spec.Template.Spec.ImagePullSecrets) {
+ return false
+ }
+
+ return true
+}
diff --git a/submarine-cloud-v3/controllers/submarine_database_test.go b/submarine-cloud-v3/controllers/submarine_database_test.go
new file mode 100644
index 00000000..1d5a3dc0
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_database_test.go
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package controllers
+
+import (
+ "context"
+ "testing"
+
+ . "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ . "github.com/onsi/gomega"
+)
+
+func TestSubmarineDatabase(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler(&SubmarineReconciler{Namespace: "submarine"})
+ submarine, err := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+ g.Expect(err).To(BeNil())
+
+ ArtifactBasePath = "../"
+ statefulset1 := r.newSubmarineDatabaseStatefulSet(context.TODO(), submarine)
+ g.Expect(statefulset1).NotTo(BeNil())
+
+ // change secret
+ submarine.Spec.Database.MysqlRootPasswordSecret = "mysql-password-secret"
+ statefulset2 := r.newSubmarineDatabaseStatefulSet(context.TODO(), submarine)
+ g.Expect(statefulset2.Spec.Template.Spec.Containers[0].Env[0].ValueFrom.SecretKeyRef.Name).To(Equal("mysql-password-secret"))
+
+ // compare
+ g.Expect(r.compareDatabaseStatefulset(statefulset1, statefulset2)).To(Equal(false))
+}
diff --git a/submarine-cloud-v3/controllers/submarine_minio.go b/submarine-cloud-v3/controllers/submarine_minio.go
index b5d5e8d6..ec886738 100644
--- a/submarine-cloud-v3/controllers/submarine_minio.go
+++ b/submarine-cloud-v3/controllers/submarine_minio.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineMinioPersistentVolumeClaim(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.PersistentVolumeClaim {
- pvc, err := ParsePersistentVolumeClaimYaml(minioYamlPath)
+ pvc, err := util.ParsePersistentVolumeClaimYaml(minioYamlPath)
if err != nil {
r.Log.Error(err, "ParsePersistentVolumeClaimYaml")
}
@@ -46,7 +47,7 @@ func (r *SubmarineReconciler) newSubmarineMinioPersistentVolumeClaim(ctx context
}
func (r *SubmarineReconciler) newSubmarineMinioDeployment(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *appsv1.Deployment {
- deployment, err := ParseDeploymentYaml(minioYamlPath)
+ deployment, err := util.ParseDeploymentYaml(minioYamlPath)
if err != nil {
r.Log.Error(err, "ParseDeploymentYaml")
}
@@ -55,11 +56,23 @@ func (r *SubmarineReconciler) newSubmarineMinioDeployment(ctx context.Context, s
if err != nil {
r.Log.Error(err, "Set Deployment ControllerReference")
}
+
+ // minio/mc image
+ minoImage := submarine.Spec.Minio.Image
+ if minoImage != "" {
+ deployment.Spec.Template.Spec.Containers[0].Image = minoImage
+ }
+ // pull secrets
+ pullSecrets := util.GetSubmarineCommonImage(submarine).PullSecrets
+ if pullSecrets != nil {
+ deployment.Spec.Template.Spec.ImagePullSecrets = r.CreatePullSecrets(&pullSecrets)
+ }
+
return deployment
}
func (r *SubmarineReconciler) newSubmarineMinioService(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Service {
- service, err := ParseServiceYaml(minioYamlPath)
+ service, err := util.ParseServiceYaml(minioYamlPath)
if err != nil {
r.Log.Error(err, "ParseServiceYaml")
}
@@ -71,6 +84,31 @@ func (r *SubmarineReconciler) newSubmarineMinioService(ctx context.Context, subm
return service
}
+// newSubmarineSeldonSecret is a function to create seldon secret which stores minio connection configurations
+func (r *SubmarineReconciler) newSubmarineMinoSecret(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Secret {
+ secret, err := util.ParseSecretYaml(minioYamlPath)
+ if err != nil {
+ r.Log.Error(err, "ParseSecretYaml")
+ }
+ secret.Namespace = submarine.Namespace
+ err = controllerutil.SetControllerReference(submarine, secret, r.Scheme)
+ if err != nil {
+ r.Log.Error(err, "Set Secret ControllerReference")
+ }
+
+ // access_ey and secret_key
+ accessKey := submarine.Spec.Minio.AccessKey
+ if accessKey != "" {
+ secret.StringData["MINIO_ACCESS_KEY"] = accessKey
+ }
+ secretKey := submarine.Spec.Minio.SecretKey
+ if secretKey != "" {
+ secret.StringData["MINIO_SECRET_KEY"] = secretKey
+ }
+
+ return secret
+}
+
// createSubmarineMinio is a function to create submarine-minio.
// Reference: https://github.com/apache/submarine/blob/master/submarine-cloud-v3/artifacts/submarine-minio.yaml
func (r *SubmarineReconciler) createSubmarineMinio(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) error {
@@ -99,7 +137,39 @@ func (r *SubmarineReconciler) createSubmarineMinio(ctx context.Context, submarin
return fmt.Errorf(msg)
}
- // Step 2: Create Deployment
+ // Step 2: Create Minio Secret
+ secret := &corev1.Secret{}
+ err = r.Get(ctx, types.NamespacedName{Name: "submarine-minio-secret", Namespace: submarine.Namespace}, secret)
+ // If the resource doesn't exist, we'll create it
+ if errors.IsNotFound(err) {
+ secret = r.newSubmarineMinoSecret(ctx, submarine)
+ err = r.Create(ctx, secret)
+ r.Log.Info("Create Minio Secret", "name", secret.Name)
+ } else {
+ newSecret := r.newSubmarineMinoSecret(ctx, submarine)
+ // compare if there are same
+ if !util.CompareSecret(secret, newSecret) {
+ // update meta with uid
+ newSecret.ObjectMeta = secret.ObjectMeta
+ err = r.Update(ctx, newSecret)
+ r.Log.Info("Update Minio Secret", "name", secret.Name)
+ }
+ }
+
+ // If an error occurs during Get/Create, we'll requeue the item so we can
+ // attempt processing again later. This could have been caused by a
+ // temporary network failure, or any other transient reason.
+ if err != nil {
+ return err
+ }
+
+ if !metav1.IsControlledBy(secret, submarine) {
+ msg := fmt.Sprintf(MessageResourceExists, secret.Name)
+ r.Recorder.Event(submarine, corev1.EventTypeWarning, ErrResourceExists, msg)
+ return fmt.Errorf(msg)
+ }
+
+ // Step 3: Create Deployment
deployment := &appsv1.Deployment{}
err = r.Get(ctx, types.NamespacedName{Name: minioName, Namespace: submarine.Namespace}, deployment)
if errors.IsNotFound(err) {
@@ -121,7 +191,7 @@ func (r *SubmarineReconciler) createSubmarineMinio(ctx context.Context, submarin
return fmt.Errorf(msg)
}
- // Step 3: Create Service
+ // Step 4: Create Service
service := &corev1.Service{}
err = r.Get(ctx, types.NamespacedName{Name: minioServiceName, Namespace: submarine.Namespace}, service)
// If the resource doesn't exist, we'll create it
diff --git a/submarine-cloud-v3/controllers/submarine_minio_test.go b/submarine-cloud-v3/controllers/submarine_minio_test.go
new file mode 100644
index 00000000..e02bf7f4
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_minio_test.go
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package controllers
+
+import (
+ "context"
+ "testing"
+
+ . "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ . "github.com/onsi/gomega"
+)
+
+func TestSubmarineMinio(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler()
+ submarine, err := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+ g.Expect(err).To(BeNil())
+
+ ArtifactBasePath = "../"
+ secret1 := r.newSubmarineMinoSecret(context.TODO(), submarine)
+ g.Expect(secret1).NotTo(BeNil())
+
+ // change secret
+ submarine.Spec.Minio.AccessKey = "submarine_minio1"
+ submarine.Spec.Minio.SecretKey = "submarine_minio2"
+ secret2 := r.newSubmarineMinoSecret(context.TODO(), submarine)
+ g.Expect(secret2.StringData["MINIO_ACCESS_KEY"]).To(Equal("submarine_minio1"))
+ g.Expect(secret2.StringData["MINIO_SECRET_KEY"]).To(Equal("submarine_minio2"))
+
+ // compare
+ g.Expect(CompareSecret(secret1, secret2)).To(Equal(false))
+}
diff --git a/submarine-cloud-v3/controllers/submarine_mlflow.go b/submarine-cloud-v3/controllers/submarine_mlflow.go
index 5a345776..495fc27e 100644
--- a/submarine-cloud-v3/controllers/submarine_mlflow.go
+++ b/submarine-cloud-v3/controllers/submarine_mlflow.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineMlflowPersistentVolumeClaim(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.PersistentVolumeClaim {
- pvc, err := ParsePersistentVolumeClaimYaml(mlflowYamlPath)
+ pvc, err := util.ParsePersistentVolumeClaimYaml(mlflowYamlPath)
if err != nil {
r.Log.Error(err, "ParsePersistentVolumeClaimYaml")
}
@@ -46,7 +47,7 @@ func (r *SubmarineReconciler) newSubmarineMlflowPersistentVolumeClaim(ctx contex
}
func (r *SubmarineReconciler) newSubmarineMlflowDeployment(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *appsv1.Deployment {
- deployment, err := ParseDeploymentYaml(mlflowYamlPath)
+ deployment, err := util.ParseDeploymentYaml(mlflowYamlPath)
if err != nil {
r.Log.Error(err, "ParseDeploymentYaml")
}
@@ -55,11 +56,44 @@ func (r *SubmarineReconciler) newSubmarineMlflowDeployment(ctx context.Context,
if err != nil {
r.Log.Error(err, "Set Deployment ControllerReference")
}
+
+ // mlflow image
+ mlflowImage := submarine.Spec.Mlflow.Image
+ if mlflowImage != "" {
+ deployment.Spec.Template.Spec.Containers[0].Image = mlflowImage
+ } else {
+ deployment.Spec.Template.Spec.Containers[0].Image = fmt.Sprintf("apache/submarine:mlflow-%s", submarine.Spec.Version)
+ }
+ commonImage := util.GetSubmarineCommonImage(submarine)
+ // busybox image
+ busyboxImage := commonImage.BusyboxImage
+ if busyboxImage != "" {
+ deployment.Spec.Template.Spec.InitContainers[0].Image = busyboxImage
+ }
+ // minio/mc image
+ mcImage := commonImage.McImage
+ if mcImage != "" {
+ deployment.Spec.Template.Spec.InitContainers[1].Image = mcImage
+ }
+ // pull secrets
+ pullSecrets := commonImage.PullSecrets
+ if pullSecrets != nil {
+ deployment.Spec.Template.Spec.ImagePullSecrets = r.CreatePullSecrets(&pullSecrets)
+ }
+
+ // If support istio in openshift, we need to add securityContext to pod in to avoid traffic error
+ if r.ClusterType == "openshift" && r.SeldonIstioEnable {
+ initcontainers := deployment.Spec.Template.Spec.InitContainers
+ for i := range initcontainers {
+ initcontainers[i].SecurityContext = util.CreateIstioSidecarSecurityContext(istioSidecarUid)
+ }
+ }
+
return deployment
}
func (r *SubmarineReconciler) newSubmarineMlflowService(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Service {
- service, err := ParseServiceYaml(mlflowYamlPath)
+ service, err := util.ParseServiceYaml(mlflowYamlPath)
if err != nil {
r.Log.Error(err, "ParseServiceYaml")
}
@@ -106,6 +140,15 @@ func (r *SubmarineReconciler) createSubmarineMlflow(ctx context.Context, submari
deployment = r.newSubmarineMlflowDeployment(ctx, submarine)
err = r.Create(ctx, deployment)
r.Log.Info("Create Deployment", "name", deployment.Name)
+ } else {
+ newDeployment := r.newSubmarineMlflowDeployment(ctx, submarine)
+ // compare if there are same
+ if !r.compareMlflowDeployment(deployment, newDeployment) {
+ // update meta with uid
+ newDeployment.ObjectMeta = deployment.ObjectMeta
+ err = r.Update(ctx, newDeployment)
+ r.Log.Info("Update Deployment", "name", deployment.Name)
+ }
}
// If an error occurs during Get/Create, we'll requeue the item so we can
@@ -149,3 +192,62 @@ func (r *SubmarineReconciler) createSubmarineMlflow(ctx context.Context, submari
return nil
}
+
+// compareMlflowDeployment will determine if two Deployments are equal
+func (r *SubmarineReconciler) compareMlflowDeployment(oldDeployment, newDeployment *appsv1.Deployment) bool {
+ // spec.replicas
+ if *oldDeployment.Spec.Replicas != *newDeployment.Spec.Replicas {
+ return false
+ }
+
+ if len(oldDeployment.Spec.Template.Spec.Containers) != 1 {
+ return false
+ }
+ // spec.template.spec.containers[0].env
+ if !util.CompareEnv(oldDeployment.Spec.Template.Spec.Containers[0].Env,
+ newDeployment.Spec.Template.Spec.Containers[0].Env) {
+ return false
+ }
+ // spec.template.spec.containers[0].image
+ if oldDeployment.Spec.Template.Spec.Containers[0].Image !=
+ newDeployment.Spec.Template.Spec.Containers[0].Image {
+ return false
+ }
+
+ if len(oldDeployment.Spec.Template.Spec.InitContainers) != 2 || len(newDeployment.Spec.Template.Spec.InitContainers) != 2 {
+ return false
+ }
+ for index, old := range oldDeployment.Spec.Template.Spec.InitContainers {
+ container := newDeployment.Spec.Template.Spec.InitContainers[index]
+ // spec.template.spec.initContainers.image
+ if old.Image != container.Image {
+ return false
+ }
+ // spec.template.spec.initContainers.command
+ if !util.CompareSlice(old.Command, container.Command) {
+ return false
+ }
+ // spec.template.spec.initContainers.SecurityContext
+ if r.ClusterType == "openshift" && r.SeldonIstioEnable {
+ sc := old.SecurityContext
+ if sc == nil {
+ return false
+ } else {
+ if util.CompareInt64(sc.RunAsUser, container.SecurityContext.RunAsUser) {
+ return false
+ }
+ if util.CompareInt64(sc.RunAsGroup, container.SecurityContext.RunAsGroup) {
+ return false
+ }
+ }
+ }
+ }
+
+ // spec.template.spec.imagePullSecrets
+ if !util.ComparePullSecrets(oldDeployment.Spec.Template.Spec.ImagePullSecrets,
+ newDeployment.Spec.Template.Spec.ImagePullSecrets) {
+ return false
+ }
+
+ return true
+}
diff --git a/submarine-cloud-v3/controllers/submarine_mlflow_test.go b/submarine-cloud-v3/controllers/submarine_mlflow_test.go
new file mode 100644
index 00000000..b06fbf8a
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_mlflow_test.go
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package controllers
+
+import (
+ "context"
+ submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
+ "testing"
+
+ . "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ . "github.com/onsi/gomega"
+)
+
+func TestSubmarineMlflow(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler()
+ submarine, err := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+ g.Expect(err).To(BeNil())
+
+ ArtifactBasePath = "../"
+ deployment1 := r.newSubmarineMlflowDeployment(context.TODO(), submarine)
+ g.Expect(deployment1).NotTo(BeNil())
+ g.Expect(deployment1.Spec.Template.Spec.Containers[0].Image).To(Equal("apache/submarine:mlflow-" + submarine.Spec.Version))
+
+ // test change params
+ submarine.Spec.Mlflow.Image = "harbor.com/apache/submarine/mlflow-" + submarine.Spec.Version
+ submarine.Spec.Common = &submarineapacheorgv1alpha1.SubmarineCommon{
+ Image: submarineapacheorgv1alpha1.CommonImage{
+ McImage: "harbor.com/minio/mc",
+ BusyboxImage: "harbor.com/busybox:1.28",
+ PullSecrets: []string{"pull-secret"},
+ },
+ }
+ deployment2 := r.newSubmarineMlflowDeployment(context.TODO(), submarine)
+ g.Expect(deployment2.Spec.Template.Spec.Containers[0].Image).To(Equal("harbor.com/apache/submarine/mlflow-" + submarine.Spec.Version))
+ g.Expect(deployment2.Spec.Template.Spec.InitContainers[0].Image).To(Equal("harbor.com/busybox:1.28"))
+ g.Expect(deployment2.Spec.Template.Spec.InitContainers[1].Image).To(Equal("harbor.com/minio/mc"))
+ g.Expect(deployment2.Spec.Template.Spec.ImagePullSecrets[0].Name).To(Equal("pull-secret"))
+
+ // test compare
+ g.Expect(r.compareMlflowDeployment(deployment1, deployment2)).To(Equal(false))
+}
+
+func TestSubmarineMlflowOpenshift(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler(&SubmarineReconciler{SeldonIstioEnable: true, ClusterType: "openshift"})
+ submarine, _ := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+
+ ArtifactBasePath = "../"
+ deployment := r.newSubmarineMlflowDeployment(context.TODO(), submarine)
+ g.Expect(deployment).NotTo(BeNil())
+ g.Expect(*deployment.Spec.Template.Spec.InitContainers[0].SecurityContext.RunAsUser).To(Equal(int64(istioSidecarUid)))
+ g.Expect(*deployment.Spec.Template.Spec.InitContainers[1].SecurityContext.RunAsUser).To(Equal(int64(istioSidecarUid)))
+}
diff --git a/submarine-cloud-v3/controllers/submarine_observer_rbac.go b/submarine-cloud-v3/controllers/submarine_observer_rbac.go
index b1126b7b..9a36fedf 100644
--- a/submarine-cloud-v3/controllers/submarine_observer_rbac.go
+++ b/submarine-cloud-v3/controllers/submarine_observer_rbac.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
corev1 "k8s.io/api/core/v1"
rbacv1 "k8s.io/api/rbac/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineObserverRole(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *rbacv1.Role {
- role, err := ParseRoleYaml(observerRbacYamlPath)
+ role, err := util.ParseRoleYaml(observerRbacYamlPath)
if err != nil {
r.Log.Error(err, "ParseRoleYaml")
}
@@ -46,7 +47,7 @@ func (r *SubmarineReconciler) newSubmarineObserverRole(ctx context.Context, subm
}
func (r *SubmarineReconciler) newSubmarineObserverRoleBinding(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *rbacv1.RoleBinding {
- roleBinding, err := ParseRoleBindingYaml(observerRbacYamlPath)
+ roleBinding, err := util.ParseRoleBindingYaml(observerRbacYamlPath)
if err != nil {
r.Log.Error(err, "Set RoleBinding ControllerReference")
}
diff --git a/submarine-cloud-v3/controllers/submarine_serve.go b/submarine-cloud-v3/controllers/submarine_serve.go
new file mode 100644
index 00000000..53f358d7
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_serve.go
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package controllers
+
+import (
+ "context"
+ "fmt"
+ submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ corev1 "k8s.io/api/core/v1"
+ "k8s.io/apimachinery/pkg/api/errors"
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/apimachinery/pkg/types"
+
+ "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
+)
+
+// newSubmarineSeldonSecret is a function to create seldon secret which stores minio connection configurations
+func (r *SubmarineReconciler) newSubmarineSeldonSecret(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Secret {
+ secret, err := util.ParseSecretYaml(serveYamlPath)
+ if err != nil {
+ r.Log.Error(err, "ParseSecretYaml")
+ }
+ secret.Namespace = submarine.Namespace
+ err = controllerutil.SetControllerReference(submarine, secret, r.Scheme)
+ if err != nil {
+ r.Log.Error(err, "Set Secret ControllerReference")
+ }
+
+ // access_ey and secret_key
+ accessKey := submarine.Spec.Minio.AccessKey
+ if accessKey != "" {
+ secret.StringData["RCLONE_CONFIG_S3_ACCESS_KEY_ID"] = accessKey
+ }
+ secretKey := submarine.Spec.Minio.SecretKey
+ if secretKey != "" {
+ secret.StringData["RCLONE_CONFIG_S3_SECRET_ACCESS_KEY"] = secretKey
+ }
+
+ return secret
+}
+
+// createSubmarineServe is a function to create submarine-serve.
+// Reference: https://github.com/apache/submarine/blob/master/submarine-cloud-v3/artifacts/submarine-serve.yaml
+func (r *SubmarineReconciler) createSubmarineServe(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) error {
+ r.Log.Info("Enter createSubmarineServe")
+
+ // Step 1: Create Seldon Secret
+ secret := &corev1.Secret{}
+ err := r.Get(ctx, types.NamespacedName{Name: "submarine-serve-secret", Namespace: submarine.Namespace}, secret)
+ // If the resource doesn't exist, we'll create it
+ if errors.IsNotFound(err) {
+ secret = r.newSubmarineSeldonSecret(ctx, submarine)
+ err = r.Create(ctx, secret)
+ r.Log.Info("Create Seldon Secret", "name", secret.Name)
+ } else {
+ newSecret := r.newSubmarineSeldonSecret(ctx, submarine)
+ // compare if there are same
+ if !util.CompareSecret(secret, newSecret) {
+ // update meta with uid
+ newSecret.ObjectMeta = secret.ObjectMeta
+ err = r.Update(ctx, secret)
+ r.Log.Info("Update Seldon Secret", "name", secret.Name)
+ }
+ }
+
+ // If an error occurs during Get/Create, we'll requeue the item so we can
+ // attempt processing again later. This could have been caused by a
+ // temporary network failure, or any other transient reason.
+ if err != nil {
+ return err
+ }
+
+ if !metav1.IsControlledBy(secret, submarine) {
+ msg := fmt.Sprintf(MessageResourceExists, secret.Name)
+ r.Recorder.Event(submarine, corev1.EventTypeWarning, ErrResourceExists, msg)
+ return fmt.Errorf(msg)
+ }
+
+ return nil
+}
diff --git a/submarine-cloud-v3/controllers/submarine_serve_test.go b/submarine-cloud-v3/controllers/submarine_serve_test.go
new file mode 100644
index 00000000..cc2c1721
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_serve_test.go
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package controllers
+
+import (
+ "context"
+ "testing"
+
+ . "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ . "github.com/onsi/gomega"
+)
+
+func TestSubmarineServe(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler()
+ submarine, err := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+ g.Expect(err).To(BeNil())
+
+ ArtifactBasePath = "../"
+ secret1 := r.newSubmarineSeldonSecret(context.TODO(), submarine)
+ g.Expect(secret1).NotTo(BeNil())
+
+ // change secret
+ submarine.Spec.Minio.AccessKey = "submarine_minio1"
+ submarine.Spec.Minio.SecretKey = "submarine_minio2"
+ secret2 := r.newSubmarineSeldonSecret(context.TODO(), submarine)
+ g.Expect(secret2.StringData["RCLONE_CONFIG_S3_ACCESS_KEY_ID"]).To(Equal("submarine_minio1"))
+ g.Expect(secret2.StringData["RCLONE_CONFIG_S3_SECRET_ACCESS_KEY"]).To(Equal("submarine_minio2"))
+
+ // compare
+ g.Expect(CompareSecret(secret1, secret2)).To(Equal(false))
+}
diff --git a/submarine-cloud-v3/controllers/submarine_server.go b/submarine-cloud-v3/controllers/submarine_server.go
index f02e13ec..51acaa9f 100644
--- a/submarine-cloud-v3/controllers/submarine_server.go
+++ b/submarine-cloud-v3/controllers/submarine_server.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineServerServiceAccount(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.ServiceAccount {
- serviceAccount, err := ParseServiceAccountYaml(serverYamlPath)
+ serviceAccount, err := util.ParseServiceAccountYaml(serverYamlPath)
if err != nil {
r.Log.Error(err, "ParseServiceAccountYaml")
}
@@ -46,7 +47,7 @@ func (r *SubmarineReconciler) newSubmarineServerServiceAccount(ctx context.Conte
}
func (r *SubmarineReconciler) newSubmarineServerService(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Service {
- service, err := ParseServiceYaml(serverYamlPath)
+ service, err := util.ParseServiceYaml(serverYamlPath)
if err != nil {
r.Log.Error(err, "ParseServiceYaml")
}
@@ -85,9 +86,22 @@ func (r *SubmarineReconciler) newSubmarineServerDeployment(ctx context.Context,
Name: "SUBMARINE_UID",
Value: string(submarine.UID),
},
+ {
+ Name: "SUBMARINE_ISTIO_SELDON_GATEWAY",
+ Value: r.SeldonGateway,
+ },
+ {
+ Name: "SUBMARINE_ISTIO_SUBMARINE_GATEWAY",
+ Value: r.SubmarineGateway,
+ },
+ }
+ // extra envs
+ extraEnv := submarine.Spec.Server.Env
+ if extraEnv != nil {
+ operatorEnv = append(operatorEnv, extraEnv...)
}
- deployment, err := ParseDeploymentYaml(serverYamlPath)
+ deployment, err := util.ParseDeploymentYaml(serverYamlPath)
if err != nil {
r.Log.Error(err, "ParseDeploymentYaml")
}
@@ -99,6 +113,38 @@ func (r *SubmarineReconciler) newSubmarineServerDeployment(ctx context.Context,
deployment.Spec.Replicas = &serverReplicas
deployment.Spec.Template.Spec.Containers[0].Env = append(deployment.Spec.Template.Spec.Containers[0].Env, operatorEnv...)
+ // server image
+ serverImage := submarine.Spec.Server.Image
+ if serverImage != "" {
+ deployment.Spec.Template.Spec.Containers[0].Image = serverImage
+ } else {
+ deployment.Spec.Template.Spec.Containers[0].Image = fmt.Sprintf("apache/submarine:server-%s", submarine.Spec.Version)
+ }
+ commonImage := util.GetSubmarineCommonImage(submarine)
+ // busybox image
+ busyboxImage := commonImage.BusyboxImage
+ if busyboxImage != "" {
+ deployment.Spec.Template.Spec.InitContainers[0].Image = busyboxImage
+ }
+ // minio/mc image
+ mcImage := commonImage.McImage
+ if mcImage != "" {
+ deployment.Spec.Template.Spec.InitContainers[1].Image = mcImage
+ }
+ // pull secrets
+ pullSecrets := commonImage.PullSecrets
+ if pullSecrets != nil {
+ deployment.Spec.Template.Spec.ImagePullSecrets = r.CreatePullSecrets(&pullSecrets)
+ }
+
+ // If support istio in openshift, we need to add securityContext to pod in to avoid traffic error
+ if r.ClusterType == "openshift" && r.SeldonIstioEnable {
+ initcontainers := deployment.Spec.Template.Spec.InitContainers
+ for i := range initcontainers {
+ initcontainers[i].SecurityContext = util.CreateIstioSidecarSecurityContext(istioSidecarUid)
+ }
+ }
+
return deployment
}
@@ -165,6 +211,15 @@ func (r *SubmarineReconciler) createSubmarineServer(ctx context.Context, submari
deployment = r.newSubmarineServerDeployment(ctx, submarine)
err = r.Create(ctx, deployment)
r.Log.Info("Create Deployment", "name", deployment.Name)
+ } else {
+ newDeployment := r.newSubmarineServerDeployment(ctx, submarine)
+ // compare if there are same
+ if !r.compareServerDeployment(deployment, newDeployment) {
+ // update meta with uid
+ newDeployment.ObjectMeta = deployment.ObjectMeta
+ err = r.Update(ctx, newDeployment)
+ r.Log.Info("Update Deployment", "name", deployment.Name)
+ }
}
// If an error occurs during Get/Create, we'll requeue the item so we can
@@ -183,18 +238,64 @@ func (r *SubmarineReconciler) createSubmarineServer(ctx context.Context, submari
return fmt.Errorf(msg)
}
- // Update the replicas of the server deployment if it is not equal to spec
- if submarine.Spec.Server.Replicas != nil && *submarine.Spec.Server.Replicas != *deployment.Spec.Replicas {
- msg := fmt.Sprintf("Submarine %s server spec replicas", submarine.Name)
- r.Log.Info(msg, "server spec", *submarine.Spec.Server.Replicas, "actual", *deployment.Spec.Replicas)
+ return nil
+}
+
+// compareServerDeployment will determine if two Deployments are equal
+func (r *SubmarineReconciler) compareServerDeployment(oldDeployment, newDeployment *appsv1.Deployment) bool {
+ // spec.replicas
+ if *oldDeployment.Spec.Replicas != *newDeployment.Spec.Replicas {
+ return false
+ }
+
+ if len(oldDeployment.Spec.Template.Spec.Containers) != 1 {
+ return false
+ }
+ // spec.template.spec.containers[0].env
+ if !util.CompareEnv(oldDeployment.Spec.Template.Spec.Containers[0].Env,
+ newDeployment.Spec.Template.Spec.Containers[0].Env) {
+ return false
+ }
+ // spec.template.spec.containers[0].image
+ if oldDeployment.Spec.Template.Spec.Containers[0].Image !=
+ newDeployment.Spec.Template.Spec.Containers[0].Image {
+ return false
+ }
- deployment = r.newSubmarineServerDeployment(ctx, submarine)
- err = r.Update(ctx, deployment)
+ if len(oldDeployment.Spec.Template.Spec.InitContainers) != 2 || len(newDeployment.Spec.Template.Spec.InitContainers) != 2 {
+ return false
+ }
+ for index, old := range oldDeployment.Spec.Template.Spec.InitContainers {
+ // spec.template.spec.initContainers.image
+ container := newDeployment.Spec.Template.Spec.InitContainers[index]
+ if old.Image != container.Image {
+ return false
+ }
+ // spec.template.spec.initContainers.command
+ if !util.CompareSlice(old.Command, container.Command) {
+ return false
+ }
+ // spec.template.spec.initContainers.SecurityContext
+ if r.ClusterType == "openshift" && r.SeldonIstioEnable {
+ sc := container.SecurityContext
+ if sc == nil {
+ return false
+ } else {
+ if util.CompareInt64(sc.RunAsUser, container.SecurityContext.RunAsUser) {
+ return false
+ }
+ if util.CompareInt64(sc.RunAsGroup, container.SecurityContext.RunAsGroup) {
+ return false
+ }
+ }
+ }
}
- if err != nil {
- return err
+ // spec.template.spec.imagePullSecrets
+ if !util.ComparePullSecrets(oldDeployment.Spec.Template.Spec.ImagePullSecrets,
+ newDeployment.Spec.Template.Spec.ImagePullSecrets) {
+ return false
}
- return nil
+ return true
}
diff --git a/submarine-cloud-v3/controllers/submarine_server_rbac.go b/submarine-cloud-v3/controllers/submarine_server_rbac.go
index 65336121..0871ea76 100644
--- a/submarine-cloud-v3/controllers/submarine_server_rbac.go
+++ b/submarine-cloud-v3/controllers/submarine_server_rbac.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
corev1 "k8s.io/api/core/v1"
rbacv1 "k8s.io/api/rbac/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineServerRole(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *rbacv1.Role {
- role, err := ParseRoleYaml(serverRbacYamlPath)
+ role, err := util.ParseRoleYaml(serverRbacYamlPath)
if err != nil {
r.Log.Error(err, "ParseRoleYaml")
}
@@ -56,7 +57,7 @@ func (r *SubmarineReconciler) newSubmarineServerRole(ctx context.Context, submar
}
func (r *SubmarineReconciler) newSubmarineServerRoleBinding(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *rbacv1.RoleBinding {
- roleBinding, err := ParseRoleBindingYaml(serverRbacYamlPath)
+ roleBinding, err := util.ParseRoleBindingYaml(serverRbacYamlPath)
if err != nil {
r.Log.Error(err, "Set RoleBinding ControllerReference")
}
diff --git a/submarine-cloud-v3/controllers/submarine_server_test.go b/submarine-cloud-v3/controllers/submarine_server_test.go
new file mode 100644
index 00000000..a7508863
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_server_test.go
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package controllers
+
+import (
+ "context"
+ submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
+ v1 "k8s.io/api/core/v1"
+ "testing"
+
+ . "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ . "github.com/onsi/gomega"
+)
+
+func TestSubmarineServer(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler()
+ submarine, err := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+ g.Expect(err).To(BeNil())
+
+ ArtifactBasePath = "../"
+ submarine.UID = "adfd95a4-b363-4b58-b0cf-3b8c67b18a29"
+ deployment1 := r.newSubmarineServerDeployment(context.TODO(), submarine)
+ g.Expect(deployment1).NotTo(BeNil())
+ g.Expect(deployment1.Spec.Template.Spec.Containers[0].Image).To(Equal("apache/submarine:server-" + submarine.Spec.Version))
+ g.Expect(deployment1.Spec.Template.Spec.Containers[0].Env).To(ContainElement(v1.EnvVar{Name: "SUBMARINE_UID", Value: "adfd95a4-b363-4b58-b0cf-3b8c67b18a29"}))
+ g.Expect(deployment1.Spec.Template.Spec.Containers[0].Env).To(ContainElement(v1.EnvVar{Name: "SUBMARINE_ISTIO_SELDON_GATEWAY", Value: r.SeldonGateway}))
+ g.Expect(deployment1.Spec.Template.Spec.Containers[0].Env).To(ContainElement(v1.EnvVar{Name: "SUBMARINE_ISTIO_SUBMARINE_GATEWAY", Value: r.SubmarineGateway}))
+
+ // test change params
+ submarine.Spec.Server.Image = "harbor.com/apache/submarine/server-" + submarine.Spec.Version
+ submarine.Spec.Common = &submarineapacheorgv1alpha1.SubmarineCommon{
+ Image: submarineapacheorgv1alpha1.CommonImage{
+ McImage: "harbor.com/minio/mc",
+ BusyboxImage: "harbor.com/busybox:1.28",
+ PullSecrets: []string{"pull-secret"},
+ },
+ }
+ deployment2 := r.newSubmarineServerDeployment(context.TODO(), submarine)
+ g.Expect(deployment2.Spec.Template.Spec.Containers[0].Image).To(Equal("harbor.com/apache/submarine/server-" + submarine.Spec.Version))
+ g.Expect(deployment2.Spec.Template.Spec.InitContainers[0].Image).To(Equal("harbor.com/busybox:1.28"))
+ g.Expect(deployment2.Spec.Template.Spec.InitContainers[1].Image).To(Equal("harbor.com/minio/mc"))
+ g.Expect(deployment2.Spec.Template.Spec.ImagePullSecrets[0].Name).To(Equal("pull-secret"))
+
+ // test compare
+ g.Expect(r.compareServerDeployment(deployment1, deployment2)).To(Equal(false))
+}
+
+func TestSubmarineServerOpenshift(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler(&SubmarineReconciler{SeldonIstioEnable: true, ClusterType: "openshift"})
+ submarine, _ := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+
+ ArtifactBasePath = "../"
+ deployment := r.newSubmarineServerDeployment(context.TODO(), submarine)
+ g.Expect(deployment).NotTo(BeNil())
+ g.Expect(*deployment.Spec.Template.Spec.InitContainers[0].SecurityContext.RunAsUser).To(Equal(int64(istioSidecarUid)))
+ g.Expect(*deployment.Spec.Template.Spec.InitContainers[1].SecurityContext.RunAsUser).To(Equal(int64(istioSidecarUid)))
+}
diff --git a/submarine-cloud-v3/controllers/submarine_storage_rbac.go b/submarine-cloud-v3/controllers/submarine_storage_rbac.go
index 7a6d2234..768bed1c 100644
--- a/submarine-cloud-v3/controllers/submarine_storage_rbac.go
+++ b/submarine-cloud-v3/controllers/submarine_storage_rbac.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
corev1 "k8s.io/api/core/v1"
rbacv1 "k8s.io/api/rbac/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineStorageRole(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *rbacv1.Role {
- role, err := ParseRoleYaml(storageRbacYamlPath)
+ role, err := util.ParseRoleYaml(storageRbacYamlPath)
if err != nil {
r.Log.Error(err, "ParseRoleYaml")
}
@@ -54,7 +55,7 @@ func (r *SubmarineReconciler) newSubmarineStorageRole(ctx context.Context, subma
}
func (r *SubmarineReconciler) newSubmarineStorageRoleBinding(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *rbacv1.RoleBinding {
- roleBinding, err := ParseRoleBindingYaml(storageRbacYamlPath)
+ roleBinding, err := util.ParseRoleBindingYaml(storageRbacYamlPath)
if err != nil {
r.Log.Error(err, "Set RoleBinding ControllerReference")
}
@@ -67,7 +68,7 @@ func (r *SubmarineReconciler) newSubmarineStorageRoleBinding(ctx context.Context
}
func (r *SubmarineReconciler) newSubmarineStorageServiceAccount(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.ServiceAccount {
- serviceAccount, err := ParseServiceAccountYaml(storageRbacYamlPath)
+ serviceAccount, err := util.ParseServiceAccountYaml(storageRbacYamlPath)
if err != nil {
r.Log.Error(err, "ParseServiceAccountYaml")
}
diff --git a/submarine-cloud-v3/controllers/submarine_tensorboard.go b/submarine-cloud-v3/controllers/submarine_tensorboard.go
index 898740fd..4f7bebb1 100644
--- a/submarine-cloud-v3/controllers/submarine_tensorboard.go
+++ b/submarine-cloud-v3/controllers/submarine_tensorboard.go
@@ -20,6 +20,7 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
@@ -33,7 +34,7 @@ import (
)
func (r *SubmarineReconciler) newSubmarineTensorboardPersistentVolumeClaim(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.PersistentVolumeClaim {
- pvc, err := ParsePersistentVolumeClaimYaml(tensorboardYamlPath)
+ pvc, err := util.ParsePersistentVolumeClaimYaml(tensorboardYamlPath)
if err != nil {
r.Log.Error(err, "ParsePersistentVolumeClaimYaml")
}
@@ -46,7 +47,7 @@ func (r *SubmarineReconciler) newSubmarineTensorboardPersistentVolumeClaim(ctx c
}
func (r *SubmarineReconciler) newSubmarineTensorboardDeployment(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *appsv1.Deployment {
- deployment, err := ParseDeploymentYaml(tensorboardYamlPath)
+ deployment, err := util.ParseDeploymentYaml(tensorboardYamlPath)
if err != nil {
r.Log.Error(err, "ParseDeploymentYaml")
}
@@ -55,11 +56,23 @@ func (r *SubmarineReconciler) newSubmarineTensorboardDeployment(ctx context.Cont
if err != nil {
r.Log.Error(err, "Set Deployment ControllerReference")
}
+
+ // tensorboard image
+ tensorboardImage := submarine.Spec.Tensorboard.Image
+ if tensorboardImage != "" {
+ deployment.Spec.Template.Spec.Containers[0].Image = tensorboardImage
+ }
+ // pull secrets
+ pullSecrets := util.GetSubmarineCommonImage(submarine).PullSecrets
+ if pullSecrets != nil {
+ deployment.Spec.Template.Spec.ImagePullSecrets = r.CreatePullSecrets(&pullSecrets)
+ }
+
return deployment
}
func (r *SubmarineReconciler) newSubmarineTensorboardService(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *corev1.Service {
- service, err := ParseServiceYaml(tensorboardYamlPath)
+ service, err := util.ParseServiceYaml(tensorboardYamlPath)
if err != nil {
r.Log.Error(err, "ParseServiceYaml")
}
@@ -109,6 +122,15 @@ func (r *SubmarineReconciler) createSubmarineTensorboard(ctx context.Context, su
deployment = r.newSubmarineTensorboardDeployment(ctx, submarine)
err = r.Create(ctx, deployment)
r.Log.Info("Create Deployment", "name", deployment.Name)
+ } else {
+ newDeployment := r.newSubmarineTensorboardDeployment(ctx, submarine)
+ // compare if there are same
+ if !r.compareTensorboardDeployment(deployment, newDeployment) {
+ // update meta with uid
+ newDeployment.ObjectMeta = deployment.ObjectMeta
+ err = r.Update(ctx, newDeployment)
+ r.Log.Info("Update Deployment", "name", deployment.Name)
+ }
}
// If an error occurs during Get/Create, we'll requeue the item so we can
@@ -149,3 +171,28 @@ func (r *SubmarineReconciler) createSubmarineTensorboard(ctx context.Context, su
return nil
}
+
+// compareTensorboardDeployment will determine if two Deployments are equal
+func (r *SubmarineReconciler) compareTensorboardDeployment(oldDeployment, newDeployment *appsv1.Deployment) bool {
+ // spec.replicas
+ if *oldDeployment.Spec.Replicas != *newDeployment.Spec.Replicas {
+ return false
+ }
+
+ if len(oldDeployment.Spec.Template.Spec.Containers) != 1 {
+ return false
+ }
+ // spec.template.spec.containers[0].image
+ if oldDeployment.Spec.Template.Spec.Containers[0].Image !=
+ newDeployment.Spec.Template.Spec.Containers[0].Image {
+ return false
+ }
+
+ // spec.template.spec.imagePullSecrets
+ if !util.ComparePullSecrets(oldDeployment.Spec.Template.Spec.ImagePullSecrets,
+ newDeployment.Spec.Template.Spec.ImagePullSecrets) {
+ return false
+ }
+
+ return true
+}
diff --git a/submarine-cloud-v3/controllers/submarine_tensorboard_test.go b/submarine-cloud-v3/controllers/submarine_tensorboard_test.go
new file mode 100644
index 00000000..357c49ea
--- /dev/null
+++ b/submarine-cloud-v3/controllers/submarine_tensorboard_test.go
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package controllers
+
+import (
+ "context"
+ submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
+ "testing"
+
+ . "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ . "github.com/onsi/gomega"
+)
+
+func TestSubmarineTensorboard(t *testing.T) {
+ g := NewGomegaWithT(t)
+ r := createSubmarineReconciler()
+ submarine, err := MakeSubmarineFromYamlByNamespace("../config/samples/_v1alpha1_submarine.yaml", "submarine")
+ g.Expect(err).To(BeNil())
+
+ ArtifactBasePath = "../"
+ deployment1 := r.newSubmarineTensorboardDeployment(context.TODO(), submarine)
+ g.Expect(deployment1).NotTo(BeNil())
+
+ // test change params
+ submarine.Spec.Tensorboard.Image = "harbor.com/tensorflow/tensorflow:1.11.0"
+ submarine.Spec.Common = &submarineapacheorgv1alpha1.SubmarineCommon{
+ Image: submarineapacheorgv1alpha1.CommonImage{
+ PullSecrets: []string{"pull-secret"},
+ },
+ }
+ deployment2 := r.newSubmarineTensorboardDeployment(context.TODO(), submarine)
+ g.Expect(deployment2.Spec.Template.Spec.Containers[0].Image).To(Equal("harbor.com/tensorflow/tensorflow:1.11.0"))
+ g.Expect(deployment2.Spec.Template.Spec.ImagePullSecrets[0].Name).To(Equal("pull-secret"))
+
+ // test compare
+ g.Expect(r.compareTensorboardDeployment(deployment1, deployment2)).To(Equal(false))
+}
diff --git a/submarine-cloud-v3/controllers/submarine_virtualservice.go b/submarine-cloud-v3/controllers/submarine_virtualservice.go
index b15a0d44..ea729cb8 100644
--- a/submarine-cloud-v3/controllers/submarine_virtualservice.go
+++ b/submarine-cloud-v3/controllers/submarine_virtualservice.go
@@ -20,6 +20,8 @@ package controllers
import (
"context"
"fmt"
+ "github.com/apache/submarine/submarine-cloud-v3/controllers/util"
+ "reflect"
istiov1alpha3 "istio.io/client-go/pkg/apis/networking/v1alpha3"
@@ -34,25 +36,30 @@ import (
)
func (r *SubmarineReconciler) newSubmarineVirtualService(ctx context.Context, submarine *submarineapacheorgv1alpha1.Submarine) *istiov1alpha3.VirtualService {
- virtualService, err := ParseVirtualService(virtualServiceYamlPath)
+ virtualService, err := util.ParseVirtualService(virtualServiceYamlPath)
if err != nil {
r.Log.Error(err, "ParseVirtualService")
}
virtualService.Namespace = submarine.Namespace
- virtualserviceHosts := submarine.Spec.Virtualservice.Hosts
- if virtualserviceHosts != nil {
- // Use `Hosts` defined in submarine spec
- virtualService.Spec.Hosts = virtualserviceHosts
+ // Virtualservice is optional
+ // so that if user makes a declaration, we need to modify the configuration
+ specVirtual := submarine.Spec.Virtualservice
+ if specVirtual != nil {
+ virtualserviceHosts := specVirtual.Hosts
+ if virtualserviceHosts != nil {
+ // Use `Hosts` defined in submarine spec
+ virtualService.Spec.Hosts = virtualserviceHosts
+ }
+ virtualserviceGateways := specVirtual.Gateways
+ if virtualserviceGateways != nil {
+ // Use `Gateways` defined in submarine spec
+ virtualService.Spec.Gateways = virtualserviceGateways
+ } else {
+ virtualService.Spec.Gateways[0] = fmt.Sprintf("%s/submarine-gateway", r.Namespace)
+ }
} else {
- // Use default `<namespace>.submarine`
- virtualService.Spec.Hosts = append(virtualService.Spec.Hosts, submarine.Namespace+".submarine")
- }
-
- virtualserviceGateways := submarine.Spec.Virtualservice.Gateways
- if virtualserviceGateways != nil {
- // Use `Gateways` defined in submarine spec
- virtualService.Spec.Gateways = virtualserviceGateways
+ virtualService.Spec.Gateways[0] = fmt.Sprintf("%s/submarine-gateway", r.Namespace)
}
err = controllerutil.SetControllerReference(submarine, virtualService, r.Scheme)
@@ -74,6 +81,15 @@ func (r *SubmarineReconciler) createVirtualService(ctx context.Context, submarin
virtualService = r.newSubmarineVirtualService(ctx, submarine)
err = r.Create(ctx, virtualService)
r.Log.Info("Create VirtualService", "name", virtualService.Name)
+ } else {
+ newVirtualService := r.newSubmarineVirtualService(ctx, submarine)
+ // compare if there are same
+ if !CompareVirtualService(virtualService, newVirtualService) {
+ // update meta with uid
+ newVirtualService.ObjectMeta = virtualService.ObjectMeta
+ err = r.Update(ctx, newVirtualService)
+ r.Log.Info("Update VirtualService", "name", virtualService.Name)
+ }
}
// If an error occurs during Get/Create, we'll requeue the item so we can
@@ -91,3 +107,20 @@ func (r *SubmarineReconciler) createVirtualService(ctx context.Context, submarin
return nil
}
+
+// CompareVirtualService will determine if two VirtualServices are equal
+func CompareVirtualService(oldVirtualService, newVirtualService *istiov1alpha3.VirtualService) bool {
+ // spec.hosts
+ if !util.CompareSlice(oldVirtualService.Spec.Hosts, newVirtualService.Spec.Hosts) {
+ return false
+ }
+ // spec.gateways
+ if !util.CompareSlice(oldVirtualService.Spec.Gateways, newVirtualService.Spec.Gateways) {
+ return false
+ }
+ // spec.http
+ if !reflect.DeepEqual(oldVirtualService.Spec.Http, newVirtualService.Spec.Http) {
+ return false
+ }
+ return true
+}
diff --git a/submarine-cloud-v3/controllers/suite_test.go b/submarine-cloud-v3/controllers/suite_test.go
index 39b4f485..cfac5109 100644
--- a/submarine-cloud-v3/controllers/suite_test.go
+++ b/submarine-cloud-v3/controllers/suite_test.go
@@ -1,18 +1,19 @@
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package controllers
diff --git a/submarine-cloud-v3/controllers/parser.go b/submarine-cloud-v3/controllers/util/parser.go
similarity index 85%
rename from submarine-cloud-v3/controllers/parser.go
rename to submarine-cloud-v3/controllers/util/parser.go
index d1231c69..c0a0b84b 100644
--- a/submarine-cloud-v3/controllers/parser.go
+++ b/submarine-cloud-v3/controllers/util/parser.go
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package controllers
+package util
import (
"encoding/json"
@@ -23,6 +23,7 @@ import (
"io"
"os"
"path/filepath"
+ "strings"
"github.com/pkg/errors"
istiov1alpha3 "istio.io/client-go/pkg/apis/networking/v1alpha3"
@@ -33,6 +34,12 @@ import (
"k8s.io/apimachinery/pkg/util/yaml"
)
+const artifactBasePathEnv = "ARTIFACT_BASE_PATH"
+
+// ArtifactBasePath is default submarine path,
+// We can use it instead of the yaml path to solve the relative path problem when testing
+var ArtifactBasePath = os.Getenv(artifactBasePathEnv)
+
// PathToOSFile turn the file at the relativePath into a type of *os.File.
func pathToOSFile(relativePath string) (*os.File, error) {
path, err := filepath.Abs(relativePath)
@@ -54,6 +61,13 @@ func parseYaml(relativePath, kind string) ([]byte, error) {
var err error
var marshaled []byte
+ if ArtifactBasePath != "" {
+ if strings.HasSuffix(ArtifactBasePath, "/") {
+ relativePath = ArtifactBasePath + relativePath
+ } else {
+ relativePath = strings.Join([]string{ArtifactBasePath, relativePath}, "/")
+ }
+ }
if manifest, err = pathToOSFile(relativePath); err != nil {
return nil, err
}
@@ -79,7 +93,7 @@ func parseYaml(relativePath, kind string) ([]byte, error) {
return marshaled, nil
}
-// ParseServiceAccount parse ServiceAccount from yaml file.
+// ParseServiceAccountYaml parse ServiceAccount from yaml file.
func ParseServiceAccountYaml(relativePath string) (*v1.ServiceAccount, error) {
var serviceAccount v1.ServiceAccount
marshaled, err := parseYaml(relativePath, "ServiceAccount")
@@ -166,3 +180,14 @@ func ParseVirtualService(relativePath string) (*istiov1alpha3.VirtualService, er
json.Unmarshal(marshaled, &virtualService)
return &virtualService, nil
}
+
+// ParseSecretYaml parse Secret from yaml file.
+func ParseSecretYaml(relativePath string) (*v1.Secret, error) {
+ var secret v1.Secret
+ marshaled, err := parseYaml(relativePath, "Secret")
+ if err != nil {
+ return nil, err
+ }
+ json.Unmarshal(marshaled, &secret)
+ return &secret, nil
+}
diff --git a/submarine-cloud-v3/controllers/util/util.go b/submarine-cloud-v3/controllers/util/util.go
new file mode 100644
index 00000000..6a1d079a
--- /dev/null
+++ b/submarine-cloud-v3/controllers/util/util.go
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package util
+
+import (
+ submarineapacheorgv1alpha1 "github.com/apache/submarine/submarine-cloud-v3/api/v1alpha1"
+ corev1 "k8s.io/api/core/v1"
+ "reflect"
+)
+
+// GetSubmarineCommon will get `spec.common` and initialize it if it is nil to prevent NPE
+func GetSubmarineCommon(submarine *submarineapacheorgv1alpha1.Submarine) *submarineapacheorgv1alpha1.SubmarineCommon {
+ common := submarine.Spec.Common
+ if common == nil {
+ common = &submarineapacheorgv1alpha1.SubmarineCommon{}
+ }
+ return common
+}
+
+// GetSubmarineCommonImage will get `spec.common.image` and initialize it if it is nil to prevent NPE
+func GetSubmarineCommonImage(submarine *submarineapacheorgv1alpha1.Submarine) *submarineapacheorgv1alpha1.CommonImage {
+ common := GetSubmarineCommon(submarine)
+ image := &common.Image
+ if image == nil {
+ image = &submarineapacheorgv1alpha1.CommonImage{}
+ }
+ return image
+}
+
+// CompareSlice will determine if two slices are equal
+func CompareSlice(a, b []string) bool {
+ // If one is nil, the other must also be nil.
+ if (a == nil) != (b == nil) {
+ return false
+ }
+ if len(a) != len(b) {
+ return false
+ }
+ for key, value := range a {
+ if value != b[key] {
+ return false
+ }
+ }
+ return true
+}
+
+// CompareMap will determine if two maps are equal
+func CompareMap(a, b map[string]string) bool {
+ // If one is nil, the other must also be nil.
+ if (a == nil) != (b == nil) {
+ return false
+ }
+ if len(a) != len(b) {
+ return false
+ }
+ for k, v := range a {
+ if w, ok := b[k]; !ok || v != w {
+ return false
+ }
+ }
+ return true
+}
+
+// ComparePullSecrets will determine if two LocalObjectReferences are equal
+func ComparePullSecrets(a, b []corev1.LocalObjectReference) bool {
+ // If one is nil, the other must also be nil.
+ if (a == nil) != (b == nil) {
+ return false
+ }
+ if len(a) != len(b) {
+ return false
+ }
+ for key, value := range a {
+ if value.Name != b[key].Name {
+ return false
+ }
+ }
+ return true
+}
+
+// CompareEnv will determine if two EnvVars are equal
+func CompareEnv(a, b []corev1.EnvVar) bool {
+ // If one is nil, the other must also be nil.
+ if (a == nil) != (b == nil) {
+ return false
+ }
+ if len(a) != len(b) {
+ return false
+ }
+ for key, value := range a {
+ cv := b[key]
+ if value.Name != cv.Name || value.Value != cv.Value || !reflect.DeepEqual(value.ValueFrom, cv.ValueFrom) {
+ return false
+ }
+ }
+ return true
+}
+
+// CompareInt64 will determine if two int64 are equal
+func CompareInt64(a, b *int64) bool {
+ if (a == nil) != (b == nil) {
+ return false
+ }
+ if *a != *b {
+ return false
+ }
+ return true
+}
+
+// GetSecretData will return secret data ( map[string]string )
+func GetSecretData(secret *corev1.Secret) map[string]string {
+ if secret.StringData != nil {
+ return secret.StringData
+ } else {
+ var parsedData map[string]string
+ parsedData = make(map[string]string)
+ for key, value := range secret.Data {
+ parsedData[key] = string(value)
+ }
+ return parsedData
+ }
+}
+
+// CompareSecret will determine if two Secrets are equal
+func CompareSecret(oldSecret, newSecret *corev1.Secret) bool {
+ return CompareMap(GetSecretData(oldSecret), GetSecretData(newSecret))
+}
+
+// CreateIstioSidecarSecurityContext will support runAsUser for sidecar proxy with istio, especially using istio CNI
+// https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
+func CreateIstioSidecarSecurityContext(istioSidecarUid int64) *corev1.SecurityContext {
+ securityContext := corev1.SecurityContext{}
+ securityContext.RunAsUser = &istioSidecarUid
+ securityContext.RunAsGroup = &istioSidecarUid
+ return &securityContext
+}
diff --git a/submarine-cloud-v3/hack/boilerplate.go.txt b/submarine-cloud-v3/hack/boilerplate.go.txt
index 29c55ecd..3e7c6c26 100644
--- a/submarine-cloud-v3/hack/boilerplate.go.txt
+++ b/submarine-cloud-v3/hack/boilerplate.go.txt
@@ -1,15 +1,16 @@
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
\ No newline at end of file
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
diff --git a/submarine-cloud-v3/main.go b/submarine-cloud-v3/main.go
index 7c6fde13..0c5e787d 100644
--- a/submarine-cloud-v3/main.go
+++ b/submarine-cloud-v3/main.go
@@ -1,23 +1,27 @@
/*
-Copyright 2022.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package main
import (
"flag"
+ "fmt"
+ "go.uber.org/zap/zapcore"
+ "k8s.io/client-go/tools/clientcmd"
"os"
// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
@@ -51,6 +55,10 @@ var (
// kubeconfig string
// Flags of submarine
+ istioEnable bool
+ submarineGateway string
+ seldonIstioEnable bool
+ seldonGateway string
clusterType string
createPodSecurityPolicy bool
)
@@ -80,24 +88,59 @@ func main() {
"Enable leader election for controller manager. "+
"Enabling this will ensure there is only one active controller manager.")
// Flags of submarine
+ flag.BoolVar(&istioEnable, "istioenable", true, "Istio enable")
+ flag.StringVar(&submarineGateway, "submarineateway", "", "Submarine gateway, used for server, minio, tensorboard, mlflow and notebook")
+ flag.BoolVar(&seldonIstioEnable, "seldonistioenable", true, "Seldon istio enable")
+ flag.StringVar(&seldonGateway, "seldongateway", "", "Seldon gateway, used for model serve")
flag.StringVar(&clusterType, "clustertype", "kubernetes", "K8s cluster type, can be kubernetes or openshift")
flag.BoolVar(&createPodSecurityPolicy, "createpsp", true, "Specifies whether a PodSecurityPolicy should be created. This configuration enables the database/minio/server to set securityContext.runAsUser")
opts := zap.Options{
Development: true,
+ // format timestamp with 2006-01-02T15:04:05.000Z0700
+ TimeEncoder: zapcore.ISO8601TimeEncoder,
}
opts.BindFlags(flag.CommandLine)
flag.Parse()
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
+ // get current namespace
+ loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
+ kubeConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, &clientcmd.ConfigOverrides{})
+ namespace, _, err := kubeConfig.Namespace()
+
+ // if `seldonGateway` is empty, used ${namespace}/seldon-gateway
+ // By default, the operator and seldon-gateway will be under the same namespace when deployed with helm
+ if seldonGateway == "" {
+ seldonGateway = fmt.Sprintf("%s/seldon-gateway", namespace)
+ }
+ // if `submarineGateway` is empty, used ${namespace}/seldon-gateway
+ // By default, the operator and submarine-gateway will be under the same namespace when deployed with helm
+ if submarineGateway == "" {
+ submarineGateway = fmt.Sprintf("%s/submarine-gateway", namespace)
+ }
+
+ setupLog.Info("Starting submarine operator with ",
+ "metrics-bind-address", &metricsAddr,
+ "health-probe-bind-address", &probeAddr,
+ "leader-elect", &enableLeaderElection,
+ "namespace", namespace,
+ "istioenable", &istioEnable,
+ "submarineateway", &submarineGateway,
+ "seldonistioenable", &seldonIstioEnable,
+ "seldongateway", &seldonGateway,
+ "clustertype", &clusterType,
+ "createpsp", &createPodSecurityPolicy,
+ )
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
- Scheme: scheme,
- MetricsBindAddress: metricsAddr,
- Port: 9443,
- HealthProbeBindAddress: probeAddr,
- LeaderElection: enableLeaderElection,
- LeaderElectionID: "5d52732c.submarine.apache.org",
+ Scheme: scheme,
+ MetricsBindAddress: metricsAddr,
+ Port: 9443,
+ HealthProbeBindAddress: probeAddr,
+ LeaderElectionNamespace: namespace,
+ LeaderElection: enableLeaderElection,
+ LeaderElectionID: "5d52732c.submarine.apache.org",
})
if err != nil {
setupLog.Error(err, "unable to start manager")
@@ -109,6 +152,11 @@ func main() {
Scheme: mgr.GetScheme(),
Recorder: mgr.GetEventRecorderFor(controllerAgentName),
Log: ctrl.Log.WithName(controllerAgentName),
+ Namespace: namespace,
+ IstioEnable: istioEnable,
+ SubmarineGateway: submarineGateway,
+ SeldonIstioEnable: seldonIstioEnable,
+ SeldonGateway: seldonGateway,
ClusterType: clusterType,
CreatePodSecurityPolicy: createPodSecurityPolicy,
}).SetupWithManager(mgr); err != nil {
diff --git a/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java b/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
index 9d1a403e..6b0b9e45 100644
--- a/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
+++ b/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
@@ -65,8 +65,17 @@ public class SubmarineConfVars {
METASTORE_JDBC_USERNAME("metastore.jdbc.username", "metastore"),
METASTORE_JDBC_PASSWORD("metastore.jdbc.password", "password"),
+ /* S3(Minio) config */
+ SUBMARINE_S3_ACCESS_KEY_ID("submarine.s3.accessKeyId", "submarine_minio"),
+ SUBMARINE_S3_SECRET_ACCESS_KEY("submarine.s3.secretAccessKey", "submarine_minio"),
+ SUBMARINE_S3_ENDPOINT("submarine.s3.endpoint", "http://submarine-minio-service:9000"),
+
SUBMARINE_NOTEBOOK_DEFAULT_OVERWRITE_JSON("submarine.notebook.default.overwrite_json", ""),
+ SUBMARINE_ISTIO_SUBMARINE_GATEWAY("submarine.istio.submarine.gateway", "submarine/submarine-gateway"),
+ /* serve(seldon) config */
+ SUBMARINE_ISTIO_SELDON_GATEWAY("submarine.istio.seldon.gateway", "submarine/seldon-gateway"),
+
WORKBENCH_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE(
"workbench.websocket.max.text.message.size", "1024000"),
WORKBENCH_WEB_WAR("workbench.web.war", "submarine-workbench/workbench-web/dist"),
diff --git a/submarine-sdk/pysubmarine/submarine/tracking/client.py b/submarine-sdk/pysubmarine/submarine/tracking/client.py
index 7c4a2677..bebbc466 100644
--- a/submarine-sdk/pysubmarine/submarine/tracking/client.py
+++ b/submarine-sdk/pysubmarine/submarine/tracking/client.py
@@ -50,9 +50,21 @@ class SubmarineClient:
`Where Runs Get Recorded <../tracking.html#where-runs-get-recorded>`_
for more info.
"""
- os.environ["MLFLOW_S3_ENDPOINT_URL"] = s3_registry_uri or S3_ENDPOINT_URL
- os.environ["AWS_ACCESS_KEY_ID"] = aws_access_key_id or AWS_ACCESS_KEY_ID
- os.environ["AWS_SECRET_ACCESS_KEY"] = aws_secret_access_key or AWS_SECRET_ACCESS_KEY
+ # s3 endpoint url
+ if s3_registry_uri is not None:
+ os.environ["MLFLOW_S3_ENDPOINT_URL"] = s3_registry_uri
+ elif "MLFLOW_S3_ENDPOINT_URL" not in os.environ:
+ os.environ["MLFLOW_S3_ENDPOINT_URL"] = S3_ENDPOINT_URL
+ # access key
+ if aws_access_key_id is not None:
+ os.environ["AWS_ACCESS_KEY_ID"] = aws_access_key_id
+ elif "AWS_ACCESS_KEY_ID" not in os.environ:
+ os.environ["AWS_ACCESS_KEY_ID"] = AWS_ACCESS_KEY_ID
+ # access secret
+ if aws_secret_access_key is not None:
+ os.environ["AWS_SECRET_ACCESS_KEY"] = aws_secret_access_key
+ elif "AWS_SECRET_ACCESS_KEY" not in os.environ:
+ os.environ["AWS_SECRET_ACCESS_KEY"] = AWS_SECRET_ACCESS_KEY
self.artifact_repo = Repository()
self.db_uri = db_uri or submarine.get_db_uri()
self.store = utils.get_tracking_sqlalchemy_store(self.db_uri)
diff --git a/submarine-serve/README.md b/submarine-serve/README.md
index 04f73e75..7b11a20d 100644
--- a/submarine-serve/README.md
+++ b/submarine-serve/README.md
@@ -14,27 +14,4 @@
# Submarine Serve
-Submarine serve uses istio 1.6.8 and seldon-core 1.10.0 for serving.
-
-## Install
-
-- Install submarine first
-
-```bash
-cd submarine
-helm dependency update ./helm-charts/submarine
-helm install submarine ./helm-charts/submarine
-```
-
-- Install submarine serve package istio and seldon-core
-
-```bash
-./submarine-serve/installation/install.sh
-```
-
-### Uninstall Submarine Serve Package
-
-```bash
-kubectl delete ns istio-system
-kubectl delete ns seldon-system
-```
+Submarine serve uses istio 1.13.+ (default) and seldon-core 1.10.0 for serving.
diff --git a/submarine-serve/installation/install.sh b/submarine-serve/installation/install.sh
deleted file mode 100755
index 05c85455..00000000
--- a/submarine-serve/installation/install.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -euxo pipefail
-
-if [ -L ${BASH_SOURCE-$0} ]; then
- PWD=$(dirname $(readlink "${BASH_SOURCE-$0}"))
-else
- PWD=$(dirname ${BASH_SOURCE-$0})
-fi
-export CURRENT_PATH=$(cd "${PWD}">/dev/null; pwd)
-SUBMARINE_HOME=${CURRENT_PATH}/../..
-
-if [[ ! -f ${CURRENT_PATH}/istioctl ]]; then
- wget https://github.com/istio/istio/releases/download/1.6.8/istio-1.6.8-linux-amd64.tar.gz
- tar zxvf istio-1.6.8-linux-amd64.tar.gz
- rm -rf istio-1.6.8-linux-amd64.tar.gz
- cp ./istio-1.6.8/bin/istioctl ${CURRENT_PATH}/istioctl
- rm -rf istio-1.6.8
-fi
-
-${CURRENT_PATH}/istioctl install --skip-confirmation
diff --git a/submarine-serve/pom.xml b/submarine-serve/pom.xml
index 72559da0..d08ed49a 100644
--- a/submarine-serve/pom.xml
+++ b/submarine-serve/pom.xml
@@ -50,7 +50,6 @@
<groupId>org.apache.submarine</groupId>
<artifactId>submarine-commons-utils</artifactId>
<version>${project.version}</version>
- <scope>test</scope>
</dependency>
<dependency>
diff --git a/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualService.java b/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualService.java
index 608d95a4..709b10fa 100644
--- a/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualService.java
+++ b/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualService.java
@@ -65,8 +65,7 @@ public class IstioVirtualService implements KubernetesObject {
protected IstioVirtualServiceSpec parseVirtualServiceSpec(String namespace, String name) {
IstioVirtualServiceSpec spec = new IstioVirtualServiceSpec();
spec.addHost(IstioConstants.DEFAULT_INGRESS_HOST);
- // TODO(operator): Do not hard code the gateway
- spec.addGateway("submarine/submarine-gateway");
+ spec.addGateway(IstioConstants.SUBMARINE_GATEWAY);
String matchURIPrefix = "/notebook/" + namespace + "/" + name;
spec.setHTTPRoute(new IstioHTTPRoute(matchURIPrefix, name, 80));
return spec;
diff --git a/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualServiceSpec.java b/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualServiceSpec.java
index bf63bc42..4cc4c59a 100644
--- a/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualServiceSpec.java
+++ b/submarine-serve/src/main/java/org/apache/submarine/serve/istio/IstioVirtualServiceSpec.java
@@ -42,7 +42,7 @@ public class IstioVirtualServiceSpec {
public IstioVirtualServiceSpec(Long id, String modelResourceName, Integer modelVersion) {
hosts.add(IstioConstants.DEFAULT_INGRESS_HOST);
- gateways.add(IstioConstants.DEFAULT_GATEWAY);
+ gateways.add(IstioConstants.SELDON_GATEWAY);
// model resource name is service name
IstioHTTPDestination destination = new IstioHTTPDestination(modelResourceName);
IstioHTTPMatchRequest matchRequest = new IstioHTTPMatchRequest(
diff --git a/submarine-serve/src/main/java/org/apache/submarine/serve/utils/IstioConstants.java b/submarine-serve/src/main/java/org/apache/submarine/serve/utils/IstioConstants.java
index 10ae72d4..d7bc5b2c 100644
--- a/submarine-serve/src/main/java/org/apache/submarine/serve/utils/IstioConstants.java
+++ b/submarine-serve/src/main/java/org/apache/submarine/serve/utils/IstioConstants.java
@@ -18,7 +18,14 @@
*/
package org.apache.submarine.serve.utils;
+import org.apache.submarine.commons.utils.SubmarineConfVars;
+import org.apache.submarine.commons.utils.SubmarineConfiguration;
+
+/**
+ * Istio constants
+ */
public class IstioConstants {
+
public static final String API_VERSION = "networking.istio.io/v1beta1";
public static final String KIND = "VirtualService";
@@ -29,14 +36,22 @@ public class IstioConstants {
public static final String PLURAL = "virtualservices";
- public static final String REWRITE_URL = "/";
+ public static final String REWRITE_URL = "/";
- public static final String DEFAULT_NAMESPACE = "default";
+ /* istio submarine gateway */
+ public static final String SUBMARINE_GATEWAY;
- public static final String DEFAULT_GATEWAY = "istio-system/seldon-gateway";
+ /* istio seldon gateway */
+ public static final String SELDON_GATEWAY;
public static final Integer DEFAULT_SERVE_POD_PORT = 8000;
- public static final String DEFAULT_INGRESS_HOST = "*";
+ public static final String DEFAULT_INGRESS_HOST = "*";
+
+ static {
+ final SubmarineConfiguration conf = SubmarineConfiguration.getInstance();
+ SUBMARINE_GATEWAY = conf.getString(SubmarineConfVars.ConfVars.SUBMARINE_ISTIO_SUBMARINE_GATEWAY);
+ SELDON_GATEWAY = conf.getString(SubmarineConfVars.ConfVars.SUBMARINE_ISTIO_SELDON_GATEWAY);
+ }
}
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/Client.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/Client.java
index 1e455abd..fd397144 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/Client.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/Client.java
@@ -38,24 +38,37 @@ import io.minio.Result;
import io.minio.messages.DeleteError;
import io.minio.messages.DeleteObject;
import io.minio.messages.Item;
+import org.apache.submarine.commons.utils.SubmarineConfVars;
+import org.apache.submarine.commons.utils.SubmarineConfiguration;
import org.apache.submarine.commons.utils.exception.SubmarineRuntimeException;
-
+/**
+ * S3(Minio) default client
+ */
public class Client {
+
+ /* minio client */
public MinioClient minioClient;
+ /* submarine config */
+ private static final SubmarineConfiguration conf = SubmarineConfiguration.getInstance();
+
public Client() {
minioClient = MinioClient.builder()
- .endpoint(S3Constants.ENDPOINT)
- .credentials(S3Constants.ACCESSKEY, S3Constants.SECRETKEY)
- .build();
+ .endpoint(conf.getString(SubmarineConfVars.ConfVars.SUBMARINE_S3_ENDPOINT))
+ .credentials(
+ conf.getString(SubmarineConfVars.ConfVars.SUBMARINE_S3_ACCESS_KEY_ID),
+ conf.getString(SubmarineConfVars.ConfVars.SUBMARINE_S3_SECRET_ACCESS_KEY)
+ ).build();
}
public Client(String endpoint) {
minioClient = MinioClient.builder()
.endpoint(endpoint)
- .credentials(S3Constants.ACCESSKEY, S3Constants.SECRETKEY)
- .build();
+ .credentials(
+ conf.getString(SubmarineConfVars.ConfVars.SUBMARINE_S3_ACCESS_KEY_ID),
+ conf.getString(SubmarineConfVars.ConfVars.SUBMARINE_S3_SECRET_ACCESS_KEY)
+ ).build();
}
/**
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/S3Constants.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/S3Constants.java
index c92cacfd..ff080aee 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/S3Constants.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/s3/S3Constants.java
@@ -19,12 +19,20 @@
package org.apache.submarine.server.s3;
+/**
+ * S3(Minio) Constants
+ */
public class S3Constants {
- public static final String ENDPOINT = "http://submarine-minio-service:9000";
- public static final String ACCESSKEY = "submarine_minio";
+ /* default endpoint, it is the same as `submarine.s3.endpoint` */
+ public static final String DEFAULT_ENDPOINT = "http://submarine-minio-service:9000";
+
+ /* default access key, it is the same as `submarine.s3.accessKeyId` */
+ public static final String DEFAULT_ACCESSKEY = "submarine_minio";
- public static final String SECRETKEY = "submarine_minio";
+ /* default secret key, it is the same as `submarine.s3.secretAccessKey` */
+ public static final String DEFAULT_SECRETKEY = "submarine_minio";
+ /* default bucket */
public static final String BUCKET = "submarine";
}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
index 991e03cf..368ae564 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
@@ -23,12 +23,14 @@ import io.kubernetes.client.custom.Quantity;
import io.kubernetes.client.openapi.models.V1Container;
import io.kubernetes.client.openapi.models.V1EnvVar;
+import io.kubernetes.client.openapi.models.V1EnvVarSource;
import io.kubernetes.client.openapi.models.V1ObjectMetaBuilder;
import io.kubernetes.client.openapi.models.V1PersistentVolumeClaimVolumeSource;
import io.kubernetes.client.openapi.models.V1PodSecurityContext;
import io.kubernetes.client.openapi.models.V1PodSpec;
import io.kubernetes.client.openapi.models.V1PodTemplateSpec;
import io.kubernetes.client.openapi.models.V1ResourceRequirements;
+import io.kubernetes.client.openapi.models.V1SecretKeySelector;
import io.kubernetes.client.openapi.models.V1SecretVolumeSource;
import io.kubernetes.client.openapi.models.V1Volume;
import io.kubernetes.client.openapi.models.V1VolumeMount;
@@ -40,6 +42,7 @@ import org.apache.submarine.server.api.spec.ExperimentSpec;
import org.apache.submarine.server.api.spec.ExperimentTaskSpec;
import org.apache.submarine.server.api.spec.EnvironmentSpec;
import org.apache.submarine.server.manager.EnvironmentManager;
+import org.apache.submarine.server.s3.S3Constants;
import org.apache.submarine.server.submitter.k8s.experiment.codelocalizer.AbstractCodeLocalizer;
import org.apache.submarine.server.submitter.k8s.experiment.codelocalizer.CodeLocalizer;
import org.apache.submarine.server.submitter.k8s.experiment.codelocalizer.SSHGitCodeLocalizer;
@@ -54,6 +57,16 @@ public class ExperimentSpecParser {
private static final SubmarineConfiguration conf = SubmarineConfiguration.getInstance();
+ /**
+ * S3 secret
+ */
+ public static final String MLFLOW_S3_ENDPOINT_URL = "MLFLOW_S3_ENDPOINT_URL";
+ public static final String AWS_ACCESS_KEY_ID = "AWS_ACCESS_KEY_ID";
+ public static final String AWS_SECRET_ACCESS_KEY = "AWS_SECRET_ACCESS_KEY";
+ public static final String MINIO_ACCESS_KEY = "MINIO_ACCESS_KEY";
+ public static final String MINIO_SECRET_KEY = "MINIO_SECRET_KEY";
+ public static final String DEFAULT_SUBMARINE_MINIO_SECRET = "submarine-minio-secret";
+
public static V1PodTemplateSpec parseTemplateSpec(
ExperimentTaskSpec taskSpec, ExperimentSpec experimentSpec) throws InvalidSpecException {
V1PodTemplateSpec templateSpec = new V1PodTemplateSpec();
@@ -248,6 +261,26 @@ public class ExperimentSpecParser {
env.setValue(entry.getValue());
envVars.add(env);
}
+ // add s3(minio) url and secrets
+ envVars.add(
+ new V1EnvVar().name(MLFLOW_S3_ENDPOINT_URL).value(S3Constants.DEFAULT_ENDPOINT)
+ );
+ envVars.add(new V1EnvVar().name(AWS_ACCESS_KEY_ID)
+ .valueFrom(new V1EnvVarSource()
+ .secretKeyRef(new V1SecretKeySelector()
+ .key(MINIO_ACCESS_KEY)
+ .name(DEFAULT_SUBMARINE_MINIO_SECRET)
+ )
+ )
+ );
+ envVars.add(new V1EnvVar().name(AWS_SECRET_ACCESS_KEY)
+ .valueFrom(new V1EnvVarSource()
+ .secretKeyRef(new V1SecretKeySelector()
+ .key(MINIO_SECRET_KEY)
+ .name(DEFAULT_SUBMARINE_MINIO_SECRET)
+ )
+ )
+ );
return envVars;
}
diff --git a/website/docs/gettingStarted/quickstart.md b/website/docs/gettingStarted/quickstart.md
index 9cccbd04..21f4a522 100644
--- a/website/docs/gettingStarted/quickstart.md
+++ b/website/docs/gettingStarted/quickstart.md
@@ -70,9 +70,10 @@ kubectl label namespace submarine-user-test istio-injection=enabled
3. Install the submarine operator and dependencies by helm chart
```bash
-# We move seldon-core install to helm, thus we need to update our dependency.
+# Update helm dependency.
helm dependency update ./helm-charts/submarine
-helm install submarine ./helm-charts/submarine -n submarine
+# Install submarine operator in namespace submarine.
+helm install submarine ./helm-charts/submarine --set seldon-core-operator.istio.gateway=submarine/seldon-gateway -n submarine
```
4. Create a Submarine custom resource and the operator will create the submarine server, database, etc. for us.
@@ -81,12 +82,6 @@ helm install submarine ./helm-charts/submarine -n submarine
kubectl apply -f submarine-cloud-v2/artifacts/examples/example-submarine.yaml -n submarine-user-test
```
-5. Install submarine serve dependent minio secret key file
-
-```bash
-kubectl apply -f ./submarine-serve/installation/seldon-secret.yaml -n submarine-user-test
-```
-
### Ensure submarine is ready
```bash
@@ -283,7 +278,7 @@ Metadata:
...
Spec:
Gateways:
- istio-system/seldon-gateway
+ submarine/seldon-gateway
Hosts:
*
Http:
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org