You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by pi...@apache.org on 2021/02/06 15:01:09 UTC
[submarine] branch master updated: SUBMARINE-717. Add NFS support
for persistent volume
This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new c17c53d SUBMARINE-717. Add NFS support for persistent volume
c17c53d is described below
commit c17c53dd03305ad83a535313a7e3a8e1cb24cf87
Author: ByronHsu <by...@gmail.com>
AuthorDate: Thu Feb 4 00:15:15 2021 +0800
SUBMARINE-717. Add NFS support for persistent volume
### What is this PR for?
1. User can use their own NFS server to store persistent storage data.
2. For developers, who might not have an existing nfs server, they can create an nfs server in kubernetes cluster by the yaml provided in `dev-support/`.
3. Added the guidance of nfs usage in docs.
4. Store MySQL data in PV.
### What type of PR is it?
[Feature]
### Todos
* [ ] - Store notebook data in nfs
* [ ] - Store tensorboard data in nfs
### What is the Jira issue?
https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-717
### How should this be tested?
https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-717
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: ByronHsu <by...@gmail.com>
Author: Byron <by...@gmail.com>
Signed-off-by: Kevin <pi...@apache.org>
Closes #498 from ByronHsu/NFS and squashes the following commits:
44ff8d4 [ByronHsu] Merge branch 'master' into NFS
3f93fb8 [ByronHsu] default is host volume
40ec1bf [ByronHsu] remove persistent volume in nfs-server
ea2ee8a [ByronHsu] update path
f12b45f [Byron] minor fix
a9196b8 [Byron] add nfs support for mysql
---
.../nfs-server/nfs-server.yaml | 64 ++++++++++++----------
docs/userdocs/k8s/helm.md | 50 +++++++++++++++++
.../submarine/templates/submarine-database.yaml | 47 +++++++++++++++-
helm-charts/submarine/values.yaml | 6 ++
4 files changed, 135 insertions(+), 32 deletions(-)
diff --git a/helm-charts/submarine/templates/submarine-database.yaml b/dev-support/nfs-server/nfs-server.yaml
similarity index 52%
copy from helm-charts/submarine/templates/submarine-database.yaml
copy to dev-support/nfs-server/nfs-server.yaml
index bd0ab96..6bf9d2d 100644
--- a/helm-charts/submarine/templates/submarine-database.yaml
+++ b/dev-support/nfs-server/nfs-server.yaml
@@ -14,44 +14,48 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
-apiVersion: apps/v1
+---
+kind: Service
+apiVersion: v1
+metadata:
+ name: nfs-server
+spec:
+ type: ClusterIP
+ clusterIP: 10.96.0.2
+ ports:
+ - name: nfs
+ port: 2049
+ - name: mountd
+ port: 20048
+ - name: rpcbind
+ port: 111
+ selector:
+ role: nfs-server
+---
kind: Deployment
+apiVersion: apps/v1
metadata:
- name: "{{ .Values.submarine.database.name }}"
+ name: nfs-server
spec:
- replicas: {{ .Values.submarine.database.replicas }}
+ replicas: 1
selector:
matchLabels:
- app: "{{ .Values.submarine.database.name }}"
+ role: nfs-server
template:
metadata:
+ name: nfs-server
labels:
- app: "{{ .Values.submarine.database.name }}"
- {{ if .Values.submarine.database.dev }}
- annotations:
- timestamp: {{ now | quote }}
- {{ end }}
+ role: nfs-server
spec:
containers:
- - name: "{{ .Values.submarine.database.name }}"
- image: "{{ .Values.submarine.database.image }}"
- imagePullPolicy: {{ .Values.submarine.database.imagePullPolicy }}
+ - name: nfs-server
+ image: k8s.gcr.io/volume-nfs:0.8
ports:
- - containerPort: 3306
- env:
- - name: MYSQL_ROOT_PASSWORD
- value: "{{ .Values.submarine.database.mysqlRootPassword }}"
-
----
-apiVersion: v1
-kind: Service
-metadata:
- name: "{{ .Values.submarine.database.name }}"
-spec:
- ports:
- - name: "{{ .Values.submarine.database.name }}"
- port: 3306
- targetPort: {{ .Values.submarine.database.servicePort }}
- selector:
- app: "{{ .Values.submarine.database.name }}"
+ - name: nfs
+ containerPort: 2049
+ - name: mountd
+ containerPort: 20048
+ - name: rpcbind
+ containerPort: 111
+ securityContext:
+ privileged: true
diff --git a/docs/userdocs/k8s/helm.md b/docs/userdocs/k8s/helm.md
index 6a79e4e..1d36f25 100644
--- a/docs/userdocs/k8s/helm.md
+++ b/docs/userdocs/k8s/helm.md
@@ -78,6 +78,56 @@ submarine-traefik-9bb6f8577-66sx6 1/1 Running 0 5
tf-job-operator-7844656dd-lfgmd 1/1 Running 0 5s
```
+### Configure volume type
+Submarine can support various [volume types](https://kubernetes.io/docs/concepts/storage/volumes/#nfs), currently including hostPath (default) and NFS. It can be easily configured in the `./helm-charts/submarine/values.yaml`, or you can override the default values in `values.yaml` by [helm CLI](https://helm.sh/docs/helm/helm_install/).
+
+#### hostPath
+- In hostPath, you can store data directly in your node.
+- Usage:
+ 1. Configure setting in `./helm-charts/submarine/values.yaml`.
+ 2. To enable hostPath storage, set `.storage.type` to `host`.
+ 3. To set the root path for your storage, set `.storage.host.root` to `<any-path>`
+- Example:
+ ```yaml
+ # ./helm-charts/submarine/values.yaml
+ storage:
+ type: host
+ host:
+ root: /tmp
+ ```
+
+
+#### NFS (Network File System)
+- In NFS, it allows multiple clients to access a shared space.
+- Prerequisite:
+ 1. A pre-existing NFS server. You have two options.
+ 1. Create NFS server
+ ```bash
+ kubectl create -f ./dev-support/nfs-server/nfs-server.yaml
+ ```
+ It will create a nfs-server pod in kubernetes cluster, and expose nfs-server ip at `10.96.0.2`
+ 2. Use your own NFS server
+ 2. Install NFS dependencies in your nodes
+ - Ubuntu
+ ```bash
+ apt-get install -y nfs-common
+ ```
+ - CentOS
+ ```bash
+ yum install nfs-util
+ ```
+- Usage:
+ 1. Configure setting in `./helm-charts/submarine/values.yaml`.
+ 2. To enable NFS storage, set `.storage.type` to `nfs`.
+ 3. To set the ip for NFS server, set `.storage.nfs.ip` to `<any-ip>`
+- Example:
+ ```yaml
+ # ./helm-charts/submarine/values.yaml
+ storage:
+ type: nfs
+ nfs:
+ ip: 10.96.0.2
+ ```
### Access to Submarine Server
Submarine server by default expose 8080 port within K8s cluster. After Submarine v0.5
uses Traefik as reverse-proxy by default. If you don't want to
diff --git a/helm-charts/submarine/templates/submarine-database.yaml b/helm-charts/submarine/templates/submarine-database.yaml
index bd0ab96..562503c 100644
--- a/helm-charts/submarine/templates/submarine-database.yaml
+++ b/helm-charts/submarine/templates/submarine-database.yaml
@@ -14,7 +14,43 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: submarine-database-pv
+spec:
+ accessModes:
+ - ReadWriteMany
+ capacity:
+ storage: 1Gi
+{{- with .Values.submarine.storage }}
+
+ {{- if eq (.type | lower) "nfs" }}
+ nfs:
+ server: {{ .nfs.ip }}
+ path: "/"
+ {{- else }}
+ hostPath:
+ path: "{{ .host.root }}"
+ type: DirectoryOrCreate
+ {{- end }}
+
+{{- end}}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: submarine-database-pvc
+spec:
+ accessModes:
+ - ReadWriteMany
+ storageClassName: ""
+ resources:
+ requests:
+ storage: 1Gi
+ volumeName: submarine-database-pv
+---
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -36,13 +72,20 @@ spec:
containers:
- name: "{{ .Values.submarine.database.name }}"
image: "{{ .Values.submarine.database.image }}"
- imagePullPolicy: {{ .Values.submarine.database.imagePullPolicy }}
+ imagePullPolicy: "{{ .Values.submarine.database.imagePullPolicy }}"
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "{{ .Values.submarine.database.mysqlRootPassword }}"
-
+ volumeMounts:
+ - mountPath: /var/lib/mysql
+ name: volume
+ subPath: submarine-database
+ volumes:
+ - name: volume
+ persistentVolumeClaim:
+ claimName: submarine-database-pvc
---
apiVersion: v1
kind: Service
diff --git a/helm-charts/submarine/values.yaml b/helm-charts/submarine/values.yaml
index 23f668d..cbd5479 100644
--- a/helm-charts/submarine/values.yaml
+++ b/helm-charts/submarine/values.yaml
@@ -33,6 +33,12 @@ submarine:
dev: false # if true, restart database pod every time at helm upgrade
traefik:
enabled: true
+ storage:
+ type: host # "host" or "nfs"
+ host:
+ root: /.submarine/host
+ nfs:
+ ip: 10.96.0.2
tensorboard:
storage: 10Gi
hostPath: /tmp/tfboard/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org