You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by ji...@apache.org on 2021/01/04 15:44:24 UTC
[submarine] branch master updated: SUBMARINE-701. Support
Tensorboard in Experiment
This is an automated email from the ASF dual-hosted git repository.
jiwq pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 0750ffc SUBMARINE-701. Support Tensorboard in Experiment
0750ffc is described below
commit 0750ffc01bac393fd92c18a54559a57710bf860c
Author: Byron <by...@gmail.com>
AuthorDate: Mon Jan 4 15:54:17 2021 +0800
SUBMARINE-701. Support Tensorboard in Experiment
### What is this PR for?
Support new feature in 0.6.0: tensorboard integration.
- Usage
1. Create a job request that uses tensorboard
2. Write tensorboard log to `/logs/mylog` (The subpath is required due to this issue in tensorflow [https://github.com/kubeflow/tf-operator/issues/1053](https://github.com/kubeflow/tf-operator/issues/1053). We cannot directly write log file to mountPath.)
3. Link to `http://<host>:<ip>/tfboard-${job-name}`, and you can monitor the tensorboard with ease!
- Implementation
When creating a new job, the backend will not only create original experiment but also several k8s resources required in tensorboard
The resources can be classified into two categories:
1. Storage
2. Tensorboard serving
**Storage**
The resources required for storage are **persistent volume** and **persistent volume claim**.
I set the storage path of persistent volume on host path, and mount this path to MLjob (enable job to generate logs to volume) and Tensorboard (enable tfboard to access logs).
**Tensorboard Serving**
The resources required here are **deployments, service, and ingressroute**.
I create the tensorboard apps with deployments and service, and then redirect it to custom path with the help of ingressroute.
- Example
- tensorboard-example.json
```bash
{
"meta": {
"name": "tensorflow-dist-mnist-byron-1234",
"namespace": "default",
"framework": "TensorFlow",
"cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/logs/mylog --learning_rate=0.01 --batch_size=20",
"envVars": {
"ENV_1": "ENV1"
}
},
"environment": {
"image": "apache/submarine:tf-mnist-with-summaries-1.0"
},
"spec": {
"Worker": {
"replicas": 1,
"resources": "cpu=1,memory=1024M"
}
}
}
```
### What type of PR is it?
[Feature]
### Todos
- [ ] Frontend support
- [ ] The logs of job cannot be written directly on the mountPath (As describe in above). We should fix this problem.
- [ ] Make log path configurable (Currently, it is hard-coded as `/logs` )
- [ ] Support smb-server for shared storage
### What is the Jira issue?
https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-701
### How should this be tested?
https://travis-ci.org/github/ByronHsu/submarine/jobs/751658488
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Byron <by...@gmail.com>
Author: ByronHsu <by...@gmail.com>
Closes #484 from ByronHsu/SUBMARINE-701 and squashes the following commits:
9ecb04f [Byron] add timeout between each testcase
8168cc2 [Byron] convert to lower camel case
f6028a4 [Byron] change variable in spec parser to lowercamelcase
3d668c5 [Byron] update
2bf9696 [Byron] make error meesage more understandable
961621b [ByronHsu] Update submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
6e4c077 [ByronHsu] Update submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
64e5c4e [ByronHsu] Update submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
6b2f604 [ByronHsu] Update submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
1c1258f [ByronHsu] Update submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
29af97a [Byron] update rbac
b58cccb [Byron] update rbac
2d8940b [Byron] add license to new files
8a386d1 [Byron] tensorboard works fine in server
e4c82fb [Byron] update k8sSubmitterTest
8014ea5 [Byron] test fine with k8sjobsubmittertest
8a8080a [Byron] add tensorboardspec parser
25a694b [Byron] add volumeSpecParser & tfb constants
---
helm-charts/submarine/templates/rbac.yaml | 9 +
.../manifests/submarine-cluster/rbac.yaml | 8 +-
.../server/api/experiment/Experiment.java | 9 +
.../submarine/server/api/spec/ExperimentMeta.java | 11 ++
.../submarine/server/api/spec/ExperimentSpec.java | 10 ++
.../apache/submarine/server/SubmarineServer.java | 2 +-
.../server/experiment/ExperimentManager.java | 6 +
.../server/submitter/k8s/K8sSubmitter.java | 186 ++++++++++++++++++++-
.../k8s/model/ingressroute/SpecRoute.java | 11 ++
.../k8s/model/middlewares/Middlewares.java | 117 +++++++++++++
.../MiddlewaresSpec.java} | 42 +----
.../submitter/k8s/parser/ExperimentSpecParser.java | 21 ++-
.../k8s/parser/TensorboardSpecParser.java | 141 ++++++++++++++++
.../submitter/k8s/parser/VolumeSpecParser.java | 85 ++++++++++
.../submitter/k8s/util/TensorboardUtils.java | 38 +++++
.../server/submitter/k8s/K8SJobSubmitterTest.java | 16 +-
.../server/submitter/k8s/SpecBuilder.java | 1 +
.../k8s/parser/TensorboardSpecParserTest.java | 80 +++++++++
.../submitter/k8s/parser/VolumeSpecParserTest.java | 64 +++++++
.../src/test/resources/tf_tfboard_mnist_req.json | 20 +++
.../apache/submarine/rest/ExperimentRestApiIT.java | 14 +-
21 files changed, 845 insertions(+), 46 deletions(-)
diff --git a/helm-charts/submarine/templates/rbac.yaml b/helm-charts/submarine/templates/rbac.yaml
index acad2e1..bed8c19 100644
--- a/helm-charts/submarine/templates/rbac.yaml
+++ b/helm-charts/submarine/templates/rbac.yaml
@@ -56,6 +56,15 @@ rules:
resources:
- pods
- pods/log
+ - services
+ - persistentvolumes
+ - persistentvolumeclaims
+ verbs:
+ - '*'
+- apiGroups:
+ - "apps"
+ resources:
+ - deployments
verbs:
- '*'
---
diff --git a/submarine-cloud/manifests/submarine-cluster/rbac.yaml b/submarine-cloud/manifests/submarine-cluster/rbac.yaml
index 62931d4..6232d3b 100644
--- a/submarine-cloud/manifests/submarine-cluster/rbac.yaml
+++ b/submarine-cloud/manifests/submarine-cluster/rbac.yaml
@@ -28,7 +28,13 @@ items:
- services
- endpoints
- pods
- verbs: ["list", "get"]
+ - persistentvolumes
+ - persistentvolumeclaims
+ verbs: ["*"]
+ - apiGroups: ["apps"]
+ resources:
+ - deployments
+ verbs: ["*"]
- apiGroups: ["kubeflow.org"]
resources:
- tfjobs
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/Experiment.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/Experiment.java
index 2f14460..932b1a6 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/Experiment.java
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/Experiment.java
@@ -34,6 +34,7 @@ public class Experiment {
private String runningTime;
private String finishedTime;
private ExperimentSpec spec;
+ private String tfboardURL;
/**
* Get the job id which is unique in submarine
@@ -135,6 +136,14 @@ public class Experiment {
this.spec = spec;
}
+ public String getTfboardURL() {
+ return tfboardURL;
+ }
+
+ public void setTfboardURL(String tfboardURL) {
+ this.tfboardURL = tfboardURL;
+ }
+
public enum Status {
STATUS_ACCEPTED("Accepted"),
STATUS_CREATED("Created"),
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java
index de98fd7..d682740 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java
@@ -131,4 +131,15 @@ public class ExperimentMeta {
return names;
}
}
+
+ @Override
+ public String toString() {
+ return "ExperimentMeta{" +
+ "name='" + name + '\'' +
+ ", namespace='" + namespace + '\'' +
+ ", framework='" + framework + '\'' +
+ ", cmd='" + cmd + '\'' +
+ ", envVars=" + envVars +
+ '}';
+ }
}
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
index 5e8b23d..b0c283a 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
@@ -63,4 +63,14 @@ public class ExperimentSpec {
public void setCode(CodeSpec code) {
this.code = code;
}
+
+ @Override
+ public String toString() {
+ return "ExperimentSpec{" +
+ "meta=" + meta +
+ ", environment=" + environment +
+ ", spec=" + spec +
+ ", code=" + code +
+ '}';
+ }
}
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/SubmarineServer.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/SubmarineServer.java
index f955eeb..43ccbf9 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/SubmarineServer.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/SubmarineServer.java
@@ -307,7 +307,7 @@ public class SubmarineServer extends ResourceConfig {
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
- throws ServletException, IOException {
+ throws ServletException, IOException {
response.setContentType("text/html");
response.encodeRedirectURL("/");
response.setStatus(HttpServletResponse.SC_OK);
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
index 6e38429..6da31db 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
@@ -101,6 +101,12 @@ public class ExperimentManager {
Experiment experiment = submitter.createExperiment(spec);
experiment.setExperimentId(id);
+ /*
+ TODO(byronhsu) Importing tensorboardUtils will
+ cause a dependency circle. Hard-code it as a temporary solution
+ */
+
+ experiment.setTfboardURL("/tfboard-" + spec.getMeta().getName() + "/");
spec.getMeta().getEnvVars().remove(RestConstants.JOB_ID);
spec.getMeta().getEnvVars().remove(RestConstants.SUBMARINE_TRACKING_URI);
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
index a97aa63..5521f99 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
@@ -23,22 +23,26 @@ import java.io.FileReader;
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
import java.util.Map;
import java.util.Set;
-import java.util.List;
-
import com.google.gson.Gson;
import com.google.gson.JsonSyntaxException;
import io.kubernetes.client.ApiClient;
import io.kubernetes.client.ApiException;
import io.kubernetes.client.Configuration;
import io.kubernetes.client.JSON;
+import io.kubernetes.client.apis.AppsV1Api;
import io.kubernetes.client.apis.CoreV1Api;
import io.kubernetes.client.apis.CustomObjectsApi;
import io.kubernetes.client.models.V1DeleteOptionsBuilder;
+import io.kubernetes.client.models.V1Deployment;
import io.kubernetes.client.models.V1ObjectMeta;
+import io.kubernetes.client.models.V1PersistentVolume;
+import io.kubernetes.client.models.V1PersistentVolumeClaim;
import io.kubernetes.client.models.V1Pod;
import io.kubernetes.client.models.V1PodList;
+import io.kubernetes.client.models.V1Service;
import io.kubernetes.client.models.V1Status;
import io.kubernetes.client.util.ClientBuilder;
import io.kubernetes.client.util.KubeConfig;
@@ -59,8 +63,11 @@ import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRoute
import org.apache.submarine.server.submitter.k8s.model.ingressroute.SpecRoute;
import org.apache.submarine.server.submitter.k8s.parser.ExperimentSpecParser;
import org.apache.submarine.server.submitter.k8s.parser.NotebookSpecParser;
+import org.apache.submarine.server.submitter.k8s.parser.TensorboardSpecParser;
+import org.apache.submarine.server.submitter.k8s.parser.VolumeSpecParser;
import org.apache.submarine.server.submitter.k8s.util.MLJobConverter;
import org.apache.submarine.server.submitter.k8s.util.NotebookUtils;
+import org.apache.submarine.server.submitter.k8s.util.TensorboardUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -81,11 +88,14 @@ public class K8sSubmitter implements Submitter {
private CoreV1Api coreApi;
- public K8sSubmitter() {}
+ private AppsV1Api appsV1Api;
+
+ public K8sSubmitter() { }
@Override
public void initialize(SubmarineConfiguration conf) {
ApiClient client = null;
+
try {
String path = System.getenv(KUBECONFIG_ENV);
KubeConfig config = KubeConfig.loadKubeConfig(new FileReader(path));
@@ -108,13 +118,25 @@ public class K8sSubmitter implements Submitter {
if (coreApi == null) {
coreApi = new CoreV1Api(client);
}
+ if (appsV1Api == null) {
+ appsV1Api = new AppsV1Api();
+ }
+
+ client.setDebugging(true);
}
@Override
public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeException {
Experiment experiment;
+ final String name = spec.getMeta().getName();
+
try {
MLJob mlJob = ExperimentSpecParser.parseJob(spec);
+
+ createTFBoardPersistentVolume(name);
+ createTFBoardPersistentVolumeClaim(name, spec.getMeta().getNamespace());
+ createTFBoard(name, spec.getMeta().getNamespace());
+
Object object = api.createNamespacedCustomObject(mlJob.getGroup(), mlJob.getVersion(),
mlJob.getMetadata().getNamespace(), mlJob.getPlural(), mlJob, "true");
experiment = parseExperimentResponseObject(object, ParseOp.PARSE_OP_RESULT);
@@ -165,8 +187,15 @@ public class K8sSubmitter implements Submitter {
@Override
public Experiment deleteExperiment(ExperimentSpec spec) throws SubmarineRuntimeException {
Experiment experiment;
+ final String name = spec.getMeta().getName(); // spec.getMeta().getEnvVars().get(RestConstants.JOB_ID);
+
try {
MLJob mlJob = ExperimentSpecParser.parseJob(spec);
+
+ deleteTFBoardPersistentVolume(name);
+ deleteTFBoardPersistentVolumeClaim(name, spec.getMeta().getNamespace());
+ deleteTFBoard(name, spec.getMeta().getNamespace());
+
Object object = api.deleteNamespacedCustomObject(mlJob.getGroup(), mlJob.getVersion(),
mlJob.getMetadata().getNamespace(), mlJob.getPlural(), mlJob.getMetadata().getName(),
MLJobConverter.toDeleteOptionsFromMLJob(mlJob), null, null, null);
@@ -320,6 +349,157 @@ public class K8sSubmitter implements Submitter {
return notebookList;
}
+ public void createTFBoard(String name, String namespace) throws ApiException {
+ final String deployName = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String podName = TensorboardUtils.POD_PREFIX + name;
+ final String svcName = TensorboardUtils.SVC_PREFIX + name;
+ final String ingressName = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String routePath = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deployName, image, routePath, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svcName, podName);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingressName, namespace, routePath, svcName
+ );
+
+ try {
+ appsV1Api.createNamespacedDeployment(namespace, deployment, "true", null, null);
+ coreApi.createNamespacedService(namespace, svc, "true", null, null);
+ api.createNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(),
+ ingressRoute.getPlural(), ingressRoute, "true");
+ } catch (ApiException e) {
+ LOG.error("Exception when creating TensorBoard " + e.getMessage(), e);
+ throw e;
+ }
+ }
+
+ public void deleteTFBoard(String name, String namespace) throws ApiException {
+ final String deployName = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String podName = TensorboardUtils.POD_PREFIX + name;
+ final String svcName = TensorboardUtils.SVC_PREFIX + name;
+ final String ingressName = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String routePath = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deployName, image, routePath, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svcName, podName);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingressName, namespace, routePath, svcName
+ );
+
+ try {
+ appsV1Api.deleteNamespacedDeployment(deployName, namespace, "true",
+ null, null, null, null, null);
+ coreApi.deleteNamespacedService(svcName, namespace, "true",
+ null, null, null, null, null);
+ api.deleteNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(), ingressRoute.getPlural(), ingressName,
+ new V1DeleteOptionsBuilder().withApiVersion(ingressRoute.getApiVersion()).build(),
+ null, null, null);
+
+ } catch (ApiException e) {
+ LOG.error("Exception when deleting TensorBoard " + e.getMessage(), e);
+ throw e;
+ }
+ }
+
+ public void createTFBoardPersistentVolume(String name) throws ApiException {
+ final String pvName = TensorboardUtils.PV_PREFIX + name;
+ final String hostPath = TensorboardUtils.HOST_PREFIX + name;
+ final String storage = TensorboardUtils.STORAGE;
+
+ V1PersistentVolume pv = VolumeSpecParser.parsePersistentVolume(pvName, hostPath, storage);
+
+ try {
+ V1PersistentVolume result = coreApi.createPersistentVolume(pv, "true", null, null);
+ } catch (ApiException e) {
+ LOG.error("Exception when creating persistent volume " + e.getMessage(), e);
+ throw e;
+ }
+ }
+
+ public void deleteTFBoardPersistentVolume(String name) throws ApiException {
+ /*
+ This version of Kubernetes-client/java has bug here.
+ It will trigger exception as in https://github.com/kubernetes-client/java/issues/86
+ but it can still work fine and delete the PV.
+ */
+ final String pvName = TensorboardUtils.PV_PREFIX + name;
+
+ try {
+ V1Status result = coreApi.deletePersistentVolume(
+ pvName, "true", null,
+ null, null, null, null
+ );
+ } catch (ApiException e) {
+ LOG.error("Exception when deleting persistent volume " + e.getMessage(), e);
+ throw e;
+ } catch (JsonSyntaxException e) {
+ if (e.getCause() instanceof IllegalStateException) {
+ IllegalStateException ise = (IllegalStateException) e.getCause();
+ if (ise.getMessage() != null && ise.getMessage().contains("Expected a string but was BEGIN_OBJECT"))
+ LOG.debug("Catching exception because of issue " +
+ "https://github.com/kubernetes-client/java/issues/86", e);
+ else throw e;
+ }
+ else throw e;
+ }
+ }
+
+ public void createTFBoardPersistentVolumeClaim(String name, String namespace) throws ApiException {
+ final String pvcName = TensorboardUtils.PVC_PREFIX + name;
+ final String volume = TensorboardUtils.PV_PREFIX + name;
+ final String storage = TensorboardUtils.STORAGE;
+
+ V1PersistentVolumeClaim pvc = VolumeSpecParser.parsePersistentVolumeClaim(pvcName, volume, storage);
+
+ try {
+ V1PersistentVolumeClaim result = coreApi.createNamespacedPersistentVolumeClaim(
+ namespace, pvc, "true", null, null
+ );
+ } catch (ApiException e) {
+ LOG.error("Exception when creating persistent volume claim " + e.getMessage(), e);
+ throw e;
+ }
+ }
+
+ public void deleteTFBoardPersistentVolumeClaim(String name, String namespace) throws ApiException {
+ /*
+ This version of Kubernetes-client/java has bug here.
+ It will trigger exception as in https://github.com/kubernetes-client/java/issues/86
+ but it can still work fine and delete the PVC
+ */
+ final String pvcName = TensorboardUtils.PVC_PREFIX + name;
+
+ try {
+ V1Status result = coreApi.deleteNamespacedPersistentVolumeClaim(
+ pvcName, namespace, "true",
+ null, null, null,
+ null, null
+ );
+ } catch (ApiException e) {
+ LOG.error("Exception when deleting persistent volume claim " + e.getMessage(), e);
+ throw e;
+ } catch (JsonSyntaxException e) {
+ if (e.getCause() instanceof IllegalStateException) {
+ IllegalStateException ise = (IllegalStateException) e.getCause();
+ if (ise.getMessage() != null && ise.getMessage().contains("Expected a string but was BEGIN_OBJECT"))
+ LOG.debug("Catching exception because of issue " +
+ "https://github.com/kubernetes-client/java/issues/86", e);
+ else throw e;
+ }
+ else throw e;
+ }
+ }
+
private String getJobLabelSelector(ExperimentSpec experimentSpec) {
// TODO(JohnTing): SELECTOR_KEY should be obtained from individual models in MLJOB
if (experimentSpec.getMeta().getFramework()
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/ingressroute/SpecRoute.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/ingressroute/SpecRoute.java
index d6208e7..f00e02e 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/ingressroute/SpecRoute.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/ingressroute/SpecRoute.java
@@ -39,6 +39,9 @@ public class SpecRoute {
@SerializedName("services")
private Set<Map<String, Object>> services;
+ @SerializedName("middlewares")
+ private Set<Map<String, String>> middlewares;
+
public String getMatch() {
return match;
}
@@ -62,4 +65,12 @@ public class SpecRoute {
public void setServices(Set<Map<String, Object>> services) {
this.services = services;
}
+
+ public Set<Map<String, String>> getMiddlewares() {
+ return middlewares;
+ }
+
+ public void setMiddlewares(Set<Map<String, String>> middlewares) {
+ this.middlewares = middlewares;
+ }
}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/middlewares/Middlewares.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/middlewares/Middlewares.java
new file mode 100644
index 0000000..1736206
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/middlewares/Middlewares.java
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.middlewares;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1ObjectMeta;
+
+public class Middlewares {
+ // reference: https://doc.traefik.io/traefik/reference/dynamic-configuration/kubernetes-crd/#definitions
+
+ public static final String CRD_MIDDLEWARES_GROUP_V1 = "traefik.containo.us";
+ public static final String CRD_MIDDLEWARES_VERSION_V1 = "v1alpha1";
+ public static final String CRD_APIVERSION_V1 = CRD_MIDDLEWARES_GROUP_V1 +
+ "/" + CRD_MIDDLEWARES_VERSION_V1;
+ public static final String CRD_MIDDLEWARES_KIND_V1 = "Middleware";
+ public static final String CRD_MIDDLEWARES_PLURAL_V1 = "middlewares";
+
+ @SerializedName("apiVersion")
+ private String apiVersion;
+
+ @SerializedName("kind")
+ private String kind;
+
+ @SerializedName("metadata")
+ private V1ObjectMeta metedata;
+
+ @SerializedName("spec")
+ private MiddlewaresSpec spec;
+
+ // transient to avoid being serialized
+ private transient String group;
+
+ private transient String version;
+
+ private transient String plural;
+
+ public Middlewares() {
+ setApiVersion(CRD_APIVERSION_V1);
+ setKind(CRD_MIDDLEWARES_KIND_V1);
+ setPlural(CRD_MIDDLEWARES_PLURAL_V1);
+ setGroup(CRD_MIDDLEWARES_GROUP_V1);
+ setVersion(CRD_MIDDLEWARES_VERSION_V1);
+ }
+
+ public String getApiVersion() {
+ return apiVersion;
+ }
+
+ public void setApiVersion(String apiVersion) {
+ this.apiVersion = apiVersion;
+ }
+
+ public String getKind() {
+ return kind;
+ }
+
+ public void setKind(String kind) {
+ this.kind = kind;
+ }
+
+ public V1ObjectMeta getMetedata() {
+ return metedata;
+ }
+
+ public void setMetedata(V1ObjectMeta metedata) {
+ this.metedata = metedata;
+ }
+
+ public MiddlewaresSpec getSpec() {
+ return spec;
+ }
+
+ public void setSpec(MiddlewaresSpec spec) {
+ this.spec = spec;
+ }
+
+ public String getGroup() {
+ return group;
+ }
+
+ public void setGroup(String group) {
+ this.group = group;
+ }
+
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ public String getPlural() {
+ return plural;
+ }
+
+ public void setPlural(String plural) {
+ this.plural = plural;
+ }
+}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/ingressroute/SpecRoute.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/middlewares/MiddlewaresSpec.java
similarity index 55%
copy from submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/ingressroute/SpecRoute.java
copy to submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/middlewares/MiddlewaresSpec.java
index d6208e7..4f435ac 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/ingressroute/SpecRoute.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/middlewares/MiddlewaresSpec.java
@@ -17,49 +17,25 @@
* under the License.
*/
-package org.apache.submarine.server.submitter.k8s.model.ingressroute;
+package org.apache.submarine.server.submitter.k8s.model.middlewares;
import com.google.gson.annotations.SerializedName;
import java.util.Map;
-import java.util.Set;
-public class SpecRoute {
+public class MiddlewaresSpec {
- public SpecRoute() {
+ @SerializedName("replacePathRegex")
+ private Map<String, String> replacePathRegex;
+ public MiddlewaresSpec() {
}
- @SerializedName("match")
- private String match;
-
- @SerializedName("kind")
- private String kind;
-
- @SerializedName("services")
- private Set<Map<String, Object>> services;
-
- public String getMatch() {
- return match;
- }
-
- public void setMatch(String match) {
- this.match = match;
- }
-
- public String getKind() {
- return kind;
- }
-
- public void setKind(String kind) {
- this.kind = kind;
- }
-
- public Set<Map<String, Object>> getServices() {
- return services;
+ public Map<String, String> getReplacePathRegex() {
+ return replacePathRegex;
}
- public void setServices(Set<Map<String, Object>> services) {
- this.services = services;
+ public void setReplacePathRegex(Map<String, String> replacePathRegex) {
+ this.replacePathRegex = replacePathRegex;
}
}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
index e4190b7..29e3b0b 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
@@ -20,9 +20,11 @@
package org.apache.submarine.server.submitter.k8s.parser;
import io.kubernetes.client.custom.Quantity;
+
import io.kubernetes.client.models.V1Container;
import io.kubernetes.client.models.V1EnvVar;
import io.kubernetes.client.models.V1ObjectMeta;
+import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource;
import io.kubernetes.client.models.V1PodSecurityContext;
import io.kubernetes.client.models.V1PodSpec;
import io.kubernetes.client.models.V1PodTemplateSpec;
@@ -30,7 +32,6 @@ import io.kubernetes.client.models.V1ResourceRequirements;
import io.kubernetes.client.models.V1SecretVolumeSource;
import io.kubernetes.client.models.V1Volume;
import io.kubernetes.client.models.V1VolumeMount;
-
import org.apache.submarine.commons.utils.SubmarineConfVars;
import org.apache.submarine.commons.utils.SubmarineConfiguration;
import org.apache.submarine.server.api.environment.Environment;
@@ -52,6 +53,7 @@ import org.apache.submarine.server.submitter.k8s.model.pytorchjob.PyTorchJobSpec
import org.apache.submarine.server.submitter.k8s.model.tfjob.TFJob;
import org.apache.submarine.server.submitter.k8s.model.tfjob.TFJobReplicaType;
import org.apache.submarine.server.submitter.k8s.model.tfjob.TFJobSpec;
+import org.apache.submarine.server.submitter.k8s.util.TensorboardUtils;
import java.util.ArrayList;
import java.util.Arrays;
@@ -156,6 +158,7 @@ public class ExperimentSpecParser {
V1Container container = new V1Container();
container.setName(experimentSpec.getMeta().getFramework().toLowerCase());
// image
+
if (taskSpec.getImage() != null) {
container.setImage(taskSpec.getImage());
} else {
@@ -174,7 +177,17 @@ public class ExperimentSpecParser {
resources.setLimits(parseResources(taskSpec));
container.setResources(resources);
container.setEnv(parseEnvVars(taskSpec, experimentSpec.getMeta().getEnvVars()));
-
+
+ // volumeMount
+ container.addVolumeMountsItem(new V1VolumeMount().mountPath("/logs").name("volume"));
+
+ // volume
+ final String name = experimentSpec.getMeta().getName();
+ V1Volume podVolume = new V1Volume().name("volume");
+ podVolume.setPersistentVolumeClaim(
+ new V1PersistentVolumeClaimVolumeSource().claimName(TensorboardUtils.PVC_PREFIX + name)
+ );
+ podSpec.addVolumesItem(podVolume);
/**
* Init Git localize Container
*/
@@ -221,7 +234,7 @@ public class ExperimentSpecParser {
podSpec.setSecurityContext(podSecurityContext);
}
}
-
+
V1EnvVar codeEnvVar = new V1EnvVar();
codeEnvVar.setName(AbstractCodeLocalizer.CODE_LOCALIZER_PATH_ENV_VAR);
codeEnvVar.setValue(AbstractCodeLocalizer.CODE_LOCALIZER_PATH);
@@ -233,7 +246,7 @@ public class ExperimentSpecParser {
containers.add(container);
podSpec.setContainers(containers);
-
+
/**
* Init Containers
*/
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParser.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParser.java
new file mode 100644
index 0000000..dfba33f
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParser.java
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+
+import io.kubernetes.client.custom.IntOrString;
+import io.kubernetes.client.models.V1Container;
+import io.kubernetes.client.models.V1ContainerPort;
+import io.kubernetes.client.models.V1Deployment;
+import io.kubernetes.client.models.V1DeploymentSpec;
+import io.kubernetes.client.models.V1LabelSelector;
+import io.kubernetes.client.models.V1ObjectMeta;
+import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+import io.kubernetes.client.models.V1Service;
+import io.kubernetes.client.models.V1ServicePort;
+import io.kubernetes.client.models.V1ServiceSpec;
+import io.kubernetes.client.models.V1Volume;
+import io.kubernetes.client.models.V1VolumeMount;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRoute;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRouteSpec;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.SpecRoute;
+import org.apache.submarine.server.submitter.k8s.util.TensorboardUtils;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TensorboardSpecParser {
+ public static V1Deployment parseDeployment(String name, String image, String routePath, String pvc) {
+
+ V1Deployment deployment = new V1Deployment();
+
+ V1ObjectMeta deploymentMetedata = new V1ObjectMeta();
+ deploymentMetedata.setName(name);
+ deployment.setMetadata(deploymentMetedata);
+ V1DeploymentSpec deploymentSpec = new V1DeploymentSpec();
+ deploymentSpec.setSelector(
+ new V1LabelSelector().matchLabels(Collections.singletonMap("app", name)) // match the template
+ );
+
+ V1PodTemplateSpec deploymentTemplateSpec = new V1PodTemplateSpec();
+ deploymentTemplateSpec.setMetadata(
+ new V1ObjectMeta().labels(Collections.singletonMap("app", name)) // bind to replicaset and service
+ );
+
+ V1PodSpec deploymentTemplatePodSpec = new V1PodSpec();
+
+ V1Container container = new V1Container();
+ container.setName(name);
+ container.setImage(image);
+ container.setCommand(Arrays.asList(
+ "tensorboard", "--logdir=/logs",
+ String.format("--path_prefix=%s", routePath)
+ ));
+ container.setImagePullPolicy("IfNotPresent");
+ container.addPortsItem(new V1ContainerPort().containerPort(TensorboardUtils.DEFAULT_TENSORBOARD_PORT));
+ container.addVolumeMountsItem(new V1VolumeMount().mountPath("/logs").name("volume"));
+ deploymentTemplatePodSpec.addContainersItem(container);
+
+ V1Volume volume = new V1Volume().name("volume");
+ volume.setPersistentVolumeClaim(
+ new V1PersistentVolumeClaimVolumeSource().claimName(pvc)
+ );
+ deploymentTemplatePodSpec.addVolumesItem(volume);
+
+ deploymentTemplateSpec.setSpec(deploymentTemplatePodSpec);
+
+ deploymentSpec.setTemplate(deploymentTemplateSpec);
+
+ deployment.setSpec(deploymentSpec);
+
+ return deployment;
+ }
+
+ public static V1Service parseService(String svcName, String podName) {
+ V1Service svc = new V1Service();
+ svc.metadata(new V1ObjectMeta().name(svcName));
+
+ V1ServiceSpec svcSpec = new V1ServiceSpec();
+ svcSpec.setSelector(Collections.singletonMap("app", podName)); // bind to pod
+ svcSpec.addPortsItem(new V1ServicePort().protocol("TCP").targetPort(
+ new IntOrString(TensorboardUtils.DEFAULT_TENSORBOARD_PORT)).port(TensorboardUtils.SERVICE_PORT));
+ svc.setSpec(svcSpec);
+ return svc;
+ }
+
+ public static IngressRoute parseIngressRoute(String ingressName, String namespace,
+ String routePath, String svcName) {
+
+ IngressRoute ingressRoute = new IngressRoute();
+ ingressRoute.setMetadata(
+ new V1ObjectMeta().name(ingressName).namespace((namespace))
+ );
+
+ IngressRouteSpec ingressRouteSpec = new IngressRouteSpec();
+ ingressRouteSpec.setEntryPoints(new HashSet<>(Collections.singletonList("web")));
+ SpecRoute specRoute = new SpecRoute();
+ specRoute.setKind("Rule");
+ specRoute.setMatch(String.format("PathPrefix(`%s`)", routePath));
+
+ Map<String, Object> service = new HashMap<String, Object>() {{
+ put("kind", "Service");
+ put("name", svcName);
+ put("port", TensorboardUtils.SERVICE_PORT);
+ put("namespace", namespace);
+ }};
+
+ specRoute.setServices(new HashSet<Map<String, Object>>() {{
+ add(service);
+ }});
+
+ ingressRouteSpec.setRoutes(new HashSet<SpecRoute>() {{
+ add(specRoute);
+ }});
+
+ ingressRoute.setSpec(ingressRouteSpec);
+
+ return ingressRoute;
+ }
+
+}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java
new file mode 100644
index 0000000..6621c67
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+
+import io.kubernetes.client.custom.Quantity;
+import io.kubernetes.client.models.V1HostPathVolumeSource;
+import io.kubernetes.client.models.V1ObjectMeta;
+import io.kubernetes.client.models.V1PersistentVolume;
+import io.kubernetes.client.models.V1PersistentVolumeClaim;
+import io.kubernetes.client.models.V1PersistentVolumeClaimSpec;
+import io.kubernetes.client.models.V1PersistentVolumeSpec;
+import io.kubernetes.client.models.V1ResourceRequirements;
+
+import java.util.Collections;
+
+public class VolumeSpecParser {
+ public static V1PersistentVolume parsePersistentVolume(String name, String hostPath, String storage) {
+ V1PersistentVolume pv = new V1PersistentVolume();
+ /*
+ Required value
+ 1. metadata.name
+ 2. spec.accessModes
+ 3. spec.capacity
+ 4. spec.storageClassName
+ Others are not necessary
+ */
+
+ V1ObjectMeta pvMetadata = new V1ObjectMeta();
+ pvMetadata.setName(name);
+ pv.setMetadata(pvMetadata);
+
+ V1PersistentVolumeSpec pvSpec = new V1PersistentVolumeSpec();
+ pvSpec.setAccessModes(Collections.singletonList("ReadWriteMany"));
+ pvSpec.setCapacity(Collections.singletonMap("storage", new Quantity(storage)));
+ pvSpec.setStorageClassName("standard");
+ pvSpec.setHostPath(new V1HostPathVolumeSource().path(hostPath));
+ pv.setSpec(pvSpec);
+
+ return pv;
+ }
+
+ public static V1PersistentVolumeClaim parsePersistentVolumeClaim(
+ String name, String volume, String storage) {
+ V1PersistentVolumeClaim pvc = new V1PersistentVolumeClaim();
+ /*
+ Required value
+ 1. metadata.name
+ 2. spec.accessModes
+ 3. spec.storageClassName
+ 4. spec.resources
+ Others are not necessary
+ */
+
+ V1ObjectMeta pvcMetadata = new V1ObjectMeta();
+ pvcMetadata.setName(name);
+ pvc.setMetadata(pvcMetadata);
+
+ V1PersistentVolumeClaimSpec pvcSpec = new V1PersistentVolumeClaimSpec();
+ pvcSpec.setAccessModes(Collections.singletonList("ReadWriteMany"));
+ pvcSpec.setStorageClassName("standard");
+ pvcSpec.setResources(new V1ResourceRequirements().putRequestsItem("storage", new Quantity(storage)));
+ pvcSpec.setVolumeName(volume); // bind pvc to specific pv
+ pvc.setSpec(pvcSpec);
+
+ return pvc;
+ }
+
+}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/TensorboardUtils.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/TensorboardUtils.java
new file mode 100644
index 0000000..a7d8c9f
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/TensorboardUtils.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.util;
+
+public class TensorboardUtils {
+ /*
+ Prefix constants
+ */
+ public static final String PV_PREFIX = "tfboard-pv-";
+ public static final String HOST_PREFIX = "/tmp/tfboard-logs/";
+ public static final String STORAGE = "1Gi";
+ public static final String PVC_PREFIX = "tfboard-pvc-";
+ public static final String DEPLOY_PREFIX = "tfboard-";
+ public static final String POD_PREFIX = "tfboard-";
+ public static final String IMAGE_NAME = "tensorflow/tensorflow:1.11.0";
+ public static final String SVC_PREFIX = "tfboard-svc-";
+ public static final String INGRESS_PREFIX = "tfboard-ingressroute";
+ public static final String PATH_PREFIX = "/tfboard-";
+ public static final Integer DEFAULT_TENSORBOARD_PORT = 6006;
+ public static final Integer SERVICE_PORT = 8080;
+}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/K8SJobSubmitterTest.java b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/K8SJobSubmitterTest.java
index a27f88e..4add1e0 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/K8SJobSubmitterTest.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/K8SJobSubmitterTest.java
@@ -29,6 +29,8 @@ import org.apache.submarine.server.api.spec.ExperimentSpec;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* We have two ways to test submitter for K8s cluster, local and travis CI.
@@ -47,7 +49,7 @@ import org.junit.Test;
* Travis: See '.travis.yml'
*/
public class K8SJobSubmitterTest extends SpecBuilder {
-
+ private static final Logger LOG = LoggerFactory.getLogger(K8SJobSubmitterTest.class);
private K8sSubmitter submitter;
@Before
@@ -70,6 +72,18 @@ public class K8SJobSubmitterTest extends SpecBuilder {
run(spec);
}
+ @Test
+ public void testCreateTFJob() throws IOException, URISyntaxException {
+ ExperimentSpec spec = (ExperimentSpec) buildFromJsonFile(ExperimentSpec.class, tfTfboardJobwReqFile);
+ Experiment experiment = submitter.createExperiment(spec);
+ }
+
+ @Test
+ public void testDeleteTFJob() throws IOException, URISyntaxException {
+ ExperimentSpec spec = (ExperimentSpec) buildFromJsonFile(ExperimentSpec.class, tfTfboardJobwReqFile);
+ Experiment experiment = submitter.deleteExperiment(spec);
+ }
+
private void run(ExperimentSpec spec) throws SubmarineRuntimeException {
// create
Experiment experimentCreated = submitter.createExperiment(spec);
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java
index d94a161..eb41aca 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java
@@ -44,6 +44,7 @@ public abstract class SpecBuilder {
"/pytorch_job_req_http_git_code_localizer.json";
protected final String pytorchJobWithSSHGitCodeLocalizerFile =
"/pytorch_job_req_ssh_git_code_localizer.json";
+ protected final String tfTfboardJobwReqFile = "/tf_tfboard_mnist_req.json";
protected Object buildFromJsonFile(Object obj, String filePath) throws IOException,
URISyntaxException {
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParserTest.java b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParserTest.java
new file mode 100644
index 0000000..2503322
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParserTest.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+
+import io.kubernetes.client.models.V1Deployment;
+import io.kubernetes.client.models.V1Service;
+import junit.framework.TestCase;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRoute;
+import org.apache.submarine.server.submitter.k8s.util.TensorboardUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TensorboardSpecParserTest extends TestCase {
+
+ @Test
+ public void testParseDeployment() {
+ final String id = "123456789";
+
+ final String name = TensorboardUtils.DEPLOY_PREFIX + id;
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String routePath = TensorboardUtils.PATH_PREFIX + id;
+ final String pvc = TensorboardUtils.PVC_PREFIX + id;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(name, image, routePath, pvc);
+
+ Assert.assertEquals(name, deployment.getMetadata().getName());
+ Assert.assertEquals(image,
+ deployment.getSpec().getTemplate().getSpec().getContainers().get(0).getImage());
+
+ Assert.assertEquals(pvc,
+ deployment.getSpec().getTemplate().getSpec().getVolumes()
+ .get(0).getPersistentVolumeClaim().getClaimName());
+ }
+
+ @Test
+ public void testParseService() {
+ final String id = "123456789";
+
+ final String svcName = TensorboardUtils.SVC_PREFIX + id;
+ final String podName = TensorboardUtils.DEPLOY_PREFIX + id;
+
+ V1Service svc = TensorboardSpecParser.parseService(svcName, podName);
+
+ Assert.assertEquals(svcName, svc.getMetadata().getName());
+ Assert.assertEquals(podName, svc.getSpec().getSelector().get("app"));
+ }
+
+ @Test
+ public void testParseIngressRoute() {
+ final String id = "123456789";
+ final String namespace = "default";
+
+ final String ingressName = TensorboardUtils.INGRESS_PREFIX + id;
+ final String routePath = TensorboardUtils.PATH_PREFIX + id;
+ final String svcName = TensorboardUtils.SVC_PREFIX + id;
+
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingressName, namespace, routePath, svcName
+ );
+
+ Assert.assertEquals(ingressRoute.getMetadata().getName(), ingressName);
+ }
+}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParserTest.java b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParserTest.java
new file mode 100644
index 0000000..afacadb
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParserTest.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+import io.kubernetes.client.custom.Quantity;
+import io.kubernetes.client.models.V1PersistentVolume;
+import io.kubernetes.client.models.V1PersistentVolumeClaim;
+import org.apache.submarine.server.submitter.k8s.util.TensorboardUtils;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VolumeSpecParserTest {
+ private static final Logger LOG = LoggerFactory.getLogger(VolumeSpecParserTest.class);
+
+ @Test
+ public void testParsePersistentVolume() {
+ final String id = "123456789";
+
+ final String name = TensorboardUtils.PV_PREFIX + id;
+ final String hostPath = TensorboardUtils.HOST_PREFIX + id;
+ final String storage = TensorboardUtils.STORAGE;
+
+ V1PersistentVolume pv = VolumeSpecParser.parsePersistentVolume(name, hostPath, storage);
+ LOG.info(pv.toString());
+
+ Assert.assertEquals(name, pv.getMetadata().getName());
+ Assert.assertEquals(hostPath, pv.getSpec().getHostPath().getPath());
+ Assert.assertEquals(new Quantity(storage), pv.getSpec().getCapacity().get("storage"));
+ }
+
+ @Test
+ public void testParsePersistentVolumeClaim() {
+ final String id = "123456789";
+
+ final String name = TensorboardUtils.PVC_PREFIX + id;
+ final String volume = TensorboardUtils.PV_PREFIX + id;
+ final String storage = TensorboardUtils.STORAGE;
+
+ V1PersistentVolumeClaim pvc = VolumeSpecParser.parsePersistentVolumeClaim(name, volume, storage);
+
+ LOG.info(pvc.toString());
+ Assert.assertEquals(name, pvc.getMetadata().getName());
+ Assert.assertEquals(volume, pvc.getSpec().getVolumeName());
+ Assert.assertEquals(new Quantity(storage), pvc.getSpec().getResources().getRequests().get("storage"));
+ }
+}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/resources/tf_tfboard_mnist_req.json b/submarine-server/server-submitter/submitter-k8s/src/test/resources/tf_tfboard_mnist_req.json
new file mode 100644
index 0000000..0a5889e
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/resources/tf_tfboard_mnist_req.json
@@ -0,0 +1,20 @@
+{
+ "meta": {
+ "name": "tensorflow-tensorboard-dist-mnist",
+ "namespace": "default",
+ "framework": "TensorFlow",
+ "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/logs/mylog --learning_rate=0.01 --batch_size=20",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "image": "apache/submarine:tf-mnist-with-summaries-1.0"
+ },
+ "spec": {
+ "Worker": {
+ "replicas": 1,
+ "resources": "cpu=1,memory=1024M"
+ }
+ }
+}
diff --git a/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java b/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java
index c83e055..736fd64 100644
--- a/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java
+++ b/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java
@@ -46,6 +46,7 @@ import org.apache.submarine.server.response.JsonResponse;
import org.apache.submarine.server.rest.RestConstants;
import org.joda.time.DateTime;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.slf4j.Logger;
@@ -102,6 +103,11 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
kfOperatorMap.put("pytorch", new KfOperator("v1", "pytorchjobs"));
}
+ @Before
+ public void setUp() throws Exception {
+ Thread.sleep(5000); // timeout for each case, ensuring k8s-client has enough time to delete resoures
+ }
+
@Test
public void testJobServerPing() throws IOException {
GetMethod response = httpGet("/api/" + RestConstants.V1 + "/"
@@ -186,7 +192,7 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
String patchBody = loadContent("tensorflow/tf-mnist-with-ssh-git-code-localizer-req.json");
run(body, patchBody, "application/json");
}
-
+
private void run(String body, String patchBody, String contentType) throws Exception {
// create
LOG.info("Create training job by Job REST API");
@@ -219,7 +225,8 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
// https://tools.ietf.org/html/rfc5789
// delete
- DeleteMethod deleteMethod = httpDelete(BASE_API_PATH + "/" + createdExperiment.getExperimentId().toString());
+ DeleteMethod deleteMethod = httpDelete(
+ BASE_API_PATH + "/" + createdExperiment.getExperimentId().toString());
Assert.assertEquals(Response.Status.OK.getStatusCode(), deleteMethod.getStatusCode());
json = deleteMethod.getResponseBodyAsString();
@@ -296,7 +303,8 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
assertK8sResultEquals(env, createdJob);
}
- private void verifyGetJobApiResult(Experiment createdExperiment, Experiment foundExperiment) throws Exception {
+ private void verifyGetJobApiResult(
+ Experiment createdExperiment, Experiment foundExperiment) throws Exception {
Assert.assertEquals(createdExperiment.getExperimentId(), foundExperiment.getExperimentId());
Assert.assertEquals(createdExperiment.getUid(), foundExperiment.getUid());
Assert.assertEquals(createdExperiment.getName(), foundExperiment.getName());
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org