You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@submarine.apache.org by GitBox <gi...@apache.org> on 2020/12/27 10:24:04 UTC
[GitHub] [submarine] ByronHsu opened a new pull request #484: SUBMARINE 701. 0.6.0 New Feature: Support Tensorboard in Experiment
ByronHsu opened a new pull request #484:
URL: https://github.com/apache/submarine/pull/484
### What is this PR for?
Support new feature in 0.6.0: tensorboard integration.
- Usage
1. Create a job request that uses tensorboard
2. Write tensorboard log to `/logs/mylog` (The subpath is required due to this issue in tensorflow [https://github.com/kubeflow/tf-operator/issues/1053](https://github.com/kubeflow/tf-operator/issues/1053). We cannot directly write log file to mountPath.)
3. Link to `http://<host>:<ip>/tfboard-${job-name}`, and you can monitor the tensorboard with ease!
- Implementation
When creating a new job, the backend will not only create original experiment but also several k8s resources required in tensorboard
The resources can be classified into two categories:
1. Storage
2. Tensorboard serving
**Storage**
The resources required for storage are **persistent volume** and **persistent volume claim**.
I set the storage path of persistent volume on host path, and mount this path to MLjob (enable job to generate logs to volume) and Tensorboard (enable tfboard to access logs).
**Tensorboard Serving**
The resources required here are **deployments, service, and ingressroute**.
I create the tensorboard apps with deployments and service, and then redirect it to custom path with the help of ingressroute.
- Example
- tensorboard-example.json
```bash
{
"meta": {
"name": "tensorflow-dist-mnist-byron-1234",
"namespace": "default",
"framework": "TensorFlow",
"cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/logs/mylog --learning_rate=0.01 --batch_size=20",
"envVars": {
"ENV_1": "ENV1"
}
},
"environment": {
"image": "apache/submarine:tf-mnist-with-summaries-1.0"
},
"spec": {
"Worker": {
"replicas": 1,
"resources": "cpu=1,memory=1024M"
}
}
}
```
### What type of PR is it?
[Feature]
### Todos
- [ ] Frontend support
- [ ] The logs of job cannot be written directly on the mountPath (As describe in above). We should fix this problem.
- [ ] Make log path configurable (Currently, it is hard-coded as `/logs` )
- [ ] Support smb-server for shared storage
### What is the Jira issue?
https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-701
### How should this be tested?
https://travis-ci.org/github/ByronHsu/submarine/jobs/751658488
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] ByronHsu commented on pull request #484: SUBMARINE-701. Support Tensorboard in Experiment
Posted by GitBox <gi...@apache.org>.
ByronHsu commented on pull request #484:
URL: https://github.com/apache/submarine/pull/484#issuecomment-751607892
@lowc1012 @jiwq @xunliu @pingsutw please help me review the code. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] lowc1012 commented on a change in pull request #484: SUBMARINE-701. Support Tensorboard in Experiment
Posted by GitBox <gi...@apache.org>.
lowc1012 commented on a change in pull request #484:
URL: https://github.com/apache/submarine/pull/484#discussion_r549106867
##########
File path: helm-charts/submarine/templates/rbac.yaml
##########
@@ -56,6 +56,14 @@ rules:
resources:
- pods
- pods/log
+ - services
+ - persistentvolumes
+ - persistentvolumeclaims
+ verbs:
+ - '*'
+- apiGroups: ["extensions", "apps"]
+ resources:
+ - deployments
Review comment:
Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served since k8s v1.16
https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] lowc1012 commented on a change in pull request #484: SUBMARINE-701. Support Tensorboard in Experiment
Posted by GitBox <gi...@apache.org>.
lowc1012 commented on a change in pull request #484:
URL: https://github.com/apache/submarine/pull/484#discussion_r549106867
##########
File path: helm-charts/submarine/templates/rbac.yaml
##########
@@ -56,6 +56,14 @@ rules:
resources:
- pods
- pods/log
+ - services
+ - persistentvolumes
+ - persistentvolumeclaims
+ verbs:
+ - '*'
+- apiGroups: ["extensions", "apps"]
+ resources:
+ - deployments
Review comment:
Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served since k8s v1.16
https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] lowc1012 commented on a change in pull request #484: SUBMARINE-701. 0.6.0 New Feature: Support Tensorboard in Experiment
Posted by GitBox <gi...@apache.org>.
lowc1012 commented on a change in pull request #484:
URL: https://github.com/apache/submarine/pull/484#discussion_r549106867
##########
File path: helm-charts/submarine/templates/rbac.yaml
##########
@@ -56,6 +56,14 @@ rules:
resources:
- pods
- pods/log
+ - services
+ - persistentvolumes
+ - persistentvolumeclaims
+ verbs:
+ - '*'
+- apiGroups: ["extensions", "apps"]
+ resources:
+ - deployments
Review comment:
The Deployment is in API version "apps/v1" after k8s v.1.9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #484: SUBMARINE-701. Support Tensorboard in Experiment
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #484:
URL: https://github.com/apache/submarine/pull/484#discussion_r549913986
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -108,13 +118,25 @@ public void initialize(SubmarineConfiguration conf) {
if (coreApi == null) {
coreApi = new CoreV1Api(client);
}
+ if (appsV1Api == null) {
+ appsV1Api = new AppsV1Api();
+ }
+
+ client.setDebugging(true);
}
@Override
public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeException {
Experiment experiment;
+ final String id = spec.getMeta().getName(); // spec.getMeta().getEnvVars().get(RestConstants.JOB_ID);
Review comment:
```suggestion
final String name = spec.getMeta().getName();
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -108,13 +118,25 @@ public void initialize(SubmarineConfiguration conf) {
if (coreApi == null) {
coreApi = new CoreV1Api(client);
}
+ if (appsV1Api == null) {
+ appsV1Api = new AppsV1Api();
+ }
+
+ client.setDebugging(true);
}
@Override
public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeException {
Experiment experiment;
+ final String id = spec.getMeta().getName(); // spec.getMeta().getEnvVars().get(RestConstants.JOB_ID);
+
try {
MLJob mlJob = ExperimentSpecParser.parseJob(spec);
+
+ createTFBoardPersistentVolume(id);
+ createTFBoardPersistentVolumeClaim(id, spec.getMeta().getNamespace());
+ createTFBoard(id, spec.getMeta().getNamespace());
Review comment:
```suggestion
createTFBoardPersistentVolume(name);
createTFBoardPersistentVolumeClaim(name, spec.getMeta().getNamespace());
createTFBoard(name, spec.getMeta().getNamespace());
```
##########
File path: submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
##########
@@ -101,6 +101,9 @@ public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeE
Experiment experiment = submitter.createExperiment(spec);
experiment.setExperimentId(id);
+ // importing tensorboardUtils will cause dependency circle. Hard-code it as a temporary solution
Review comment:
```suggestion
// [TODO] Importing tensorboardUtils will cause a dependency circle. Hard-code it as a temporary solution
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParser.java
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+
+import io.kubernetes.client.custom.IntOrString;
+import io.kubernetes.client.models.V1Container;
+import io.kubernetes.client.models.V1ContainerPort;
+import io.kubernetes.client.models.V1Deployment;
+import io.kubernetes.client.models.V1DeploymentSpec;
+import io.kubernetes.client.models.V1LabelSelector;
+import io.kubernetes.client.models.V1ObjectMeta;
+import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+import io.kubernetes.client.models.V1Service;
+import io.kubernetes.client.models.V1ServicePort;
+import io.kubernetes.client.models.V1ServiceSpec;
+import io.kubernetes.client.models.V1Volume;
+import io.kubernetes.client.models.V1VolumeMount;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRoute;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRouteSpec;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.SpecRoute;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TensorboardSpecParser {
+ public static V1Deployment parseDeployment(String name, String image, String route_path, String pvc) {
+
+ /*
+ [start] Deployment
+ */
+ V1Deployment deployment = new V1Deployment();
+
+ // [start] deployment - metadata
+ V1ObjectMeta deployment_metedata = new V1ObjectMeta();
+ deployment_metedata.setName(name);
+ deployment.setMetadata(deployment_metedata);
+ // [end] deployment - metadata
+
+ // [start] deployment - spec
+ V1DeploymentSpec deployment_spec = new V1DeploymentSpec();
+ deployment_spec.setSelector(
+ new V1LabelSelector().matchLabels(Collections.singletonMap("app", name)) // match the template
+ );
+
+ // [start] deployment - spec - template
+ V1PodTemplateSpec deployment_template_spec = new V1PodTemplateSpec();
+ deployment_template_spec.setMetadata(
+ new V1ObjectMeta().labels(Collections.singletonMap("app", name)) // bind to replicaset and service
+ );
+
+ // [start] deployment - spec - template - podspec
+ V1PodSpec deployment_template_pod_spec = new V1PodSpec();
+
+ // [start] deployment - spec - template - podspec - container
+ V1Container container = new V1Container();
+ container.setName(name);
+ container.setImage(image);
+ container.setCommand(Arrays.asList(
+ "tensorboard", "--logdir=/logs",
+ String.format("--path_prefix=%s", route_path)
+ ));
+ container.setImagePullPolicy("IfNotPresent");
+ container.addPortsItem(new V1ContainerPort().containerPort(6006));
+ container.addVolumeMountsItem(new V1VolumeMount().mountPath("/logs").name("volume"));
+ deployment_template_pod_spec.addContainersItem(container);
+ // [end] deployment - spec - template - podspec - container
+
+ // [start] deployment - spec - template - podspec - volume
+ V1Volume volume = new V1Volume().name("volume");
+ volume.setPersistentVolumeClaim(
+ new V1PersistentVolumeClaimVolumeSource().claimName(pvc)
+ );
+ deployment_template_pod_spec.addVolumesItem(volume);
+ // [end] deployment - spec - template - podspec - volume
+
+ deployment_template_spec.setSpec(deployment_template_pod_spec);
+ // [end] deployment - spec - template - podspec
+
+ deployment_spec.setTemplate(deployment_template_spec);
+ // [end] deployment - spec - template
+
+ deployment.setSpec(deployment_spec);
+ // [end] deployment - spec
+ return deployment;
+ }
+
+ public static V1Service parseService(String svc_name, String pod_name) {
+ V1Service svc = new V1Service();
+ svc.metadata(new V1ObjectMeta().name(svc_name));
+
+ V1ServiceSpec svc_spec = new V1ServiceSpec();
+ svc_spec.setSelector(Collections.singletonMap("app", pod_name)); // bind to pod
+ svc_spec.addPortsItem(new V1ServicePort().protocol("TCP").targetPort(new IntOrString(6006)).port(8080));
Review comment:
You could create constant variables in `TensorboardUtils`
`final String DEFAULT_TENSORBOARD_PORT = "6006";`
`final String EXPOSE_TENSORBOARD_PORT= "8080";`
##########
File path: submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/K8SJobSubmitterTest.java
##########
@@ -70,7 +72,22 @@ public void testRunTFJobPerRequest() throws URISyntaxException,
run(spec);
}
+ @Test
+ public void testCreateTFJob() throws IOException, URISyntaxException {
+ ExperimentSpec spec = (ExperimentSpec) buildFromJsonFile(ExperimentSpec.class, tfTfboardJobwReqFile);
+ Experiment experiment = submitter.createExperiment(spec);
+ Assert.assertTrue(true);
+ }
+
+ @Test
+ public void testDeleteTFJob() throws IOException, URISyntaxException {
+ ExperimentSpec spec = (ExperimentSpec) buildFromJsonFile(ExperimentSpec.class, tfTfboardJobwReqFile);
+ Experiment experiment = submitter.deleteExperiment(spec);
+ Assert.assertTrue(true);
+ }
+
private void run(ExperimentSpec spec) throws SubmarineRuntimeException {
+ System.out.println(spec.toString());
Review comment:
```suggestion
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -165,8 +187,15 @@ public Experiment patchExperiment(ExperimentSpec spec) throws SubmarineRuntimeEx
@Override
public Experiment deleteExperiment(ExperimentSpec spec) throws SubmarineRuntimeException {
Experiment experiment;
+ final String id = spec.getMeta().getName(); // spec.getMeta().getEnvVars().get(RestConstants.JOB_ID);
Review comment:
```suggestion
final String name = spec.getMeta().getName(); // spec.getMeta().getEnvVars().get(RestConstants.JOB_ID);
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -320,6 +349,161 @@ public Notebook deleteNotebook(NotebookSpec spec) throws SubmarineRuntimeExcepti
return notebookList;
}
+ public void createTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.createNamespacedDeployment(namespace, deployment, "true", null, null);
+ coreApi.createNamespacedService(namespace, svc, "true", null, null);
+ api.createNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(),
+ ingressRoute.getPlural(), ingressRoute, "true");
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
+ throw e;
+ }
+ }
+
+ public void deleteTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.deleteNamespacedDeployment(deploy_name, namespace, "true",
+ null, null, null, null, null);
+ coreApi.deleteNamespacedService(svc_name, namespace, "true",
+ null, null, null, null, null);
+ api.deleteNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(), ingressRoute.getPlural(), ingress_name,
+ new V1DeleteOptionsBuilder().withApiVersion(ingressRoute.getApiVersion()).build(),
+ null, null, null);
+
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
Review comment:
```suggestion
LOG.error("Exception when creating TensorBoard " + e.getMessage(), e);
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -165,8 +187,15 @@ public Experiment patchExperiment(ExperimentSpec spec) throws SubmarineRuntimeEx
@Override
public Experiment deleteExperiment(ExperimentSpec spec) throws SubmarineRuntimeException {
Experiment experiment;
+ final String id = spec.getMeta().getName(); // spec.getMeta().getEnvVars().get(RestConstants.JOB_ID);
+
try {
MLJob mlJob = ExperimentSpecParser.parseJob(spec);
+
+ deleteTFBoardPersistentVolume(id);
+ deleteTFBoardPersistentVolumeClaim(id, spec.getMeta().getNamespace());
+ deleteTFBoard(id, spec.getMeta().getNamespace());
Review comment:
```suggestion
deleteTFBoardPersistentVolume(name);
deleteTFBoardPersistentVolumeClaim(name, spec.getMeta().getNamespace());
deleteTFBoard(name, spec.getMeta().getNamespace());
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/test/resources/tf_tfboard_mnist_req.json
##########
@@ -0,0 +1,20 @@
+{
+ "meta": {
+ "name": "tensorflow-tensorboard-dist-mnist",
+ "namespace": "default",
+ "framework": "TensorFlow",
+ "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/logs/mylog --learning_rate=0.01 --batch_size=20",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "image": "apache/submarine:tf-mnist-with-summaries-1.0"
+ },
+ "spec": {
+ "Worker": {
+ "replicas": 1,
+ "resources": ""
Review comment:
```suggestion
"resources": "cpu=1,memory=1024M"
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -320,6 +349,161 @@ public Notebook deleteNotebook(NotebookSpec spec) throws SubmarineRuntimeExcepti
return notebookList;
}
+ public void createTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.createNamespacedDeployment(namespace, deployment, "true", null, null);
+ coreApi.createNamespacedService(namespace, svc, "true", null, null);
+ api.createNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(),
+ ingressRoute.getPlural(), ingressRoute, "true");
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
Review comment:
```suggestion
LOG.error("Exception when creating TensorBoard " + e.getMessage(), e);
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParser.java
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+
+import io.kubernetes.client.custom.IntOrString;
+import io.kubernetes.client.models.V1Container;
+import io.kubernetes.client.models.V1ContainerPort;
+import io.kubernetes.client.models.V1Deployment;
+import io.kubernetes.client.models.V1DeploymentSpec;
+import io.kubernetes.client.models.V1LabelSelector;
+import io.kubernetes.client.models.V1ObjectMeta;
+import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+import io.kubernetes.client.models.V1Service;
+import io.kubernetes.client.models.V1ServicePort;
+import io.kubernetes.client.models.V1ServiceSpec;
+import io.kubernetes.client.models.V1Volume;
+import io.kubernetes.client.models.V1VolumeMount;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRoute;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRouteSpec;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.SpecRoute;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TensorboardSpecParser {
+ public static V1Deployment parseDeployment(String name, String image, String route_path, String pvc) {
+
+ /*
+ [start] Deployment
+ */
+ V1Deployment deployment = new V1Deployment();
+
+ // [start] deployment - metadata
+ V1ObjectMeta deployment_metedata = new V1ObjectMeta();
+ deployment_metedata.setName(name);
+ deployment.setMetadata(deployment_metedata);
+ // [end] deployment - metadata
+
+ // [start] deployment - spec
+ V1DeploymentSpec deployment_spec = new V1DeploymentSpec();
+ deployment_spec.setSelector(
+ new V1LabelSelector().matchLabels(Collections.singletonMap("app", name)) // match the template
+ );
+
+ // [start] deployment - spec - template
+ V1PodTemplateSpec deployment_template_spec = new V1PodTemplateSpec();
+ deployment_template_spec.setMetadata(
+ new V1ObjectMeta().labels(Collections.singletonMap("app", name)) // bind to replicaset and service
+ );
+
+ // [start] deployment - spec - template - podspec
+ V1PodSpec deployment_template_pod_spec = new V1PodSpec();
+
+ // [start] deployment - spec - template - podspec - container
+ V1Container container = new V1Container();
+ container.setName(name);
+ container.setImage(image);
+ container.setCommand(Arrays.asList(
+ "tensorboard", "--logdir=/logs",
+ String.format("--path_prefix=%s", route_path)
+ ));
+ container.setImagePullPolicy("IfNotPresent");
+ container.addPortsItem(new V1ContainerPort().containerPort(6006));
+ container.addVolumeMountsItem(new V1VolumeMount().mountPath("/logs").name("volume"));
+ deployment_template_pod_spec.addContainersItem(container);
+ // [end] deployment - spec - template - podspec - container
+
+ // [start] deployment - spec - template - podspec - volume
+ V1Volume volume = new V1Volume().name("volume");
+ volume.setPersistentVolumeClaim(
+ new V1PersistentVolumeClaimVolumeSource().claimName(pvc)
+ );
+ deployment_template_pod_spec.addVolumesItem(volume);
+ // [end] deployment - spec - template - podspec - volume
+
+ deployment_template_spec.setSpec(deployment_template_pod_spec);
+ // [end] deployment - spec - template - podspec
+
+ deployment_spec.setTemplate(deployment_template_spec);
+ // [end] deployment - spec - template
+
+ deployment.setSpec(deployment_spec);
+ // [end] deployment - spec
+ return deployment;
+ }
+
+ public static V1Service parseService(String svc_name, String pod_name) {
+ V1Service svc = new V1Service();
+ svc.metadata(new V1ObjectMeta().name(svc_name));
+
+ V1ServiceSpec svc_spec = new V1ServiceSpec();
+ svc_spec.setSelector(Collections.singletonMap("app", pod_name)); // bind to pod
+ svc_spec.addPortsItem(new V1ServicePort().protocol("TCP").targetPort(new IntOrString(6006)).port(8080));
+ svc.setSpec(svc_spec);
+ return svc;
+ }
+
+ public static IngressRoute parseIngressRoute(String ingress_name, String namespace,
+ String route_path, String svc_name) {
+
+ IngressRoute ingressRoute = new IngressRoute();
+ ingressRoute.setMetadata(
+ new V1ObjectMeta().name(ingress_name).namespace((namespace))
+ );
+
+ IngressRouteSpec ingressRoute_spec = new IngressRouteSpec();
+ ingressRoute_spec.setEntryPoints(new HashSet<>(Collections.singletonList("web")));
+ SpecRoute spec_route = new SpecRoute();
+ spec_route.setKind("Rule");
+ spec_route.setMatch(String.format("PathPrefix(`%s`)", route_path));
+
+ Map<String, Object> service = new HashMap<String, Object>() {{
+ put("kind", "Service");
+ put("name", svc_name);
+ put("port", 8080);
Review comment:
```suggestion
put("port", EXPOSE_TENSORBOARD_PORT);
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/TensorboardSpecParser.java
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+
+import io.kubernetes.client.custom.IntOrString;
+import io.kubernetes.client.models.V1Container;
+import io.kubernetes.client.models.V1ContainerPort;
+import io.kubernetes.client.models.V1Deployment;
+import io.kubernetes.client.models.V1DeploymentSpec;
+import io.kubernetes.client.models.V1LabelSelector;
+import io.kubernetes.client.models.V1ObjectMeta;
+import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+import io.kubernetes.client.models.V1Service;
+import io.kubernetes.client.models.V1ServicePort;
+import io.kubernetes.client.models.V1ServiceSpec;
+import io.kubernetes.client.models.V1Volume;
+import io.kubernetes.client.models.V1VolumeMount;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRoute;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.IngressRouteSpec;
+import org.apache.submarine.server.submitter.k8s.model.ingressroute.SpecRoute;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+public class TensorboardSpecParser {
+ public static V1Deployment parseDeployment(String name, String image, String route_path, String pvc) {
+
+ /*
+ [start] Deployment
Review comment:
The comments here seem unnecessary, could remove them?
This code can be easily understood
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
##########
@@ -174,7 +176,18 @@ private static V1PodTemplateSpec parseTemplateSpec(
resources.setLimits(parseResources(taskSpec));
container.setResources(resources);
container.setEnv(parseEnvVars(taskSpec, experimentSpec.getMeta().getEnvVars()));
-
+
+ // volumeMount
+ container.addVolumeMountsItem(new V1VolumeMount().mountPath("/logs").name("volume"));
+
+ // volume
+ final String PVC_NAME_PREFIX = "tfboard-pvc-";
Review comment:
We could use `PVC_NAME_PREFIX` in `TensorboardUtils`
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -320,6 +349,161 @@ public Notebook deleteNotebook(NotebookSpec spec) throws SubmarineRuntimeExcepti
return notebookList;
}
+ public void createTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.createNamespacedDeployment(namespace, deployment, "true", null, null);
+ coreApi.createNamespacedService(namespace, svc, "true", null, null);
+ api.createNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(),
+ ingressRoute.getPlural(), ingressRoute, "true");
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
+ throw e;
+ }
+ }
+
+ public void deleteTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.deleteNamespacedDeployment(deploy_name, namespace, "true",
+ null, null, null, null, null);
+ coreApi.deleteNamespacedService(svc_name, namespace, "true",
+ null, null, null, null, null);
+ api.deleteNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(), ingressRoute.getPlural(), ingress_name,
+ new V1DeleteOptionsBuilder().withApiVersion(ingressRoute.getApiVersion()).build(),
+ null, null, null);
+
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
+ throw e;
+ }
+ }
+
+ public void createTFBoardPersistentVolume(String name) throws ApiException {
+ final String pv_name = TensorboardUtils.PV_PREFIX + name;
+ final String host_path = TensorboardUtils.HOST_PREFIX + name;
+ final String storage = TensorboardUtils.STORAGE;
+
+ V1PersistentVolume pv = VolumeSpecParser.parsePersistentVolume(pv_name, host_path, storage);
+
+ try {
+ V1PersistentVolume result = coreApi.createPersistentVolume(pv, "true", null, null);
+ LOG.info("result", result);
+ } catch (ApiException e) {
+ LOG.error("Exception when calling CoreV1Api#createPersistentVolume");
Review comment:
```suggestion
LOG.error("Exception when calling CoreV1Api#createPersistentVolume "+ e.getMessage(), e);
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/K8SJobSubmitterTest.java
##########
@@ -70,7 +72,22 @@ public void testRunTFJobPerRequest() throws URISyntaxException,
run(spec);
}
+ @Test
+ public void testCreateTFJob() throws IOException, URISyntaxException {
+ ExperimentSpec spec = (ExperimentSpec) buildFromJsonFile(ExperimentSpec.class, tfTfboardJobwReqFile);
+ Experiment experiment = submitter.createExperiment(spec);
+ Assert.assertTrue(true);
+ }
+
+ @Test
+ public void testDeleteTFJob() throws IOException, URISyntaxException {
+ ExperimentSpec spec = (ExperimentSpec) buildFromJsonFile(ExperimentSpec.class, tfTfboardJobwReqFile);
+ Experiment experiment = submitter.deleteExperiment(spec);
+ Assert.assertTrue(true);
Review comment:
```suggestion
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParserTest.java
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.parser;
+import io.kubernetes.client.custom.Quantity;
+import io.kubernetes.client.models.V1PersistentVolume;
+import io.kubernetes.client.models.V1PersistentVolumeClaim;
+import org.apache.submarine.server.submitter.k8s.util.TensorboardUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VolumeSpecParserTest {
+ private static final Logger LOG = LoggerFactory.getLogger(VolumeSpecParserTest.class);
+
+ @Before
+ public void before() {
+
Review comment:
```suggestion
```
##########
File path: submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/K8SJobSubmitterTest.java
##########
@@ -70,7 +72,22 @@ public void testRunTFJobPerRequest() throws URISyntaxException,
run(spec);
}
+ @Test
+ public void testCreateTFJob() throws IOException, URISyntaxException {
+ ExperimentSpec spec = (ExperimentSpec) buildFromJsonFile(ExperimentSpec.class, tfTfboardJobwReqFile);
+ Experiment experiment = submitter.createExperiment(spec);
+ Assert.assertTrue(true);
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] pingsutw commented on a change in pull request #484: SUBMARINE-701. Support Tensorboard in Experiment
Posted by GitBox <gi...@apache.org>.
pingsutw commented on a change in pull request #484:
URL: https://github.com/apache/submarine/pull/484#discussion_r549944512
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -320,6 +349,161 @@ public Notebook deleteNotebook(NotebookSpec spec) throws SubmarineRuntimeExcepti
return notebookList;
}
+ public void createTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.createNamespacedDeployment(namespace, deployment, "true", null, null);
+ coreApi.createNamespacedService(namespace, svc, "true", null, null);
+ api.createNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(),
+ ingressRoute.getPlural(), ingressRoute, "true");
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
+ throw e;
+ }
+ }
+
+ public void deleteTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.deleteNamespacedDeployment(deploy_name, namespace, "true",
+ null, null, null, null, null);
+ coreApi.deleteNamespacedService(svc_name, namespace, "true",
+ null, null, null, null, null);
+ api.deleteNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(), ingressRoute.getPlural(), ingress_name,
+ new V1DeleteOptionsBuilder().withApiVersion(ingressRoute.getApiVersion()).build(),
+ null, null, null);
+
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
Review comment:
And I think the `ApiException` message in other files also need to be modify
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
##########
@@ -320,6 +349,161 @@ public Notebook deleteNotebook(NotebookSpec spec) throws SubmarineRuntimeExcepti
return notebookList;
}
+ public void createTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.createNamespacedDeployment(namespace, deployment, "true", null, null);
+ coreApi.createNamespacedService(namespace, svc, "true", null, null);
+ api.createNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(),
+ ingressRoute.getPlural(), ingressRoute, "true");
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
+ throw e;
+ }
+ }
+
+ public void deleteTFBoard(String name, String namespace) throws ApiException {
+ final String deploy_name = TensorboardUtils.DEPLOY_PREFIX + name;
+ final String pod_name = TensorboardUtils.POD_PREFIX + name;
+ final String svc_name = TensorboardUtils.SVC_PREFIX + name;
+ final String ingress_name = TensorboardUtils.INGRESS_PREFIX + name;
+
+ final String image = TensorboardUtils.IMAGE_NAME;
+ final String route_path = TensorboardUtils.PATH_PREFIX + name;
+ final String pvc = TensorboardUtils.PVC_PREFIX + name;
+
+ V1Deployment deployment = TensorboardSpecParser.parseDeployment(deploy_name, image, route_path, pvc);
+ V1Service svc = TensorboardSpecParser.parseService(svc_name, pod_name);
+ IngressRoute ingressRoute = TensorboardSpecParser.parseIngressRoute(
+ ingress_name, namespace, route_path, svc_name
+ );
+
+ try {
+ appsV1Api.deleteNamespacedDeployment(deploy_name, namespace, "true",
+ null, null, null, null, null);
+ coreApi.deleteNamespacedService(svc_name, namespace, "true",
+ null, null, null, null, null);
+ api.deleteNamespacedCustomObject(
+ ingressRoute.getGroup(), ingressRoute.getVersion(),
+ ingressRoute.getMetadata().getNamespace(), ingressRoute.getPlural(), ingress_name,
+ new V1DeleteOptionsBuilder().withApiVersion(ingressRoute.getApiVersion()).build(),
+ null, null, null);
+
+ } catch (ApiException e) {
+ LOG.error("Exception when calling createTFBoard");
Review comment:
`LOG.error` will show which function and which line runs an error, so we don't need to display `Exception when calling .....`, we should show more detailed messages to users. like https://github.com/apache/submarine/blob/d104080a926d30bcc45ccd740f9489dd7365d0b5/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java#L264-L270
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] asfgit closed pull request #484: SUBMARINE-701. Support Tensorboard in Experiment
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #484:
URL: https://github.com/apache/submarine/pull/484
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org