You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by pi...@apache.org on 2020/07/15 15:11:39 UTC
[submarine] branch master updated: SUBMARINE-507. Submarine
Environment Management
This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new c9b9b32 SUBMARINE-507. Submarine Environment Management
c9b9b32 is described below
commit c9b9b32f8c320d1707acac342a20842d69644022
Author: Manikandan R <ma...@caastle.com>
AuthorDate: Sun Jul 5 13:20:47 2020 +0530
SUBMARINE-507. Submarine Environment Management
### What is this PR for?
https://issues.apache.org/jira/browse/SUBMARINE-507
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-507
### How should this be tested?
* First time? Setup Travis CI as described on https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? Yes/No
* Is there breaking changes for older versions? Yes/No
* Does this needs documentation? Yes/No
Author: Manikandan R <ma...@caastle.com>
Closes #299 from manirajv06/SUBMARINE-507 and squashes the following commits:
c7bedbe [Manikandan R] Fixed CI failures
7826137 [Manikandan R] Fixed review comments and CI failures.
287161d [Manikandan R] Fixed review comments.
76b361c [Manikandan R] Fixed review comments. Most importantly, ensuring conda version falls between min and max version. Also, definition of environment: {} in Experiment spec has been changed. If "image" is available, set it as k8 container image. If "name" is available, fetch the environment details using name and use the details for K8 init container process.
98283c9 [Manikandan R] 1. Fixed review comments: a. Added EnvironmentId class to handle environmentId generation. Refactored ExperimentId class and introduced AbstractUniqueIdGenerator abstract class for both ExperimentId and EnvironmentId class. b. Refactored EnvironmentManager class to merge updateEnvironment and createEnvironment functionalities. 2. Added test cases for updateEnvironment flow. Used http patch api's by using appropriate version of artifacts. Made version changes f [...]
6aa8fa0 [Manikandan R] Fixed review comments
25009e8 [Manikandan R] Merged anonymous/embedded environment with Named/Pre-defined environment in Single EnvironmentSpec class
f124b80 [Manikandan R] Persistence support for EnvironmentManager using Metastore (MySQL) database Refactored MyBatis related classes as part of creating SqlSession for Metastore database Rebased and resolved conflicts after merging ExperientAPI refactoring changes
895dda4 [Manikandan R] Modified environment related commands as part of K8 init containers Added EnvironmentManagerRestApiIT class for Integration Testing Added test case in JobManagerRestApiIT to ensure jobs use environment Pom dependency changes Fixed minor issues
72cdae8 [Manikandan R] Support for InMemory EnvironmentManager Developed Testcases for EnvironmentManagerRestApi based on InMemory EnvironmentManager Leveraged k8's Init containers to accomodate Environment details in k8's POC Spec Added Testcase in JobSpecParserTest to ensure k8's Init containers has expected Environment details
20a6859 [Manikandan R] SUBMARINE-507. WIP POC Patch. Patch has laid out the skeleton for early review to ensure changes are on the right direction
---
docs/database/metastore.sql | 12 +
.../commons/utils/AbstractUniqueIdGenerator.java | 75 ++-----
.../submarine/commons/utils/SubmarineConfVars.java | 4 +-
.../api/{spec => environment}/Environment.java | 32 ++-
.../server/api/environment/EnvironmentId.java | 72 ++++++
.../server/api/experiment/ExperimentId.java | 93 +-------
.../submarine/server/api/spec/EnvironmentSpec.java | 105 +++++++++
.../submarine/server/api/spec/ExperimentSpec.java | 8 +-
.../spec/{ExperimentSpec.java => KernelSpec.java} | 54 +++--
submarine-server/server-core/pom.xml | 17 +-
.../org/apache/submarine/server/Bootstrap.java | 28 +++
.../database/entity/BaseEntity.java | 2 +-
.../database => database/utils}/MyBatisUtil.java | 59 +++--
.../server/environment/EnvironmentManager.java | 246 +++++++++++++++++++++
.../database/entity/EnvironmentEntity.java} | 32 ++-
.../database/mappers/EnvironmentMapper.java} | 23 +-
.../submarine/server/rest/EnvironmentRestApi.java | 194 ++++++++++++++++
.../submarine/server/rest/RestConstants.java | 7 +
.../server/workbench/database/entity/Job.java | 2 +
.../server/workbench/database/entity/Metric.java | 2 +
.../server/workbench/database/entity/Param.java | 2 +
.../server/workbench/database/entity/Project.java | 1 +
.../workbench/database/entity/ProjectFiles.java | 2 +
.../server/workbench/database/entity/SysDept.java | 2 +
.../server/workbench/database/entity/SysDict.java | 2 +
.../workbench/database/entity/SysDictItem.java | 2 +
.../workbench/database/entity/SysMessage.java | 2 +
.../server/workbench/database/entity/SysUser.java | 2 +
.../server/workbench/database/entity/Team.java | 2 +
.../workbench/database/entity/TeamMember.java | 2 +
.../workbench/database/service/JobService.java | 2 +-
.../workbench/database/service/MetricService.java | 2 +-
.../workbench/database/service/ParamService.java | 2 +-
.../database/service/ProjectFilesService.java | 2 +-
.../workbench/database/service/ProjectService.java | 2 +-
.../database/service/SysDictItemService.java | 2 +-
.../database/service/SysMessageService.java | 2 +-
.../workbench/database/service/SysUserService.java | 2 +-
.../database/service/TeamMemberService.java | 2 +-
.../workbench/database/service/TeamService.java | 2 +-
.../server/workbench/rest/LoginRestApi.java | 2 +-
.../server/workbench/rest/SysDeptRestApi.java | 2 +-
.../server/workbench/rest/SysDictItemRestApi.java | 2 +-
.../server/workbench/rest/SysDictRestApi.java | 2 +-
.../server/workbench/rest/SystemRestApi.java | 2 +-
.../main/resources/mybatis-config-metastore.xml | 59 +++++
.../database/mappers/EnvironmentMapper.xml | 69 ++++++
.../submarine/database/mappers/JobMapper.xml | 2 +-
.../submarine/database/mappers/MetricMapper.xml | 2 +-
.../submarine/database/mappers/ParamMapper.xml | 2 +-
.../database/mappers/ProjectFilesMapper.xml | 2 +-
.../submarine/database/mappers/ProjectMapper.xml | 2 +-
.../submarine/database/mappers/SysDeptMapper.xml | 2 +-
.../database/mappers/SysDictItemMapper.xml | 2 +-
.../submarine/database/mappers/SysDictMapper.xml | 2 +-
.../database/mappers/SysMessageMapper.xml | 2 +-
.../submarine/database/mappers/SysUserMapper.xml | 2 +-
.../submarine/database/mappers/TeamMapper.xml | 2 +-
.../database/mappers/TeamMemberMapper.xml | 2 +-
.../src/main/resources/submarine-site.xml | 8 +
.../server/AbstractSubmarineServerTest.java | 159 ++++++++++++-
.../server/rest/EnvironmentManagerRestApiTest.java | 124 +++++++++++
.../src/test/resources/environment/test_env_1.json | 14 ++
.../src/test/resources/environment/test_env_2.json | 10 +
.../server-submitter/submitter-k8s/pom.xml | 24 ++
.../submitter/k8s/parser/ExperimentSpecParser.java | 94 +++++++-
.../submitter/k8s/ExperimentSpecParserTest.java | 83 +++++++
.../server/submitter/k8s/SpecBuilder.java | 3 +
.../src/test/resources/pytorch_job_req_env.json | 26 +++
.../resources/pytorch_job_req_invalid_env.json | 27 +++
.../rest/EnvironmentManagerRestApiIT.java | 96 ++++++++
.../apache/submarine/rest/ExperimentRestApiIT.java | 181 +++++++++++++--
.../tensorflow/tf-mnist-with-env-patch-req.json | 20 ++
.../tensorflow/tf-mnist-with-env-req.json | 25 +++
74 files changed, 1866 insertions(+), 296 deletions(-)
diff --git a/docs/database/metastore.sql b/docs/database/metastore.sql
index 041c96d..264d6c6 100644
--- a/docs/database/metastore.sql
+++ b/docs/database/metastore.sql
@@ -963,6 +963,18 @@ CREATE TABLE WRITE_SET (
WS_OPERATION_TYPE char(1) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
+CREATE TABLE `environment` (
+ `id` varchar(64) NOT NULL COMMENT 'Id of the Environment',
+ `environment_name` varchar(255) NOT NULL COMMENT 'Name of the Environment',
+ `environment_spec` text NOT NULL COMMENT 'Spec of the Environment',
+ `create_by` varchar(32) DEFAULT NULL COMMENT 'create user',
+ `create_time` datetime DEFAULT NULL COMMENT 'create time',
+ `update_by` varchar(32) DEFAULT NULL COMMENT 'last update user',
+ `update_time` datetime DEFAULT NULL COMMENT 'last update time',
+ PRIMARY KEY `id` (`id`),
+ UNIQUE KEY `environment_name` (`environment_name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
-- -----------------------------------------------------------------
-- Record schema version. Should be the last step in the init script
-- -----------------------------------------------------------------
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/ExperimentId.java b/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/AbstractUniqueIdGenerator.java
similarity index 54%
copy from submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/ExperimentId.java
copy to submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/AbstractUniqueIdGenerator.java
index 388d058..c16d25d 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/ExperimentId.java
+++ b/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/AbstractUniqueIdGenerator.java
@@ -17,51 +17,18 @@
* under the License.
*/
-package org.apache.submarine.server.api.experiment;
-
-/**
- * The unique id for experiment. Formatter: experiment_${server_timestamp}_${counter}
- * Such as: experiment_1577627710_0001
- */
-public class ExperimentId implements Comparable<ExperimentId> {
- private static final String EXPERIMENT_ID_PREFIX = "experiment_";
- private static final int EXPERIMENT_ID_MIN_DIGITS = 4;
+package org.apache.submarine.commons.utils;
+public abstract class AbstractUniqueIdGenerator<T> implements Comparable<T> {
+
+ private static final int ID_MIN_DIGITS = 4;
+
private int id;
- private long serverTimestamp;
-
- /**
- * Get the object of JobId.
- * @param jobId job id string
- * @return object
- */
- public static ExperimentId fromString(String jobId) {
- if (jobId == null) {
- return null;
- }
- String[] components = jobId.split("\\_");
- if (components.length != 3) {
- return null;
- }
- return ExperimentId.newInstance(Long.parseLong(components[1]), Integer.parseInt(components[2]));
- }
-
+ private long serverTimestamp;
+
/**
- * Ge the object of JobId.
- * @param serverTimestamp the timestamp when the server start
- * @param id count
- * @return object
- */
- public static ExperimentId newInstance(long serverTimestamp, int id) {
- ExperimentId experimentId = new ExperimentId();
- experimentId.setServerTimestamp(serverTimestamp);
- experimentId.setId(id);
- return experimentId;
- }
-
- /**
- * Get the count of job since the server started
+ * Get the count since the server started
* @return number
*/
public int getId() {
@@ -91,21 +58,24 @@ public class ExperimentId implements Comparable<ExperimentId> {
public void setServerTimestamp(long timestamp) {
this.serverTimestamp = timestamp;
}
-
+
+ @SuppressWarnings("rawtypes")
@Override
- public int compareTo(ExperimentId o) {
- return this.getId() > o.getId() ? 1 : 0;
+ public int compareTo(Object o) {
+ AbstractUniqueIdGenerator other = (AbstractUniqueIdGenerator) o;
+ return this.getId() > other.getId() ? 1 : 0;
}
-
+
@Override
public int hashCode() {
final int prime = 371237;
int result = 6521;
- result = prime * result + (int) (serverTimestamp ^ (serverTimestamp >>> 32));
+ result = prime * result + (int) (getServerTimestamp() ^ (getServerTimestamp() >>> 32));
result = prime * result + getId();
return result;
}
+ @SuppressWarnings("rawtypes")
@Override
public boolean equals(Object obj) {
if (obj == null || getClass() != obj.getClass()) {
@@ -114,23 +84,16 @@ public class ExperimentId implements Comparable<ExperimentId> {
if (this == obj) {
return true;
}
- ExperimentId other = (ExperimentId) obj;
+ AbstractUniqueIdGenerator other = (AbstractUniqueIdGenerator) obj;
if (this.getServerTimestamp() != other.getServerTimestamp()) {
return false;
}
return this.getId() == other.getId();
}
- @Override
- public String toString() {
- StringBuilder sb = new StringBuilder(64);
- sb.append(EXPERIMENT_ID_PREFIX).append(serverTimestamp).append("_");
- format(sb, getId());
- return sb.toString();
- }
- private void format(StringBuilder sb, long value) {
- int minimumDigits = EXPERIMENT_ID_MIN_DIGITS;
+ protected void format(StringBuilder sb, long value) {
+ int minimumDigits = ID_MIN_DIGITS;
if (value < 0) {
sb.append('-');
value = -value;
diff --git a/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java b/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
index de57152..3e2e547 100644
--- a/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
+++ b/submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfVars.java
@@ -68,7 +68,9 @@ public class SubmarineConfVars {
WORKBENCH_WEB_WAR("workbench.web.war", "submarine-workbench/workbench-web/dist"),
SUBMARINE_RUNTIME_CLASS("submarine.runtime.class",
"org.apache.submarine.server.submitter.yarn.YarnRuntimeFactory"),
- SUBMARINE_SUBMITTER("submarine.submitter", "k8s");
+ SUBMARINE_SUBMITTER("submarine.submitter", "k8s"),
+ ENVIRONMENT_CONDA_MIN_VERSION("environment.conda.min.version", "4.0.1"),
+ ENVIRONMENT_CONDA_MAX_VERSION("environment.conda.max.version", "4.10.10");
private String varName;
@SuppressWarnings("rawtypes")
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/environment/Environment.java
similarity index 57%
copy from submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java
copy to submarine-server/server-api/src/main/java/org/apache/submarine/server/api/environment/Environment.java
index 3ae6239..48332c7 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/environment/Environment.java
@@ -17,21 +17,35 @@
* under the License.
*/
-package org.apache.submarine.server.api.spec;
+package org.apache.submarine.server.api.environment;
+
+import org.apache.submarine.server.api.spec.EnvironmentSpec;
public class Environment {
- private String image;
/**
- * Get the image for the experiment. If the {@link ExperimentTaskSpec#getImage()} not
- * specified the image replaced with it.
- * @return image
+ * ID of the environment
+ */
+ private EnvironmentId environmentId;
+
+ /**
+ * Environment Spec
*/
- public String getImage() {
- return image;
+ private EnvironmentSpec environmentSpec;
+
+ public EnvironmentId getEnvironmentId() {
+ return environmentId;
+ }
+
+ public void setEnvironmentId(EnvironmentId environmentId) {
+ this.environmentId = environmentId;
+ }
+
+ public EnvironmentSpec getEnvironmentSpec() {
+ return environmentSpec;
}
- public void setImage(String image) {
- this.image = image;
+ public void setEnvironmentSpec(EnvironmentSpec environmentSpec) {
+ this.environmentSpec = environmentSpec;
}
}
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/environment/EnvironmentId.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/environment/EnvironmentId.java
new file mode 100644
index 0000000..0d2a54b
--- /dev/null
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/environment/EnvironmentId.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.environment;
+
+import org.apache.submarine.commons.utils.AbstractUniqueIdGenerator;
+
+/**
+ * The unique id for environment. Formatter:
+ * environment_${server_timestamp}_${counter} Such as:
+ * environment_1577627710_0001
+ */
+public class EnvironmentId extends AbstractUniqueIdGenerator<EnvironmentId> {
+
+ private static final String ENVIRONMENT_ID_PREFIX = "environment_";
+
+ /**
+ * Get the object of EnvironmentId.
+ *
+ * @param environmentId Id of the environment string
+ * @return object
+ */
+ public static EnvironmentId fromString(String environmentId) {
+ if (environmentId == null) {
+ return null;
+ }
+ String[] components = environmentId.split("\\_");
+ if (components.length != 3) {
+ return null;
+ }
+ return EnvironmentId.newInstance(Long.parseLong(components[1]),
+ Integer.parseInt(components[2]));
+ }
+
+ /**
+ * Ge the object of EnvironmentId.
+ *
+ * @param serverTimestamp the timestamp when the server start
+ * @param id count
+ * @return object
+ */
+ public static EnvironmentId newInstance(long serverTimestamp, int id) {
+ EnvironmentId experimentId = new EnvironmentId();
+ experimentId.setServerTimestamp(serverTimestamp);
+ experimentId.setId(id);
+ return experimentId;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder(64);
+ sb.append(ENVIRONMENT_ID_PREFIX).append(getServerTimestamp()).append("_");
+ format(sb, getId());
+ return sb.toString();
+ }
+}
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/ExperimentId.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/ExperimentId.java
index 388d058..42be1b1 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/ExperimentId.java
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/experiment/ExperimentId.java
@@ -19,18 +19,15 @@
package org.apache.submarine.server.api.experiment;
+import org.apache.submarine.commons.utils.AbstractUniqueIdGenerator;
+
/**
* The unique id for experiment. Formatter: experiment_${server_timestamp}_${counter}
* Such as: experiment_1577627710_0001
*/
-public class ExperimentId implements Comparable<ExperimentId> {
+public class ExperimentId extends AbstractUniqueIdGenerator<ExperimentId> {
private static final String EXPERIMENT_ID_PREFIX = "experiment_";
- private static final int EXPERIMENT_ID_MIN_DIGITS = 4;
-
- private int id;
-
- private long serverTimestamp;
-
+
/**
* Get the object of JobId.
* @param jobId job id string
@@ -60,90 +57,12 @@ public class ExperimentId implements Comparable<ExperimentId> {
return experimentId;
}
- /**
- * Get the count of job since the server started
- * @return number
- */
- public int getId() {
- return id;
- }
-
- /**
- * Set the count of job
- * @param id number
- */
- public void setId(int id) {
- this.id = id;
- }
-
- /**
- * Get the timestamp(s) when the server started
- * @return timestamp(s)
- */
- public long getServerTimestamp() {
- return serverTimestamp;
- }
-
- /**
- * Set the server started timestamp(s)
- * @param timestamp seconds
- */
- public void setServerTimestamp(long timestamp) {
- this.serverTimestamp = timestamp;
- }
-
- @Override
- public int compareTo(ExperimentId o) {
- return this.getId() > o.getId() ? 1 : 0;
- }
-
- @Override
- public int hashCode() {
- final int prime = 371237;
- int result = 6521;
- result = prime * result + (int) (serverTimestamp ^ (serverTimestamp >>> 32));
- result = prime * result + getId();
- return result;
- }
-
- @Override
- public boolean equals(Object obj) {
- if (obj == null || getClass() != obj.getClass()) {
- return false;
- }
- if (this == obj) {
- return true;
- }
- ExperimentId other = (ExperimentId) obj;
- if (this.getServerTimestamp() != other.getServerTimestamp()) {
- return false;
- }
- return this.getId() == other.getId();
- }
-
@Override
public String toString() {
StringBuilder sb = new StringBuilder(64);
- sb.append(EXPERIMENT_ID_PREFIX).append(serverTimestamp).append("_");
+ sb.append(EXPERIMENT_ID_PREFIX).append(getServerTimestamp()).append("_");
format(sb, getId());
return sb.toString();
}
-
- private void format(StringBuilder sb, long value) {
- int minimumDigits = EXPERIMENT_ID_MIN_DIGITS;
- if (value < 0) {
- sb.append('-');
- value = -value;
- }
-
- long tmp = value;
- do {
- tmp /= 10;
- } while (--minimumDigits > 0 && tmp > 0);
-
- for (int i = minimumDigits; i > 0; --i) {
- sb.append('0');
- }
- sb.append(value);
- }
}
+
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/EnvironmentSpec.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/EnvironmentSpec.java
new file mode 100644
index 0000000..6b16e22
--- /dev/null
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/EnvironmentSpec.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+/**
+ * Submarine environment spec.
+ */
+public class EnvironmentSpec {
+
+
+ /**
+ * Name of the environment
+ */
+ private String name;
+
+ /**
+ * Docker-Image of the environment
+ */
+ private String dockerImage;
+
+ /**
+ * Kernel Spec
+ */
+ private KernelSpec kernelSpec;
+
+ /**
+ * Description
+ */
+ private String description;
+
+ /**
+ * Image
+ * @return
+ */
+ private String image;
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public String getDockerImage() {
+ return dockerImage;
+ }
+
+ public void setDockerImage(String dockerImage) {
+ this.dockerImage = dockerImage;
+ }
+
+ public KernelSpec getKernelSpec() {
+ return kernelSpec;
+ }
+
+ public void setKernelSpec(KernelSpec kernelSpec) {
+ this.kernelSpec = kernelSpec;
+ }
+
+ /**
+ * @return the description
+ */
+ public String getDescription() {
+ return description;
+ }
+
+ /**
+ * @param description the description to set
+ */
+ public void setDescription(String description) {
+ this.description = description;
+ }
+
+ /**
+ * @return the image
+ */
+ public String getImage() {
+ return image;
+ }
+
+ /**
+ * @param image the image to set
+ */
+ public void setImage(String image) {
+ this.image = image;
+ }
+}
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
index 919db07..88acaa2 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
@@ -26,7 +26,7 @@ import java.util.Map;
*/
public class ExperimentSpec {
private ExperimentMeta meta;
- private Environment environment;
+ private EnvironmentSpec environment;
private Map<String, ExperimentTaskSpec> spec;
public ExperimentSpec() {
@@ -41,12 +41,12 @@ public class ExperimentSpec {
this.meta = meta;
}
- public Environment getEnvironment() {
+ public EnvironmentSpec getEnvironment() {
return environment;
}
- public void setEnvironment(Environment environment) {
- this.environment = environment;
+ public void setEnvironment(EnvironmentSpec environmentSpec) {
+ this.environment = environmentSpec;
}
public Map<String, ExperimentTaskSpec> getSpec() {
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/KernelSpec.java
similarity index 56%
copy from submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
copy to submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/KernelSpec.java
index 919db07..851ff8d 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
+++ b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/KernelSpec.java
@@ -19,41 +19,49 @@
package org.apache.submarine.server.api.spec;
-import java.util.Map;
+import java.util.List;
/**
- * Specification of the desired behavior of an experiment.
+ * Kernel Spec.
*/
-public class ExperimentSpec {
- private ExperimentMeta meta;
- private Environment environment;
- private Map<String, ExperimentTaskSpec> spec;
+public class KernelSpec {
+
+ /**
+ * Name of the kernel
+ */
+ private String name;
+
+ /**
+ * Name of the channels
+ */
+ private List<String> channels;
+
+ /**
+ * List of kernel dependencies
+ */
+ private List<String> dependencies;
- public ExperimentSpec() {
-
- }
-
- public ExperimentMeta getMeta() {
- return meta;
+ public String getName() {
+ return name;
}
- public void setMeta(ExperimentMeta meta) {
- this.meta = meta;
+ public void setName(String name) {
+ this.name = name;
}
- public Environment getEnvironment() {
- return environment;
+ public List<String> getChannels() {
+ return channels;
}
- public void setEnvironment(Environment environment) {
- this.environment = environment;
+ public void setChannels(List<String> channels) {
+ this.channels = channels;
}
- public Map<String, ExperimentTaskSpec> getSpec() {
- return spec;
+ public List<String> getDependencies() {
+ return dependencies;
}
- public void setSpec(Map<String, ExperimentTaskSpec> spec) {
- this.spec = spec;
- }
+ public void setDependencies(List<String> dependencies) {
+ this.dependencies = dependencies;
+ }
}
diff --git a/submarine-server/server-core/pom.xml b/submarine-server/server-core/pom.xml
index d064af2..34d67cf 100644
--- a/submarine-server/server-core/pom.xml
+++ b/submarine-server/server-core/pom.xml
@@ -423,7 +423,22 @@
</exclusions>
<scope>test</scope>
</dependency>
-
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>${httpclient.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>commons-codec</groupId>
+ <artifactId>commons-codec</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpcore</artifactId>
+ <version>${httpcore.version}</version>
+ </dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/Bootstrap.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/Bootstrap.java
index 5ae6bfc..2744700 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/Bootstrap.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/Bootstrap.java
@@ -58,15 +58,43 @@ public class Bootstrap extends HttpServlet {
List<Server> servers = new ArrayList<>();
servers.add(new Server().url("/api"));
oas.servers(servers);
+
+ OpenAPI oasEnvironment = new OpenAPI();
+ Info infoEnvironment = new Info().title("Submarine Environment API")
+ .description("The Submarine REST API allows you to create, update, "
+ + "delete, get and list environments\n. The API is hosted under "
+ + "the /v1/environment route on the Submarine server.\n "
+ + "For example, to list experiments on a server hosted at "
+ + "http://localhost:8080, access http://localhost:8080/api/v1/"
+ + "environment/")
+ .termsOfService("http://swagger.io/terms/")
+ .contact(
+ new Contact().email("dev@submarine.apache.org"))
+ .version("0.5.0-SNAPSHOT")
+ .license(
+ new License().name("Apache 2.0").
+ url("http://www.apache.org/licenses/LICENSE-2.0.html"));
+
+ oasEnvironment.info(infoEnvironment);
+ oasEnvironment.servers(servers);
+
SwaggerConfiguration oasConfig = new SwaggerConfiguration()
.openAPI(oas)
.resourcePackages(Stream.of("org.apache.submarine.server.rest").collect(Collectors.toSet()));
+ SwaggerConfiguration oasConfigEnvironment =
+ new SwaggerConfiguration().openAPI(oasEnvironment)
+ .resourcePackages(Stream.of("org.apache.submarine.server.rest")
+ .collect(Collectors.toSet()));
try {
new JaxrsOpenApiContextBuilder()
.servletConfig(config)
.openApiConfiguration(oasConfig)
.buildContext(true);
+ new JaxrsOpenApiContextBuilder()
+ .servletConfig(config)
+ .openApiConfiguration(oasConfigEnvironment)
+ .buildContext(true);
} catch (OpenApiConfigurationException e) {
throw new ServletException(e.getMessage(), e);
}
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/BaseEntity.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/database/entity/BaseEntity.java
similarity index 97%
rename from submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/BaseEntity.java
rename to submarine-server/server-core/src/main/java/org/apache/submarine/server/database/entity/BaseEntity.java
index 1ed11a0..0d3b0c6 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/BaseEntity.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/database/entity/BaseEntity.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.submarine.server.workbench.database.entity;
+package org.apache.submarine.server.database.entity;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.google.common.annotations.VisibleForTesting;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/MyBatisUtil.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/database/utils/MyBatisUtil.java
similarity index 57%
rename from submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/MyBatisUtil.java
rename to submarine-server/server-core/src/main/java/org/apache/submarine/server/database/utils/MyBatisUtil.java
index 733159c..9c85765 100755
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/MyBatisUtil.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/database/utils/MyBatisUtil.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.submarine.server.workbench.database;
+package org.apache.submarine.server.database.utils;
import org.apache.ibatis.io.Resources;
import org.apache.ibatis.session.SqlSession;
@@ -34,33 +34,39 @@ public class MyBatisUtil {
private static final Logger LOG = LoggerFactory.getLogger(MyBatisUtil.class);
private static SqlSessionFactory sqlSessionFactory;
+ private static SqlSessionFactory metastoreSqlSessionFactory;
static {
+ SubmarineConfiguration conf = SubmarineConfiguration.getInstance();
+ sqlSessionFactory = buildSqlSessionFactory("mybatis-config.xml",
+ conf.getJdbcUrl(), conf.getJdbcUserName(), conf.getJdbcPassword());
+ metastoreSqlSessionFactory = buildSqlSessionFactory(
+ "mybatis-config-metastore.xml", conf.getMetastoreJdbcUrl(),
+ conf.getMetastoreJdbcUserName(), conf.getMetastoreJdbcPassword());
+ }
+
+ private static SqlSessionFactory buildSqlSessionFactory(String configFile,
+ String jdbcUrl, String jdbcUserName, String jdbcPassword) {
Reader reader = null;
+ SqlSessionFactory sqlSessionFactory = null;
try {
try {
- reader = Resources.getResourceAsReader("mybatis-config.xml");
+ reader = Resources.getResourceAsReader(configFile);
} catch (IOException e) {
LOG.error(e.getMessage(), e);
throw new RuntimeException(e.getMessage());
}
-
- checkCalledByTestMethod();
-
- SubmarineConfiguration conf = SubmarineConfiguration.getInstance();
- String jdbcClassName = conf.getJdbcDriverClassName();
- String jdbcUrl = conf.getJdbcUrl();
- String jdbcUserName = conf.getJdbcUserName();
- String jdbcPassword = conf.getJdbcPassword();
- LOG.info("MyBatisUtil -> jdbcClassName: {}, jdbcUrl: {}, jdbcUserName: {}, jdbcPassword: {}",
+ checkCalledByTestMethod(jdbcUrl, jdbcUserName, jdbcPassword);
+ String jdbcClassName =
+ SubmarineConfiguration.getInstance().getJdbcDriverClassName();
+ LOG.info(
+ "MyBatisUtil -> jdbcClassName: {}, jdbcUrl: {}, jdbcUserName: {}, jdbcPassword: {}",
jdbcClassName, jdbcUrl, jdbcUserName, jdbcPassword);
-
Properties props = new Properties();
props.setProperty("jdbc.driverClassName", jdbcClassName);
props.setProperty("jdbc.url", jdbcUrl);
props.setProperty("jdbc.username", jdbcUserName);
props.setProperty("jdbc.password", jdbcPassword);
-
sqlSessionFactory = new SqlSessionFactoryBuilder().build(reader, props);
} finally {
try {
@@ -71,6 +77,7 @@ public class MyBatisUtil {
LOG.error(e.getMessage(), e);
}
}
+ return sqlSessionFactory;
}
/**
@@ -81,25 +88,31 @@ public class MyBatisUtil {
public static SqlSession getSqlSession() {
return sqlSessionFactory.openSession();
}
+
+ public static SqlSession getMetastoreSqlSession() {
+ return metastoreSqlSessionFactory.openSession();
+ }
- private static void checkCalledByTestMethod() {
- StackTraceElement[] stackTraceElements = Thread.currentThread().getStackTrace();
+ private static void checkCalledByTestMethod(String jdbcUrl,
+ String jdbcUserName, String jdbcPassword) {
+ StackTraceElement[] stackTraceElements =
+ Thread.currentThread().getStackTrace();
for (StackTraceElement element : stackTraceElements) {
if (element.getClassName().endsWith("Test")) {
- usingTestDatabase();
+ usingTestDatabase(jdbcUrl, jdbcUserName, jdbcPassword);
return;
}
}
}
- private static void usingTestDatabase() {
+ private static void usingTestDatabase(String jdbcUrl, String jdbcUserName,
+ String jdbcPassword) {
LOG.info("Run the test unit using the test database");
- // Run the test unit using the test database
+ String jdbcPropertiesSuffix = "_test";
+ String finalJdbcUrl = jdbcUrl.replace("?", jdbcPropertiesSuffix + "?");
SubmarineConfiguration conf = SubmarineConfiguration.getInstance();
- conf.setJdbcUrl("jdbc:mysql://127.0.0.1:3306/submarine_test?" +
- "useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&" +
- "failOverReadOnly=false&zeroDateTimeBehavior=convertToNull&useSSL=false");
- conf.setJdbcUserName("submarine_test");
- conf.setJdbcPassword("password_test");
+ conf.setJdbcUrl(finalJdbcUrl);
+ conf.setJdbcUserName(jdbcUserName + jdbcPropertiesSuffix);
+ conf.setJdbcPassword(jdbcPassword + jdbcPropertiesSuffix);
}
}
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/EnvironmentManager.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/EnvironmentManager.java
new file mode 100644
index 0000000..8cfcf89
--- /dev/null
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/EnvironmentManager.java
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.environment;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.ws.rs.core.Response.Status;
+
+import org.apache.ibatis.session.SqlSession;
+import org.apache.submarine.commons.utils.exception.SubmarineRuntimeException;
+import org.apache.submarine.server.SubmarineServer;
+import org.apache.submarine.server.api.environment.Environment;
+import org.apache.submarine.server.api.environment.EnvironmentId;
+import org.apache.submarine.server.api.spec.EnvironmentSpec;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
+import org.apache.submarine.server.environment.database.entity.EnvironmentEntity;
+import org.apache.submarine.server.environment.database.mappers.EnvironmentMapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+
+/**
+ * Environment Management
+ */
+public class EnvironmentManager {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(EnvironmentManager.class);
+
+ private static volatile EnvironmentManager manager;
+
+ private final AtomicInteger environmentIdCounter = new AtomicInteger(0);
+
+ /**
+ * Environment Cache
+ */
+ private final ConcurrentMap<String, Environment> cachedEnvironments =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Get the singleton instance
+ * @return object
+ */
+ public static EnvironmentManager getInstance() {
+ if (manager == null) {
+ synchronized (EnvironmentManager.class) {
+ if (manager == null) {
+ manager = new EnvironmentManager();
+ }
+ }
+ }
+ return manager;
+ }
+
+ private EnvironmentManager() {
+
+ }
+
+ /**
+ * Create Environment
+ * @param spec environment spec
+ * @return Environment environment
+ * @throws SubmarineRuntimeException the service error
+ * @throws MetaException
+ */
+ public Environment createEnvironment(EnvironmentSpec spec)
+ throws SubmarineRuntimeException {
+ checkSpec(spec);
+ LOG.info("Create Environment using spec: " + spec.toString());
+ return createOrUpdateEnvironment(spec, "c");
+ }
+
+ /**
+ * Update environment
+ * @param name Name of the environment
+ * @param spec environment spec
+ * @return Environment environment
+ * @throws SubmarineRuntimeException the service error
+ */
+ public Environment updateEnvironment(String name, EnvironmentSpec spec)
+ throws SubmarineRuntimeException {
+ Environment env = getEnvironmentDetails(name);
+ if (env == null) {
+ throw new SubmarineRuntimeException(Status.NOT_FOUND.getStatusCode(),
+ "Environment not found.");
+ }
+ checkSpec(spec);
+ LOG.info("Update Environment using spec: " + spec.toString());
+ return createOrUpdateEnvironment(spec, "u");
+ }
+
+ private Environment createOrUpdateEnvironment(EnvironmentSpec spec,
+ String operation) {
+ EnvironmentEntity entity = new EnvironmentEntity();
+ String environmentId = generateEnvironmentId().toString();
+ entity.setId(environmentId);
+ entity.setEnvironmentName(spec.getName());
+ entity.setEnvironmentSpec(
+ new GsonBuilder().disableHtmlEscaping().create().toJson(spec));
+ try (SqlSession sqlSession = MyBatisUtil.getMetastoreSqlSession()) {
+ EnvironmentMapper environmentMapper =
+ sqlSession.getMapper(EnvironmentMapper.class);
+ if (operation.equals("c")) {
+ environmentMapper.insert(entity);
+ } else {
+ environmentMapper.update(entity);
+ }
+ sqlSession.commit();
+
+ Environment environment = new Environment();
+ environment.setEnvironmentId(EnvironmentId.fromString(environmentId));
+ environment.setEnvironmentSpec(spec);
+
+ // Update cache
+ cachedEnvironments.putIfAbsent(spec.getName(), environment);
+
+ return environment;
+ } catch (Exception e) {
+ LOG.error(e.getMessage(), e);
+ throw new SubmarineRuntimeException(Status.BAD_REQUEST.getStatusCode(),
+ "Unable to process the environment spec.");
+ }
+ }
+
+ private EnvironmentId generateEnvironmentId() {
+ return EnvironmentId.newInstance(SubmarineServer.getServerTimeStamp(),
+ environmentIdCounter.incrementAndGet());
+ }
+
+ /**
+ * Delete environment
+ * @param name Name of the environment
+ * @return Environment environment
+ * @throws SubmarineRuntimeException the service error
+ */
+ public Environment deleteEnvironment(String name)
+ throws SubmarineRuntimeException {
+ Environment env = getEnvironmentDetails(name);
+ if (env == null) {
+ throw new SubmarineRuntimeException(Status.NOT_FOUND.getStatusCode(),
+ "Environment not found.");
+ }
+
+ LOG.info("Delete Environment for " + name);
+ try (SqlSession sqlSession = MyBatisUtil.getMetastoreSqlSession()) {
+ EnvironmentMapper environmentMapper =
+ sqlSession.getMapper(EnvironmentMapper.class);
+ environmentMapper.delete(name);
+ sqlSession.commit();
+
+ // Invalidate cache
+ cachedEnvironments.remove(name);
+ return env;
+ } catch (Exception e) {
+ LOG.error(e.getMessage(), e);
+ throw new SubmarineRuntimeException(Status.BAD_REQUEST.getStatusCode(),
+ "Unable to delete the environment.");
+ }
+ }
+
+ /**
+ * Get Environment
+ * @param name Name of the environment
+ * @return Environment environment
+ * @throws SubmarineRuntimeException the service error
+ */
+ public Environment getEnvironment(String name)
+ throws SubmarineRuntimeException {
+ Environment environment = getEnvironmentDetails(name);
+ if (environment == null) {
+ throw new SubmarineRuntimeException(Status.NOT_FOUND.getStatusCode(),
+ "Environment not found.");
+ }
+ return environment;
+ }
+
+ /**
+ * List environments
+ * @param status environment status, if null will return all status
+ * @return environment list
+ * @throws SubmarineRuntimeException the service error
+ */
+ public List<Environment> listEnvironments(String status)
+ throws SubmarineRuntimeException {
+ List<Environment> environmentList = new ArrayList<>();
+ return environmentList;
+ }
+
+ private void checkSpec(EnvironmentSpec spec)
+ throws SubmarineRuntimeException {
+ if (spec == null) {
+ throw new SubmarineRuntimeException(Status.OK.getStatusCode(),
+ "Invalid environment spec.");
+ }
+ }
+
+ private Environment getEnvironmentDetails(String name)
+ throws SubmarineRuntimeException {
+
+ // Is it available in cache?
+ Environment env = cachedEnvironments.get(name);
+ if (env != null) {
+ return env;
+ }
+
+ try (SqlSession sqlSession = MyBatisUtil.getMetastoreSqlSession()) {
+ EnvironmentMapper environmentMapper =
+ sqlSession.getMapper(EnvironmentMapper.class);
+ EnvironmentEntity environmentEntity = environmentMapper.select(name);
+
+ if (environmentEntity != null) {
+ env = new Environment();
+ env.setEnvironmentSpec(new Gson().fromJson(
+ environmentEntity.getEnvironmentSpec(), EnvironmentSpec.class));
+ }
+ } catch (Exception e) {
+ LOG.error(e.getMessage(), e);
+ throw new SubmarineRuntimeException(Status.BAD_REQUEST.getStatusCode(),
+ "Unable to get the environment details.");
+ }
+ return env;
+ }
+}
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/database/entity/EnvironmentEntity.java
similarity index 57%
copy from submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java
copy to submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/database/entity/EnvironmentEntity.java
index 3ae6239..4ea903d 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/database/entity/EnvironmentEntity.java
@@ -16,22 +16,30 @@
* specific language governing permissions and limitations
* under the License.
*/
+package org.apache.submarine.server.environment.database.entity;
-package org.apache.submarine.server.api.spec;
+import org.apache.submarine.server.database.entity.BaseEntity;
-public class Environment {
- private String image;
+public class EnvironmentEntity extends BaseEntity {
- /**
- * Get the image for the experiment. If the {@link ExperimentTaskSpec#getImage()} not
- * specified the image replaced with it.
- * @return image
- */
- public String getImage() {
- return image;
+ private String environmentName;
+
+ private String environmentSpec;
+
+ public String getEnvironmentName() {
+ return environmentName;
+ }
+
+ public void setEnvironmentName(String environmentName) {
+ this.environmentName = environmentName;
+ }
+
+ public String getEnvironmentSpec() {
+ return environmentSpec;
}
- public void setImage(String image) {
- this.image = image;
+ public void setEnvironmentSpec(String environmentSpec) {
+ this.environmentSpec = environmentSpec;
}
+
}
diff --git a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/database/mappers/EnvironmentMapper.java
similarity index 68%
rename from submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java
rename to submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/database/mappers/EnvironmentMapper.java
index 3ae6239..eddea32 100644
--- a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/Environment.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/environment/database/mappers/EnvironmentMapper.java
@@ -16,22 +16,17 @@
* specific language governing permissions and limitations
* under the License.
*/
+package org.apache.submarine.server.environment.database.mappers;
-package org.apache.submarine.server.api.spec;
+import org.apache.submarine.server.environment.database.entity.EnvironmentEntity;
-public class Environment {
- private String image;
+public interface EnvironmentMapper {
- /**
- * Get the image for the experiment. If the {@link ExperimentTaskSpec#getImage()} not
- * specified the image replaced with it.
- * @return image
- */
- public String getImage() {
- return image;
- }
+ EnvironmentEntity select(String environmentName);
- public void setImage(String image) {
- this.image = image;
- }
+ int insert(EnvironmentEntity environment);
+
+ int update(EnvironmentEntity environment);
+
+ int delete(String environmentName);
}
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/EnvironmentRestApi.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/EnvironmentRestApi.java
new file mode 100644
index 0000000..e280fa6
--- /dev/null
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/EnvironmentRestApi.java
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.rest;
+
+import java.util.List;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.DELETE;
+import javax.ws.rs.GET;
+import javax.ws.rs.PATCH;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.QueryParam;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
+import org.apache.submarine.commons.utils.exception.SubmarineRuntimeException;
+import org.apache.submarine.server.api.environment.Environment;
+import org.apache.submarine.server.api.spec.EnvironmentSpec;
+import org.apache.submarine.server.environment.EnvironmentManager;
+import org.apache.submarine.server.response.JsonResponse;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.media.Content;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.responses.ApiResponse;
+
+/**
+ * Environment REST API v1. It can accept {@link EnvironmentSpec} to create a
+ * environment.
+ */
+@Path(RestConstants.V1 + "/" + RestConstants.ENVIRONMENTS)
+@Produces({MediaType.APPLICATION_JSON + "; " + RestConstants.CHARSET_UTF8})
+public class EnvironmentRestApi {
+ private final EnvironmentManager environmentManager =
+ EnvironmentManager.getInstance();
+
+ /**
+ * Returns the contents of {@link environment}.
+ * @param spec environment spec
+ * @return the contents of environment
+ */
+ @POST
+ @Consumes({RestConstants.MEDIA_TYPE_YAML, MediaType.APPLICATION_JSON})
+ @Operation(summary = "Create a environment",
+ tags = {"environment"},
+ responses = {
+ @ApiResponse(description = "successful operation",
+ content = @Content(
+ schema = @Schema(
+ implementation = Environment.class)))})
+ public Response createEnvironment(EnvironmentSpec spec) {
+ try {
+ Environment environment = environmentManager.createEnvironment(spec);
+ return new JsonResponse.Builder<Environment>(Response.Status.OK)
+ .success(true).result(environment).build();
+ } catch (SubmarineRuntimeException e) {
+ return parseEnvironmentServiceException(e);
+ }
+ }
+
+ /**
+ * Update environment.
+ * @param name Name of the environment
+ * @param spec environment spec
+ * @return the detailed info about updated environment
+ */
+ @PATCH
+ @Path("/{id}")
+ @Consumes({RestConstants.MEDIA_TYPE_YAML, MediaType.APPLICATION_JSON})
+ @Operation(summary = "Update the environment with job spec",
+ tags = {"environments"},
+ responses = {
+ @ApiResponse(description = "successful operation",
+ content = @Content(
+ schema = @Schema(
+ implementation = Environment.class))),
+ @ApiResponse(
+ responseCode = "404",
+ description = "Environment not found")})
+ public Response updateEnvironment(
+ @PathParam(RestConstants.ENVIRONMENT_ID) String name,
+ EnvironmentSpec spec) {
+ try {
+ Environment environment =
+ environmentManager.updateEnvironment(name, spec);
+ return new JsonResponse.Builder<Environment>(Response.Status.OK)
+ .success(true).result(environment).build();
+ } catch (SubmarineRuntimeException e) {
+ return parseEnvironmentServiceException(e);
+ }
+ }
+
+ /**
+ * Returns the environment that deleted.
+ * @param name Name of the environment
+ * @return the detailed info about deleted environment
+ */
+ @DELETE
+ @Path("/{id}")
+ @Operation(summary = "Delete the environment",
+ tags = {"environments"},
+ responses = {
+ @ApiResponse(description = "successful operation",
+ content = @Content(
+ schema = @Schema(implementation = Environment.class))),
+ @ApiResponse(
+ responseCode = "404", description = "Environment not found")})
+ public Response deleteEnvironment(
+ @PathParam(RestConstants.ENVIRONMENT_ID) String name) {
+ try {
+ Environment environment = environmentManager.deleteEnvironment(name);
+ return new JsonResponse.Builder<Environment>(Response.Status.OK)
+ .success(true).result(environment).build();
+ } catch (SubmarineRuntimeException e) {
+ return parseEnvironmentServiceException(e);
+ }
+ }
+
+ /**
+ * List all environments.
+ * @return environment list
+ */
+ @GET
+ @Operation(summary = "List of Environments",
+ tags = {"environments"},
+ responses = {
+ @ApiResponse(description = "successful operation",
+ content = @Content(
+ schema = @Schema(
+ implementation = Environment.class)))})
+ public Response listEnvironment(@QueryParam("status") String status) {
+ try {
+ List<Environment> environmentList =
+ environmentManager.listEnvironments(status);
+ return new JsonResponse.Builder<List<Environment>>(Response.Status.OK)
+ .success(true).result(environmentList).build();
+ } catch (SubmarineRuntimeException e) {
+ return parseEnvironmentServiceException(e);
+ }
+ }
+
+ /**
+ * Returns details for the given environment.
+ * @param name Name of the environment
+ * @return the contents of environment
+ */
+ @GET
+ @Path("/{id}")
+ @Operation(summary = "Find environment by name",
+ tags = {"environment"},
+ responses = {
+ @ApiResponse(description = "successful operation",
+ content = @Content(
+ schema = @Schema(implementation = Environment.class))),
+ @ApiResponse(
+ responseCode = "404",
+ description = "Environment not found")})
+ public Response getEnvironment(
+ @PathParam(RestConstants.ENVIRONMENT_ID) String name) {
+ try {
+ Environment environment = environmentManager.getEnvironment(name);
+ return new JsonResponse.Builder<Environment>(Response.Status.OK)
+ .success(true).result(environment).build();
+ } catch (SubmarineRuntimeException e) {
+ return parseEnvironmentServiceException(e);
+ }
+ }
+
+ private Response parseEnvironmentServiceException(
+ SubmarineRuntimeException e) {
+ return new JsonResponse.Builder<String>(e.getCode()).message(e.getMessage())
+ .build();
+ }
+}
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/RestConstants.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/RestConstants.java
index f310901..3e3a388 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/RestConstants.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/RestConstants.java
@@ -36,4 +36,11 @@ public class RestConstants {
public static final String NODE = "node";
public static final String LOGS = "logs";
+
+ /**
+ * Environment
+ */
+ public static final String ENVIRONMENTS = "environments";
+
+ public static final String ENVIRONMENT_ID = "id";
}
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Job.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Job.java
index 806830a..5629028 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Job.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Job.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class Job extends BaseEntity {
// Todo(Zac) Add dict configuration.
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Metric.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Metric.java
index b4d806b..7a9df78 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Metric.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Metric.java
@@ -20,6 +20,8 @@ package org.apache.submarine.server.workbench.database.entity;
import java.math.BigInteger;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
/*
# +-------+----------+--------------+---------------+------+--------+------------------+
# | key | value | worker_index | timestamp | step | is_nan | job_name |
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Param.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Param.java
index 3078775..371d2d0 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Param.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Param.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
/*
# +----------+-------+--------------+-----------------------+
# | key | value | worker_index | job_name |
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Project.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Project.java
index 966db21..9e37636 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Project.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Project.java
@@ -18,6 +18,7 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
import org.apache.submarine.server.workbench.annotation.Dict;
import java.util.ArrayList;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/ProjectFiles.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/ProjectFiles.java
index 6e64c31..a672346 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/ProjectFiles.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/ProjectFiles.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class ProjectFiles extends BaseEntity {
private String projectId;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDept.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDept.java
index e0a159c..a614a7f 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDept.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDept.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class SysDept extends BaseEntity {
private String deptCode;
private String deptName;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDict.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDict.java
index 572100e..0fe195c 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDict.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDict.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class SysDict extends BaseEntity {
private String dictCode;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDictItem.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDictItem.java
index 3067c54..db01285 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDictItem.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysDictItem.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class SysDictItem extends BaseEntity {
private String dictCode;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysMessage.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysMessage.java
index d65a548..35c968e 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysMessage.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysMessage.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class SysMessage extends BaseEntity {
private String sender;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysUser.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysUser.java
index 32019c5..3c701cc 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysUser.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/SysUser.java
@@ -19,6 +19,8 @@
package org.apache.submarine.server.workbench.database.entity;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+
+import org.apache.submarine.server.database.entity.BaseEntity;
import org.apache.submarine.server.workbench.annotation.Dict;
import org.apache.submarine.server.workbench.database.utils.CustomJsonDateDeserializer;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Team.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Team.java
index 97bfb15..94082b5 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Team.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/Team.java
@@ -21,6 +21,8 @@ package org.apache.submarine.server.workbench.database.entity;
import java.util.ArrayList;
import java.util.List;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class Team extends BaseEntity {
private String owner;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/TeamMember.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/TeamMember.java
index 41c326b..ce8e9b8 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/TeamMember.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/entity/TeamMember.java
@@ -18,6 +18,8 @@
*/
package org.apache.submarine.server.workbench.database.entity;
+import org.apache.submarine.server.database.entity.BaseEntity;
+
public class TeamMember extends BaseEntity {
private String teamId;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/JobService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/JobService.java
index 54eeba5..0373670 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/JobService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/JobService.java
@@ -20,7 +20,7 @@ package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.RowBounds;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.Job;
import org.apache.submarine.server.workbench.database.mappers.JobMapper;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/MetricService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/MetricService.java
index b5b3cd6..aa8126e 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/MetricService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/MetricService.java
@@ -19,7 +19,7 @@
package org.apache.submarine.server.workbench.database.service;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.Metric;
import org.apache.submarine.server.workbench.database.mappers.MetricMapper;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ParamService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ParamService.java
index 0e59947..4dc40ec 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ParamService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ParamService.java
@@ -21,7 +21,7 @@ package org.apache.submarine.server.workbench.database.service;
import java.util.List;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.Param;
import org.apache.submarine.server.workbench.database.mappers.ParamMapper;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectFilesService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectFilesService.java
index 3eb9e36..da80eec 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectFilesService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectFilesService.java
@@ -19,7 +19,7 @@
package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.ProjectFiles;
import org.apache.submarine.server.workbench.database.mappers.ProjectFilesMapper;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectService.java
index 6396370..ac6186d 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/ProjectService.java
@@ -20,7 +20,7 @@ package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.RowBounds;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.Project;
import org.apache.submarine.server.workbench.database.entity.ProjectFiles;
import org.apache.submarine.server.workbench.database.mappers.ProjectFilesMapper;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysDictItemService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysDictItemService.java
index dea66c2..6a7d8c9 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysDictItemService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysDictItemService.java
@@ -19,7 +19,7 @@
package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysDictItem;
import org.apache.submarine.server.workbench.database.mappers.SysDictItemMapper;
import org.apache.submarine.server.workbench.rest.SysDictRestApi;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysMessageService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysMessageService.java
index de6dd72..084f02a 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysMessageService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysMessageService.java
@@ -19,7 +19,7 @@
package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysMessage;
import org.apache.submarine.server.workbench.database.mappers.SysMessageMapper;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysUserService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysUserService.java
index 0c714ae..0436b03 100755
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysUserService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/SysUserService.java
@@ -20,7 +20,7 @@ package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.RowBounds;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysUser;
import org.apache.submarine.server.workbench.database.mappers.SysUserMapper;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamMemberService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamMemberService.java
index aea9fb2..e64cbe7 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamMemberService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamMemberService.java
@@ -19,7 +19,7 @@
package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.TeamMember;
import org.apache.submarine.server.workbench.database.mappers.TeamMemberMapper;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamService.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamService.java
index a365eef..3eb289f 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamService.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/database/service/TeamService.java
@@ -20,7 +20,7 @@ package org.apache.submarine.server.workbench.database.service;
import org.apache.ibatis.session.RowBounds;
import org.apache.ibatis.session.SqlSession;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.Team;
import org.apache.submarine.server.workbench.database.entity.TeamMember;
import org.apache.submarine.server.workbench.database.mappers.TeamMapper;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/LoginRestApi.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/LoginRestApi.java
index 9372410..e071e52 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/LoginRestApi.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/LoginRestApi.java
@@ -22,9 +22,9 @@ import com.google.gson.Gson;
import com.google.gson.reflect.TypeToken;
import org.apache.ibatis.session.SqlSession;
import org.apache.submarine.server.workbench.annotation.SubmarineApi;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysUser;
import org.apache.submarine.server.workbench.database.mappers.SysUserMapper;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.response.JsonResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDeptRestApi.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDeptRestApi.java
index 0a1948d..9b65441 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDeptRestApi.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDeptRestApi.java
@@ -22,12 +22,12 @@ import com.github.pagehelper.PageInfo;
import org.apache.commons.lang.StringUtils;
import org.apache.ibatis.session.SqlSession;
import org.apache.submarine.server.workbench.annotation.SubmarineApi;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysDeptSelect;
import org.apache.submarine.server.workbench.database.entity.SysDeptTree;
import org.apache.submarine.server.workbench.database.entity.SysDept;
import org.apache.submarine.server.workbench.database.mappers.SysDeptMapper;
import org.apache.submarine.server.workbench.database.utils.DepartmentUtil;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.response.JsonResponse;
import org.apache.submarine.server.response.JsonResponse.ListResult;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictItemRestApi.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictItemRestApi.java
index 29f5dbc..e7ae7ff 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictItemRestApi.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictItemRestApi.java
@@ -23,10 +23,10 @@ import com.google.gson.Gson;
import org.apache.ibatis.session.RowBounds;
import org.apache.ibatis.session.SqlSession;
import org.apache.submarine.server.workbench.annotation.SubmarineApi;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysDictItem;
import org.apache.submarine.server.workbench.database.mappers.SysDictItemMapper;
import org.apache.submarine.server.workbench.database.service.SysDictItemService;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.response.JsonResponse;
import org.apache.submarine.server.response.JsonResponse.ListResult;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictRestApi.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictRestApi.java
index b5d4fff..c1a86ca 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictRestApi.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SysDictRestApi.java
@@ -23,9 +23,9 @@ import com.google.gson.Gson;
import org.apache.ibatis.session.RowBounds;
import org.apache.ibatis.session.SqlSession;
import org.apache.submarine.server.workbench.annotation.SubmarineApi;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysDict;
import org.apache.submarine.server.workbench.database.mappers.SysDictMapper;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.response.JsonResponse;
import org.apache.submarine.server.response.JsonResponse.ListResult;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SystemRestApi.java b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SystemRestApi.java
index 83ab0b3..2b835cd 100644
--- a/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SystemRestApi.java
+++ b/submarine-server/server-core/src/main/java/org/apache/submarine/server/workbench/rest/SystemRestApi.java
@@ -22,10 +22,10 @@ import com.github.pagehelper.PageInfo;
import org.apache.commons.lang.StringUtils;
import org.apache.ibatis.session.SqlSession;
import org.apache.submarine.server.workbench.annotation.SubmarineApi;
-import org.apache.submarine.server.workbench.database.MyBatisUtil;
import org.apache.submarine.server.workbench.database.entity.SysUser;
import org.apache.submarine.server.workbench.database.mappers.SystemMapper;
import org.apache.submarine.server.workbench.database.service.SysUserService;
+import org.apache.submarine.server.database.utils.MyBatisUtil;
import org.apache.submarine.server.response.JsonResponse;
import org.apache.submarine.server.response.JsonResponse.ListResult;
import org.slf4j.Logger;
diff --git a/submarine-server/server-core/src/main/resources/mybatis-config-metastore.xml b/submarine-server/server-core/src/main/resources/mybatis-config-metastore.xml
new file mode 100644
index 0000000..544f880
--- /dev/null
+++ b/submarine-server/server-core/src/main/resources/mybatis-config-metastore.xml
@@ -0,0 +1,59 @@
+<?xml version='1.0' encoding='UTF-8' ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<!DOCTYPE configuration PUBLIC '-//mybatis.org//DTD Config 3.0//EN'
+ 'http://mybatis.org/dtd/mybatis-3-config.dtd'>
+<configuration>
+ <settings>
+ <setting name="cacheEnabled" value="true"/>
+ <setting name="lazyLoadingEnabled" value="false"/>
+ <setting name="aggressiveLazyLoading" value="true"/>
+ <setting name="logImpl" value="STDOUT_LOGGING"/>
+ </settings>
+
+ <typeAliases>
+ <package name="com.github.pagehelper.model"/>
+ </typeAliases>
+
+ <plugins>
+ <plugin interceptor="com.github.pagehelper.PageInterceptor">
+ <property name="helperDialect" value="mysql"/>
+ <property name="offsetAsPageNum" value="true"/>
+ <property name="rowBoundsWithCount" value="true"/>
+ </plugin>
+ </plugins>
+
+ <environments default="development">
+ <environment id="development">
+ <transactionManager type="JDBC"/>
+ <dataSource type="POOLED">
+ <property name="driver" value="${jdbc.driverClassName}"/>
+ <property name="url" value="${jdbc.url}"/>
+ <property name="username" value="${jdbc.username}"/>
+ <property name="password" value="${jdbc.password}"/>
+ <property name="poolPingQuery" value="SELECT NOW()"/>
+ <property name="poolPingEnabled" value="true"/>
+ </dataSource>
+ </environment>
+ </environments>
+
+ <mappers>
+ <mapper resource='org/apache/submarine/database/mappers/EnvironmentMapper.xml'/>
+ </mappers>
+</configuration>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/EnvironmentMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/EnvironmentMapper.xml
new file mode 100644
index 0000000..da9bbd5
--- /dev/null
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/EnvironmentMapper.xml
@@ -0,0 +1,69 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
+<mapper namespace="org.apache.submarine.server.environment.database.mappers.EnvironmentMapper">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
+ <id property="id" column="id"/>
+ <result column="create_by" property="createBy"/>
+ <result column="create_time" property="createTime"/>
+ <result column="update_by" property="updateBy"/>
+ <result column="update_time" property="updateTime"/>
+ </resultMap>
+
+ <resultMap id="resultMap" type="org.apache.submarine.server.environment.database.entity.EnvironmentEntity" extends="BaseEntityResultMap">
+ <result column="environment_name" jdbcType="VARCHAR" property="environmentName" />
+ <result column="environment_spec" jdbcType="VARCHAR" property="environmentSpec" />
+ </resultMap>
+
+ <sql id="Base_Column_List">
+ id, environment_name, environment_spec, create_by, create_time, update_by, update_time
+ </sql>
+
+ <select id="select" parameterType="java.lang.String" resultMap="resultMap">
+ select
+ <include refid="Base_Column_List" />
+ from environment
+ where environment_name = #{environment_name,jdbcType=VARCHAR}
+ </select>
+
+ <delete id="delete" parameterType="java.lang.String">
+ delete from environment
+ where environment_name = #{environment_name,jdbcType=VARCHAR}
+ </delete>
+
+ <insert id="insert" parameterType="org.apache.submarine.server.environment.database.entity.EnvironmentEntity">
+ insert into environment (id, environment_name, environment_spec,
+ create_by, create_time, update_by, update_time)
+ values (#{id,jdbcType=VARCHAR}, #{environmentName,jdbcType=VARCHAR}, #{environmentSpec,jdbcType=VARCHAR},
+ #{createBy,jdbcType=VARCHAR}, now(), #{updateBy,jdbcType=VARCHAR}, now())
+ </insert>
+
+ <update id="update" parameterType="org.apache.submarine.server.environment.database.entity.EnvironmentEntity">
+ update environment
+ <set>
+ <if test="environmentSpec != null">
+ environment_spec = #{environmentSpec,jdbcType=VARCHAR},
+ </if>
+ update_time = now()
+ </set>
+ where environment_name = #{environmentName,jdbcType=VARCHAR}
+ </update>
+
+</mapper>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/JobMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/JobMapper.xml
index af93133..c42c63e 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/JobMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/JobMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.JobMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/MetricMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/MetricMapper.xml
index bcd711b..d2d2f11 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/MetricMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/MetricMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.MetricMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
</resultMap>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ParamMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ParamMapper.xml
index 41c5cd2..69ff1c9 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ParamMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ParamMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.ParamMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
</resultMap>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectFilesMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectFilesMapper.xml
index 2edffc5..6d20c5c 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectFilesMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectFilesMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.ProjectFilesMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectMapper.xml
index a84ed95..de45e90 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/ProjectMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.ProjectMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDeptMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDeptMapper.xml
index a9d6c48..f528b27 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDeptMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDeptMapper.xml
@@ -20,7 +20,7 @@
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.SysDeptMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id" />
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictItemMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictItemMapper.xml
index 1c10dc1..6380180 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictItemMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictItemMapper.xml
@@ -20,7 +20,7 @@
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.SysDictItemMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id" />
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictMapper.xml
index a4e45ef..997a21e 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysDictMapper.xml
@@ -20,7 +20,7 @@
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.SysDictMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysMessageMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysMessageMapper.xml
index 28f3298..7f88b20 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysMessageMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysMessageMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.SysMessageMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysUserMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysUserMapper.xml
index e15bc7a..82d8eff 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysUserMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/SysUserMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.SysUserMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id" />
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMapper.xml
index abfddf7..66a2256 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.TeamMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id"/>
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMemberMapper.xml b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMemberMapper.xml
index d8d45c0..81961f4 100644
--- a/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMemberMapper.xml
+++ b/submarine-server/server-core/src/main/resources/org/apache/submarine/database/mappers/TeamMemberMapper.xml
@@ -19,7 +19,7 @@
-->
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.apache.submarine.server.workbench.database.mappers.TeamMemberMapper">
- <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.workbench.database.entity.BaseEntity">
+ <resultMap id="BaseEntityResultMap" type="org.apache.submarine.server.database.entity.BaseEntity">
<id property="id" column="id" />
<result column="create_by" property="createBy"/>
<result column="create_time" property="createTime"/>
diff --git a/submarine-server/server-core/src/main/resources/submarine-site.xml b/submarine-server/server-core/src/main/resources/submarine-site.xml
index b27b298..bb711d3 100755
--- a/submarine-server/server-core/src/main/resources/submarine-site.xml
+++ b/submarine-server/server-core/src/main/resources/submarine-site.xml
@@ -143,4 +143,12 @@
<description>RuntimeFactory for Submarine jobs</description>
</property>
+ <property>
+ <name>environment.conda.min.version</name>
+ <value>4.0.1</value>
+ </property>
+ <property>
+ <name>environment.conda.max.version</name>
+ <value>4.10.10</value>
+ </property>
</configuration>
diff --git a/submarine-server/server-core/src/test/java/org/apache/submarine/server/AbstractSubmarineServerTest.java b/submarine-server/server-core/src/test/java/org/apache/submarine/server/AbstractSubmarineServerTest.java
index 64c5249..a15290a 100644
--- a/submarine-server/server-core/src/test/java/org/apache/submarine/server/AbstractSubmarineServerTest.java
+++ b/submarine-server/server-core/src/test/java/org/apache/submarine/server/AbstractSubmarineServerTest.java
@@ -16,9 +16,20 @@
*/
package org.apache.submarine.server;
-import com.google.gson.JsonElement;
-import com.google.gson.JsonParseException;
-import com.google.gson.JsonParser;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.regex.Pattern;
+
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.cookie.CookiePolicy;
@@ -30,21 +41,32 @@ import org.apache.commons.httpclient.methods.PutMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpPatch;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
import org.apache.submarine.commons.utils.SubmarineConfVars;
+import org.apache.submarine.server.api.environment.Environment;
+import org.apache.submarine.server.response.JsonResponse;
+import org.apache.submarine.server.rest.RestConstants;
import org.apache.submarine.server.utils.TestUtils;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
+import org.junit.Assert;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import javax.ws.rs.core.MediaType;
-import java.io.File;
-import java.io.IOException;
-import java.net.InetAddress;
-import java.net.UnknownHostException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.regex.Pattern;
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonParseException;
+import com.google.gson.JsonParser;
public abstract class AbstractSubmarineServerTest {
protected static final Logger LOG =
@@ -57,6 +79,10 @@ public abstract class AbstractSubmarineServerTest {
protected static File submarineServerHome;
protected static File confDir;
+ protected static String ENV_PATH =
+ "/api/" + RestConstants.V1 + "/" + RestConstants.ENVIRONMENTS;
+ protected static String ENV_NAME = "my-submarine-env";
+
public static String getWebsocketApiUrlToTest() {
String websocketUrl = "ws://localhost:8080" + WEBSOCKET_API_URL;
if (System.getProperty("websocketUrl") != null) {
@@ -225,6 +251,36 @@ public abstract class AbstractSubmarineServerTest {
LOG.info("{} - {}", putMethod.getStatusCode(), putMethod.getStatusText());
return putMethod;
}
+
+ protected static String httpPatch(String path, String body,
+ String contentType) throws IOException {
+ LOG.info("Connecting to {}", URL + path);
+ CloseableHttpClient httpclient = HttpClients.createDefault();
+
+ HttpPatch httpPatch = new HttpPatch(URL + path);
+ StringEntity entity = new StringEntity(body, ContentType.APPLICATION_JSON);
+ httpPatch.setEntity(entity);
+
+ // Create a custom response handler
+ ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
+ @Override
+ public String handleResponse(final HttpResponse response)
+ throws ClientProtocolException, IOException {
+ LOG.info("response " + response);
+ int status = response.getStatusLine().getStatusCode();
+ if (status >= 200 && status < 300) {
+ HttpEntity entity = response.getEntity();
+ return entity != null ? EntityUtils.toString(entity) : null;
+ } else {
+ throw new ClientProtocolException(
+ "Unexpected response status: " + status);
+ }
+ }
+ };
+ String responseBody = httpclient.execute(httpPatch, responseHandler);
+ httpclient.close();
+ return responseBody;
+ }
protected static DeleteMethod httpDelete(String path) throws IOException {
return httpDelete(path, StringUtils.EMPTY, StringUtils.EMPTY);
@@ -377,4 +433,85 @@ public abstract class AbstractSubmarineServerTest {
return StringUtils.EMPTY;
}
}
+
+ protected void run(String body, String contentType) throws Exception {
+ Gson gson = new GsonBuilder().create();
+
+ // create
+ LOG.info("Create Environment using Environment REST API");
+
+ PostMethod postMethod = httpPost(ENV_PATH, body, contentType);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ postMethod.getStatusCode());
+
+ String json = postMethod.getResponseBodyAsString();
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Environment env =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Environment.class);
+ verifyCreateEnvironmentApiResult(env);
+ }
+
+ protected void update(String body, String contentType) throws Exception {
+ Gson gson = new GsonBuilder().create();
+
+ // update
+ LOG.info("Update Environment using Environment REST API");
+
+ String json = httpPatch(ENV_PATH + "/" + ENV_NAME, body, contentType);
+
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Environment env =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Environment.class);
+ verifyUpdateEnvironmentApiResult(env);
+ }
+
+ protected void verifyCreateEnvironmentApiResult(Environment env)
+ throws Exception {
+ Assert.assertNotNull(env.getEnvironmentSpec().getName());
+ Assert.assertNotNull(env.getEnvironmentSpec());
+ }
+
+ protected void verifyUpdateEnvironmentApiResult(Environment env)
+ throws Exception {
+ Assert.assertEquals(env.getEnvironmentSpec().getDockerImage(),
+ "continuumio/miniconda3");
+ Assert.assertTrue(
+ env.getEnvironmentSpec().getKernelSpec().getDependencies().size() == 1);
+ }
+
+
+ protected void deleteEnvironment() throws IOException {
+ Gson gson = new GsonBuilder().create();
+ DeleteMethod deleteMethod = httpDelete(ENV_PATH + "/" + ENV_NAME);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ deleteMethod.getStatusCode());
+
+ String json = deleteMethod.getResponseBodyAsString();
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Environment deletedEnv =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Environment.class);
+ Assert.assertEquals(ENV_NAME, deletedEnv.getEnvironmentSpec().getName());
+ }
+
+ protected String loadContent(String resourceName) throws Exception {
+ StringBuilder content = new StringBuilder();
+ InputStream inputStream =
+ this.getClass().getClassLoader().getResourceAsStream(resourceName);
+ BufferedReader r = new BufferedReader(new InputStreamReader(inputStream));
+ String l;
+ while ((l = r.readLine()) != null) {
+ content.append(l).append("\n");
+ }
+ inputStream.close();
+ return content.toString();
+ }
}
diff --git a/submarine-server/server-core/src/test/java/org/apache/submarine/server/rest/EnvironmentManagerRestApiTest.java b/submarine-server/server-core/src/test/java/org/apache/submarine/server/rest/EnvironmentManagerRestApiTest.java
new file mode 100644
index 0000000..dd1dfd8
--- /dev/null
+++ b/submarine-server/server-core/src/test/java/org/apache/submarine/server/rest/EnvironmentManagerRestApiTest.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.rest;
+
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.httpclient.methods.GetMethod;
+import org.apache.submarine.server.AbstractSubmarineServerTest;
+import org.apache.submarine.server.api.environment.Environment;
+import org.apache.submarine.server.response.JsonResponse;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+
+@SuppressWarnings("rawtypes")
+public class EnvironmentManagerRestApiTest extends AbstractSubmarineServerTest {
+
+ private static String ENV_PATH =
+ "/api/" + RestConstants.V1 + "/" + RestConstants.ENVIRONMENTS;
+ private static String ENV_NAME = "my-submarine-env";
+
+ @BeforeClass
+ public static void startUp() throws Exception {
+ AbstractSubmarineServerTest
+ .startUp(EnvironmentManagerRestApiTest.class.getSimpleName());
+ Assert.assertTrue(checkIfServerIsRunning());
+ }
+
+ @AfterClass
+ public static void destroy() throws Exception {
+ AbstractSubmarineServerTest.shutDown();
+ }
+
+ @Test
+ public void testCreateEnvironment() throws Exception {
+ String body = loadContent("environment/test_env_1.json");
+ run(body, "application/json");
+ deleteEnvironment();
+ }
+
+ @Test
+ public void testUpdateEnvironment() throws Exception {
+
+ // Create environment
+ String body = loadContent("environment/test_env_1.json");
+ run(body, "application/json");
+
+ // Updated spec
+ String updatedBody = loadContent("environment/test_env_2.json");
+
+ update(updatedBody, "application/json");
+ deleteEnvironment();
+ }
+
+ @Test
+ public void testDeleteEnvironment() throws Exception {
+ String body = loadContent("environment/test_env_1.json");
+ run(body, "application/json");
+ deleteEnvironment();
+
+ GetMethod getMethod = httpGet(ENV_PATH + "/" + ENV_NAME);
+ Assert.assertEquals(Response.Status.NOT_FOUND.getStatusCode(),
+ getMethod.getStatusCode());
+ }
+
+ @Test
+ public void testGetEnvironment() throws Exception {
+
+ String body = loadContent("environment/test_env_1.json");
+ run(body, "application/json");
+
+ Gson gson = new GsonBuilder().create();
+ GetMethod getMethod = httpGet(ENV_PATH + "/" + ENV_NAME);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ getMethod.getStatusCode());
+
+ String json = getMethod.getResponseBodyAsString();
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Environment getEnvironment =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Environment.class);
+ Assert.assertEquals(ENV_NAME, getEnvironment.getEnvironmentSpec().getName());
+
+ deleteEnvironment();
+ }
+
+ @Test
+ public void testNotFoundEnvironment() throws Exception {
+
+ Gson gson = new GsonBuilder().create();
+
+ GetMethod getMethod = httpGet(ENV_PATH + "/" + "no_such_env_exists");
+ Assert.assertEquals(Response.Status.NOT_FOUND.getStatusCode(),
+ getMethod.getStatusCode());
+
+ String json = getMethod.getResponseBodyAsString();
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.NOT_FOUND.getStatusCode(),
+ jsonResponse.getCode());
+ }
+}
diff --git a/submarine-server/server-core/src/test/resources/environment/test_env_1.json b/submarine-server/server-core/src/test/resources/environment/test_env_1.json
new file mode 100644
index 0000000..7dd341a
--- /dev/null
+++ b/submarine-server/server-core/src/test/resources/environment/test_env_1.json
@@ -0,0 +1,14 @@
+{
+ "name": "my-submarine-env",
+ "dockerImage" : "continuumio/anaconda3",
+ "kernelSpec" : {
+ "name" : "team_default_python_3.7",
+ "channels" : ["defaults"],
+ "dependencies" :
+ ["_ipyw_jlab_nb_ext_conf=0.1.0=py37_0",
+ "alabaster=0.7.12=py37_0",
+ "anaconda=2020.02=py37_0",
+ "anaconda-client=1.7.2=py37_0",
+ "anaconda-navigator=1.9.12=py37_0"]
+ }
+}
\ No newline at end of file
diff --git a/submarine-server/server-core/src/test/resources/environment/test_env_2.json b/submarine-server/server-core/src/test/resources/environment/test_env_2.json
new file mode 100644
index 0000000..36c44ff
--- /dev/null
+++ b/submarine-server/server-core/src/test/resources/environment/test_env_2.json
@@ -0,0 +1,10 @@
+{
+ "name": "my-submarine-env",
+ "dockerImage" : "continuumio/miniconda3",
+ "kernelSpec" : {
+ "name" : "team_default_python_3.7",
+ "channels" : ["defaults"],
+ "dependencies" :
+ ["_ipyw_jlab_nb_ext_conf=0.1.0=py37_0"]
+ }
+}
\ No newline at end of file
diff --git a/submarine-server/server-submitter/submitter-k8s/pom.xml b/submarine-server/server-submitter/submitter-k8s/pom.xml
index d8c0004..d5d0d65 100644
--- a/submarine-server/server-submitter/submitter-k8s/pom.xml
+++ b/submarine-server/server-submitter/submitter-k8s/pom.xml
@@ -57,6 +57,30 @@
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.submarine</groupId>
+ <artifactId>submarine-server-core</artifactId>
+ <version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-protobuf</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.code.gson</groupId>
+ <artifactId>gson</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.tukaani</groupId>
+ <artifactId>xz</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>joda-time</groupId>
+ <artifactId>joda-time</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
<!-- Unit Tests -->
<dependency>
<groupId>junit</groupId>
diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
index 0c6c6f9..c29f1d6 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
@@ -26,10 +26,16 @@ import io.kubernetes.client.models.V1ObjectMeta;
import io.kubernetes.client.models.V1PodSpec;
import io.kubernetes.client.models.V1PodTemplateSpec;
import io.kubernetes.client.models.V1ResourceRequirements;
+
+import org.apache.submarine.commons.utils.SubmarineConfVars;
+import org.apache.submarine.commons.utils.SubmarineConfiguration;
+import org.apache.submarine.server.api.environment.Environment;
import org.apache.submarine.server.api.exception.InvalidSpecException;
import org.apache.submarine.server.api.spec.ExperimentMeta;
import org.apache.submarine.server.api.spec.ExperimentSpec;
import org.apache.submarine.server.api.spec.ExperimentTaskSpec;
+import org.apache.submarine.server.api.spec.EnvironmentSpec;
+import org.apache.submarine.server.environment.EnvironmentManager;
import org.apache.submarine.server.submitter.k8s.model.MLJob;
import org.apache.submarine.server.submitter.k8s.model.MLJobReplicaSpec;
import org.apache.submarine.server.submitter.k8s.model.MLJobReplicaType;
@@ -48,6 +54,9 @@ import java.util.Map;
public class ExperimentSpecParser {
+ private static SubmarineConfiguration conf =
+ SubmarineConfiguration.getInstance();
+
public static MLJob parseJob(ExperimentSpec experimentSpec) throws InvalidSpecException {
String framework = experimentSpec.getMeta().getFramework();
if (ExperimentMeta.SupportedMLFramework.TENSORFLOW.
@@ -132,8 +141,8 @@ public class ExperimentSpecParser {
return tfJobSpec;
}
- private static V1PodTemplateSpec parseTemplateSpec(ExperimentTaskSpec taskSpec,
- ExperimentSpec experimentSpec) {
+ private static V1PodTemplateSpec parseTemplateSpec(
+ ExperimentTaskSpec taskSpec, ExperimentSpec experimentSpec) {
V1PodTemplateSpec templateSpec = new V1PodTemplateSpec();
V1PodSpec podSpec = new V1PodSpec();
List<V1Container> containers = new ArrayList<>();
@@ -143,7 +152,9 @@ public class ExperimentSpecParser {
if (taskSpec.getImage() != null) {
container.setImage(taskSpec.getImage());
} else {
- container.setImage(experimentSpec.getEnvironment().getImage());
+ if (experimentSpec.getEnvironment().getImage() != null) {
+ container.setImage(experimentSpec.getEnvironment().getImage());
+ }
}
// cmd
if (taskSpec.getCmd() != null) {
@@ -158,6 +169,72 @@ public class ExperimentSpecParser {
container.setEnv(parseEnvVars(taskSpec, experimentSpec.getMeta().getEnvVars()));
containers.add(container);
podSpec.setContainers(containers);
+
+ /**
+ * Init Containers
+ */
+ if (experimentSpec.getEnvironment().getName() != null) {
+ Environment environment = getEnvironment(experimentSpec);
+ if (environment != null) {
+ EnvironmentSpec environmentSpec = environment.getEnvironmentSpec();
+ List<V1Container> initContainers = new ArrayList<>();
+ V1Container initContainer = new V1Container();
+ initContainer.setName(environment.getEnvironmentSpec().getName());
+ initContainer.setImage(environmentSpec.getDockerImage());
+
+ if (environmentSpec.getKernelSpec().getDependencies().size() > 0) {
+
+ String minVersion = "minVersion=\""
+ + conf.getString(
+ SubmarineConfVars.ConfVars.ENVIRONMENT_CONDA_MIN_VERSION)
+ + "\";";
+ String maxVersion = "maxVersion=\""
+ + conf.getString(
+ SubmarineConfVars.ConfVars.ENVIRONMENT_CONDA_MAX_VERSION)
+ + "\";";
+ String currentVersion = "currentVersion=$(conda -V | cut -f2 -d' ');";
+ StringBuffer condaVersionValidationCommand = new StringBuffer();
+ condaVersionValidationCommand.append(minVersion);
+ condaVersionValidationCommand.append(maxVersion);
+ condaVersionValidationCommand.append(currentVersion);
+ condaVersionValidationCommand.append("if [ \"$(printf '%s\\n' "
+ + "\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V "
+ + "| head -n2 | tail -1 )\" != \"$currentVersion\" ]; then echo "
+ + "\"Conda version should be between " + minVersion + " and "
+ + maxVersion + "\"; exit 1; else echo \"Conda current version is "
+ + currentVersion + ". Moving forward with env creation and "
+ + "activation.\"; fi");
+
+ StringBuffer createCommand = new StringBuffer();
+ String condaEnvironmentName =
+ environmentSpec.getKernelSpec().getName();
+
+ createCommand.append("conda create -n " + condaEnvironmentName);
+ for (String channel : environmentSpec.getKernelSpec().getChannels()) {
+ createCommand.append(" ");
+ createCommand.append("-c");
+ createCommand.append(" ");
+ createCommand.append(channel);
+ }
+ for (String dependency : environmentSpec.getKernelSpec()
+ .getDependencies()) {
+ createCommand.append(" ");
+ createCommand.append(dependency);
+ }
+ String activateCommand = "echo \"source activate "
+ + condaEnvironmentName + "\" > ~/.bashrc";
+ String pathCommand = "PATH=/opt/conda/envs/env/bin:$PATH";
+ String finalCommand = condaVersionValidationCommand.toString() +
+ " && " + createCommand.toString() + " && "
+ + activateCommand + " && " + pathCommand;
+ initContainer.addCommandItem("/bin/bash");
+ initContainer.addCommandItem("-c");
+ initContainer.addCommandItem(finalCommand);
+ }
+ initContainers.add(initContainer);
+ podSpec.setInitContainers(initContainers);
+ }
+ }
templateSpec.setSpec(podSpec);
return templateSpec;
}
@@ -197,4 +274,15 @@ public class ExperimentSpecParser {
}
return resources;
}
+
+ private static Environment getEnvironment(ExperimentSpec experimentSpec) {
+ if (experimentSpec.getEnvironment().getName() != null) {
+ EnvironmentManager environmentManager = EnvironmentManager.getInstance();
+ Environment environment = environmentManager
+ .getEnvironment(experimentSpec.getEnvironment().getName());
+ return environment;
+ } else {
+ return null;
+ }
+ }
}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/ExperimentSpecParserTest.java b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/ExperimentSpecParserTest.java
index cff8223..8517671 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/ExperimentSpecParserTest.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/ExperimentSpecParserTest.java
@@ -21,12 +21,20 @@ package org.apache.submarine.server.submitter.k8s;
import java.io.IOException;
import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.List;
import io.kubernetes.client.models.V1ObjectMeta;
+
+import org.apache.submarine.commons.utils.SubmarineConfVars;
+import org.apache.submarine.commons.utils.SubmarineConfiguration;
import org.apache.submarine.server.api.exception.InvalidSpecException;
import org.apache.submarine.server.api.spec.ExperimentMeta;
import org.apache.submarine.server.api.spec.ExperimentSpec;
import org.apache.submarine.server.api.spec.ExperimentTaskSpec;
+import org.apache.submarine.server.api.spec.EnvironmentSpec;
+import org.apache.submarine.server.api.spec.KernelSpec;
+import org.apache.submarine.server.environment.EnvironmentManager;
import org.apache.submarine.server.submitter.k8s.model.MLJob;
import org.apache.submarine.server.submitter.k8s.model.MLJobReplicaSpec;
import org.apache.submarine.server.submitter.k8s.model.MLJobReplicaType;
@@ -37,8 +45,14 @@ import org.apache.submarine.server.submitter.k8s.model.tfjob.TFJobReplicaType;
import org.apache.submarine.server.submitter.k8s.parser.ExperimentSpecParser;
import org.junit.Assert;
import org.junit.Test;
+import io.kubernetes.client.models.V1Container;
+
public class ExperimentSpecParserTest extends SpecBuilder {
+
+ private static SubmarineConfiguration conf =
+ SubmarineConfiguration.getInstance();
+
@Test
public void testValidTensorFlowExperiment() throws IOException,
URISyntaxException, InvalidSpecException {
@@ -174,4 +188,73 @@ public class ExperimentSpecParserTest extends SpecBuilder {
Assert.assertEquals(expectedMasterContainerCpu,
actualMasterContainerCpu);
}
+
+ @Test
+ public void testValidPyTorchJobSpecWithEnv()
+ throws IOException, URISyntaxException, InvalidSpecException {
+
+ EnvironmentManager environmentManager = EnvironmentManager.getInstance();
+
+ EnvironmentSpec spec = new EnvironmentSpec();
+ String envName = "my-submarine-env";
+ spec.setName(envName);
+ String dockerImage = "example.com/my-docker-image:0.1.2";
+ spec.setDockerImage(dockerImage);
+
+ KernelSpec kernelSpec = new KernelSpec();
+ String kernelName = "team_python";
+ kernelSpec.setName(kernelName);
+ List<String> channels = new ArrayList<String>();
+ String channel = "default";
+ channels.add(channel);
+ kernelSpec.setChannels(channels);
+
+ List<String> dependencies = new ArrayList<String>();
+ String dependency = "_ipyw_jlab_nb_ext_conf=0.1.0=py37_0";
+ dependencies.add(dependency);
+ kernelSpec.setDependencies(dependencies);
+ spec.setKernelSpec(kernelSpec);
+
+ environmentManager.createEnvironment(spec);
+
+ ExperimentSpec jobSpec = buildFromJsonFile(pytorchJobWithEnvReqFile);
+ PyTorchJob pyTorchJob = (PyTorchJob) ExperimentSpecParser.parseJob(jobSpec);
+
+ MLJobReplicaSpec mlJobReplicaSpec = pyTorchJob.getSpec().getReplicaSpecs()
+ .get(PyTorchJobReplicaType.Master);
+ Assert.assertEquals(1,
+ mlJobReplicaSpec.getTemplate().getSpec().getInitContainers().size());
+ V1Container initContainer =
+ mlJobReplicaSpec.getTemplate().getSpec().getInitContainers().get(0);
+ Assert.assertEquals(dockerImage, initContainer.getImage());
+
+ Assert.assertEquals("/bin/bash", initContainer.getCommand().get(0));
+ Assert.assertEquals("-c", initContainer.getCommand().get(1));
+
+ String minVersion = "minVersion=\""
+ + conf.getString(
+ SubmarineConfVars.ConfVars.ENVIRONMENT_CONDA_MIN_VERSION)
+ + "\";";
+ String maxVersion = "maxVersion=\""
+ + conf.getString(
+ SubmarineConfVars.ConfVars.ENVIRONMENT_CONDA_MAX_VERSION)
+ + "\";";
+ String currentVersion = "currentVersion=$(conda -V | cut -f2 -d' ');";
+ Assert.assertEquals(
+ minVersion + maxVersion + currentVersion
+ + "if [ \"$(printf '%s\\n' \"$minVersion\" \"$maxVersion\" "
+ + "\"$currentVersion\" | sort -V | head -n2 | tail -1 )\" "
+ + "!= \"$currentVersion\" ]; then echo \"Conda version " +
+ "should be between minVersion=\"4.0.1\"; " +
+ "and maxVersion=\"4.10.10\";\"; exit 1; else echo "
+ + "\"Conda current version is " + currentVersion + ". "
+ + "Moving forward with env creation and activation.\"; "
+ + "fi && " +
+ "conda create -n " + kernelName + " -c " + channel + " " + dependency
+ + " && " + "echo \"source activate " + kernelName + "\" > ~/.bashrc"
+ + " && " + "PATH=/opt/conda/envs/env/bin:$PATH",
+ initContainer.getCommand().get(2));
+
+ environmentManager.deleteEnvironment(envName);
+ }
}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java
index 6998a05..50c83f8 100644
--- a/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/SpecBuilder.java
@@ -35,6 +35,9 @@ public abstract class SpecBuilder {
// The spec files in test/resources
protected final String tfJobReqFile = "/tf_mnist_req.json";
protected final String pytorchJobReqFile = "/pytorch_job_req.json";
+ protected final String pytorchJobWithEnvReqFile = "/pytorch_job_req_env.json";
+ protected final String pytorchJobWithInvalidEnvReqFile =
+ "/pytorch_job_req_invalid_env.json";
protected ExperimentSpec buildFromJsonFile(String filePath) throws IOException,
URISyntaxException {
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_env.json b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_env.json
new file mode 100644
index 0000000..4cd2dac
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_env.json
@@ -0,0 +1,26 @@
+{
+ "meta": {
+ "name": "pytorch-dist-mnist",
+ "namespace": "submarine",
+ "framework": "PyTorch",
+ "cmd": "python /var/mnist.py --backend gloo",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "name": "my-submarine-env"
+ },
+ "spec": {
+ "Master": {
+ "name": "master",
+ "replicas": 1,
+ "resources": "cpu=2,memory=2048M"
+ },
+ "Worker": {
+ "name": "worker",
+ "replicas": 2,
+ "resources": "cpu=1,memory=1024M"
+ }
+ }
+}
diff --git a/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_invalid_env.json b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_invalid_env.json
new file mode 100644
index 0000000..2d81a78
--- /dev/null
+++ b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_invalid_env.json
@@ -0,0 +1,27 @@
+{
+ "meta": {
+ "name": "pytorch-dist-mnist",
+ "namespace": "submarine",
+ "framework": "PyTorch",
+ "cmd": "python /var/mnist.py --backend gloo",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "name": "my-submarine-env",
+ "image": "apache/submarine:pytorch-dist-mnist-1.0"
+ },
+ "spec": {
+ "Master": {
+ "name": "master",
+ "replicas": 1,
+ "resources": "cpu=2,memory=2048M"
+ },
+ "Worker": {
+ "name": "worker",
+ "replicas": 2,
+ "resources": "cpu=1,memory=1024M"
+ }
+ }
+}
diff --git a/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/EnvironmentManagerRestApiIT.java b/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/EnvironmentManagerRestApiIT.java
new file mode 100644
index 0000000..f230207
--- /dev/null
+++ b/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/EnvironmentManagerRestApiIT.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.rest;
+
+import java.io.IOException;
+
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.httpclient.methods.GetMethod;
+import org.apache.submarine.server.AbstractSubmarineServerTest;
+import org.apache.submarine.server.api.environment.Environment;
+import org.apache.submarine.server.response.JsonResponse;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+
+@SuppressWarnings("rawtypes")
+public class EnvironmentManagerRestApiIT extends AbstractSubmarineServerTest {
+
+ @BeforeClass
+ public static void startUp() throws Exception {
+ Assert.assertTrue(checkIfServerIsRunning());
+ }
+
+ @Test
+ public void testCreateEnvironment() throws Exception {
+ String body = loadContent("environment/test_env_1.json");
+ run(body, "application/json");
+ deleteEnvironment();
+ }
+
+ @Test
+ public void testGetEnvironment() throws Exception {
+
+ String body = loadContent("environment/test_env_1.json");
+ run(body, "application/json");
+
+ Gson gson = new GsonBuilder().create();
+ GetMethod getMethod = httpGet(ENV_PATH + "/" + ENV_NAME);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ getMethod.getStatusCode());
+
+ String json = getMethod.getResponseBodyAsString();
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Environment getEnvironment =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Environment.class);
+ Assert.assertEquals(ENV_NAME, getEnvironment.getEnvironmentSpec().getName());
+ deleteEnvironment();
+ }
+
+
+ @Test
+ public void testUpdateEnvironment() throws IOException {
+
+ }
+
+ @Test
+ public void testDeleteEnvironment() throws Exception {
+ String body = loadContent("environment/test_env_1.json");
+ run(body, "application/json");
+ deleteEnvironment();
+
+ GetMethod getMethod = httpGet(ENV_PATH + "/" + ENV_NAME);
+ Assert.assertEquals(Response.Status.NOT_FOUND.getStatusCode(),
+ getMethod.getStatusCode());
+
+ }
+
+ @Test
+ public void testListEnvironments() throws IOException {
+
+ }
+}
diff --git a/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java b/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java
index 7623b41..d771c0e 100644
--- a/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java
+++ b/submarine-test/test-k8s/src/test/java/org/apache/submarine/rest/ExperimentRestApiIT.java
@@ -19,37 +19,24 @@
package org.apache.submarine.rest;
-import javax.ws.rs.core.MediaType;
-import javax.ws.rs.core.Response;
-import java.io.File;
import java.io.FileReader;
import java.io.IOException;
-import java.net.URL;
-import java.nio.charset.StandardCharsets;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
-import com.google.gson.Gson;
-import com.google.gson.GsonBuilder;
-import com.google.gson.JsonObject;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
-import io.kubernetes.client.ApiClient;
-import io.kubernetes.client.ApiException;
-import io.kubernetes.client.Configuration;
-import io.kubernetes.client.JSON;
-import io.kubernetes.client.apis.CustomObjectsApi;
-import io.kubernetes.client.util.ClientBuilder;
-import io.kubernetes.client.util.KubeConfig;
import org.apache.commons.httpclient.methods.DeleteMethod;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.PostMethod;
-import org.apache.commons.io.FileUtils;
import org.apache.submarine.server.AbstractSubmarineServerTest;
import org.apache.submarine.server.api.experiment.Experiment;
import org.apache.submarine.server.api.experiment.ExperimentId;
import org.apache.submarine.server.gson.ExperimentIdDeserializer;
import org.apache.submarine.server.gson.ExperimentIdSerializer;
+import org.apache.submarine.server.api.environment.Environment;
import org.apache.submarine.server.response.JsonResponse;
import org.apache.submarine.server.rest.RestConstants;
import org.joda.time.DateTime;
@@ -59,6 +46,19 @@ import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+
+import io.kubernetes.client.ApiClient;
+import io.kubernetes.client.ApiException;
+import io.kubernetes.client.Configuration;
+import io.kubernetes.client.JSON;
+import io.kubernetes.client.apis.CustomObjectsApi;
+import io.kubernetes.client.util.ClientBuilder;
+import io.kubernetes.client.util.KubeConfig;
+
@SuppressWarnings("rawtypes")
public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
private static final Logger LOG = LoggerFactory.getLogger(ExperimentRestApiIT.class);
@@ -114,6 +114,36 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
String patchBody = loadContent("tensorflow/tf-mnist-patch-req.yaml");
run(body, patchBody, "application/yaml");
}
+
+ @Test
+ public void testTensorFlowUsingEnvWithJsonSpec() throws Exception {
+
+ // Create environment
+ String envBody = loadContent("environment/test_env_1.json");
+ run(envBody, "application/json");
+
+ Gson gson = new GsonBuilder().create();
+ GetMethod getMethod = httpGet(ENV_PATH + "/" + ENV_NAME);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ getMethod.getStatusCode());
+
+ String json = getMethod.getResponseBodyAsString();
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Environment getEnvironment =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Environment.class);
+ Assert.assertEquals(ENV_NAME, getEnvironment.getEnvironmentSpec().getName());
+
+ String body = loadContent("tensorflow/tf-mnist-with-env-req.json");
+ String patchBody =
+ loadContent("tensorflow/tf-mnist-with-env-patch-req.json");
+ run(getEnvironment, body, patchBody, "application/json");
+
+ // Delete environment
+ deleteEnvironment();
+ }
@Test
public void testPyTorchWithJsonSpec() throws Exception {
@@ -171,6 +201,54 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
Experiment deletedExperiment = gson.fromJson(gson.toJson(jsonResponse.getResult()), Experiment.class);
verifyDeleteJobApiResult(createdExperiment, deletedExperiment);
}
+
+ private void run(Environment expectedEnv, String body, String patchBody,
+ String contentType) throws Exception {
+ // create
+ LOG.info("Create training job using Environment by Job REST API");
+ PostMethod postMethod = httpPost(BASE_API_PATH, body, contentType);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ postMethod.getStatusCode());
+
+ String json = postMethod.getResponseBodyAsString();
+ JsonResponse jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Experiment createdExperiment =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Experiment.class);
+ verifyCreateJobApiResult(createdExperiment);
+
+ // find
+ GetMethod getMethod =
+ httpGet(BASE_API_PATH + "/" + createdExperiment.getExperimentId().toString());
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ getMethod.getStatusCode());
+
+ json = getMethod.getResponseBodyAsString();
+ jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Experiment foundExperiment =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Experiment.class);
+ verifyGetJobApiResult(createdExperiment, foundExperiment);
+
+ // delete
+ DeleteMethod deleteMethod =
+ httpDelete(BASE_API_PATH + "/" + createdExperiment.getExperimentId().toString());
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ deleteMethod.getStatusCode());
+
+ json = deleteMethod.getResponseBodyAsString();
+ jsonResponse = gson.fromJson(json, JsonResponse.class);
+ Assert.assertEquals(Response.Status.OK.getStatusCode(),
+ jsonResponse.getCode());
+
+ Experiment deletedExperiment =
+ gson.fromJson(gson.toJson(jsonResponse.getResult()), Experiment.class);
+ verifyDeleteJobApiResult(createdExperiment, deletedExperiment);
+ }
private void verifyCreateJobApiResult(Experiment createdExperiment) throws Exception {
Assert.assertNotNull(createdExperiment.getUid());
@@ -179,6 +257,16 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
assertK8sResultEquals(createdExperiment);
}
+
+ private void verifyCreateJobApiResult(Environment env, Experiment createdJob)
+ throws Exception {
+ Assert.assertNotNull(createdJob.getUid());
+ Assert.assertNotNull(createdJob.getAcceptedTime());
+ Assert.assertEquals(Experiment.Status.STATUS_ACCEPTED.getValue(),
+ createdJob.getStatus());
+
+ assertK8sResultEquals(env, createdJob);
+ }
private void verifyGetJobApiResult(Experiment createdExperiment, Experiment foundExperiment) throws Exception {
Assert.assertEquals(createdExperiment.getExperimentId(), foundExperiment.getExperimentId());
@@ -189,9 +277,60 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
assertK8sResultEquals(foundExperiment);
}
+ private void assertK8sResultEquals(Environment env, Experiment experiment) throws Exception {
+ KfOperator operator = kfOperatorMap.get(experiment.getSpec().getMeta().getFramework().toLowerCase());
+ JsonObject rootObject =
+ getJobByK8sApi(operator.getGroup(), operator.getVersion(),
+ operator.getNamespace(), operator.getPlural(), experiment.getName());
+ JsonArray actualCommand = (JsonArray) rootObject.getAsJsonObject("spec")
+ .getAsJsonObject("tfReplicaSpecs").getAsJsonObject("Worker")
+ .getAsJsonObject("template").getAsJsonObject("spec")
+ .getAsJsonArray("initContainers").get(0).getAsJsonObject()
+ .get("command");
+
+ JsonArray expected = new JsonArray();
+ expected.add("/bin/bash");
+ expected.add("-c");
+ String initialCommand =
+ "conda create -n " + env.getEnvironmentSpec().getKernelSpec().getName();
+
+ String channels = "";
+ for (String channel : env.getEnvironmentSpec().getKernelSpec()
+ .getChannels()) {
+ channels += " -c " + channel;
+ }
+
+ String dependencies = "";
+ for (String dependency : env.getEnvironmentSpec().getKernelSpec()
+ .getDependencies()) {
+ dependencies += " " + dependency;
+ }
+
+ String fullCommand =
+ initialCommand + channels + dependencies + " && echo \"source activate "
+ + env.getEnvironmentSpec().getKernelSpec().getName()
+ + "\" > ~/.bashrc" + " && PATH=/opt/conda/envs/env/bin:$PATH";
+ expected.add(fullCommand);
+ Assert.assertEquals(expected, actualCommand);
+
+ JsonObject metadataObject = rootObject.getAsJsonObject("metadata");
+
+ String uid = metadataObject.getAsJsonPrimitive("uid").getAsString();
+ LOG.info("Uid from Job REST is {}", experiment.getUid());
+ LOG.info("Uid from K8s REST is {}", uid);
+ Assert.assertEquals(experiment.getUid(), uid);
+
+ String creationTimestamp =
+ metadataObject.getAsJsonPrimitive("creationTimestamp").getAsString();
+ Date expectedDate = new DateTime(experiment.getAcceptedTime()).toDate();
+ Date actualDate = new DateTime(creationTimestamp).toDate();
+ LOG.info("CreationTimestamp from Job REST is {}", expectedDate);
+ LOG.info("CreationTimestamp from K8s REST is {}", actualDate);
+ Assert.assertEquals(expectedDate, actualDate);
+ }
+
private void assertK8sResultEquals(Experiment experiment) throws Exception {
- KfOperator operator = kfOperatorMap.get(experiment.getSpec().getMeta().getFramework()
- .toLowerCase());
+ KfOperator operator = kfOperatorMap.get(experiment.getSpec().getMeta().getFramework().toLowerCase());
JsonObject rootObject = getJobByK8sApi(operator.getGroup(), operator.getVersion(),
operator.getNamespace(), operator.getPlural(), experiment.getName());
JsonObject metadataObject = rootObject.getAsJsonObject("metadata");
@@ -267,12 +406,6 @@ public class ExperimentRestApiIT extends AbstractSubmarineServerTest {
Assert.assertEquals(Response.Status.OK.getStatusCode(), jsonResponse.getCode());
}
- String loadContent(String resourceName) throws Exception {
- URL fileUrl = this.getClass().getResource("/" + resourceName);
- LOG.info("Resource file: " + fileUrl);
- return FileUtils.readFileToString(new File(fileUrl.toURI()), StandardCharsets.UTF_8);
- }
-
/**
* Direct used by K8s api. It storage the operator's base info.
*/
diff --git a/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-env-patch-req.json b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-env-patch-req.json
new file mode 100644
index 0000000..38760c2
--- /dev/null
+++ b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-env-patch-req.json
@@ -0,0 +1,20 @@
+{
+ "meta": {
+ "name": "tf-mnist-json",
+ "namespace": "default",
+ "framework": "TensorFlow",
+ "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "name": "my-submarine-env"
+ },
+ "spec": {
+ "Worker": {
+ "replicas": 1,
+ "resources": "cpu=1,memory=1024M"
+ }
+ }
+}
diff --git a/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-env-req.json b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-env-req.json
new file mode 100644
index 0000000..28ee4f8
--- /dev/null
+++ b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-env-req.json
@@ -0,0 +1,25 @@
+{
+ "meta": {
+ "name": "tf-mnist-json",
+ "namespace": "default",
+ "framework": "TensorFlow",
+ "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "image": "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0",
+ "name": "my-submarine-env"
+ },
+ "spec": {
+ "Ps": {
+ "replicas": 1,
+ "resources": "cpu=1,memory=512M"
+ },
+ "Worker": {
+ "replicas": 1,
+ "resources": "cpu=1,memory=512M"
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org