You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by GitBox <gi...@apache.org> on 2019/12/29 17:09:54 UTC
[GitHub] [submarine] jiwq opened a new pull request #135: SUBMARINE-321. Add
JobManager and SubmitterManager components
jiwq opened a new pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135
### What is this PR for?
The JobManager is responsible for cache and schedule the job to submitter.
The SubmitterManager help to load the different submitter plugins.
### What type of PR is it?
[ Feature ]
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-321
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361898349
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFReplicaSpec.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+
+/**
+ * The replica spec for TFJob. It contains replicas, restart policy and pod template.
+ * The template describe the running instance for task.
+ */
+public class TFReplicaSpec {
Review comment:
@tangzhankun More info see `TFJobTest.java` please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on issue #135: SUBMARINE-321. Add
JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on issue #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#issuecomment-569597979
It LGTM. +1. Thanks for your contribution. @jiwq
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] asfgit closed pull request #135: SUBMARINE-321. Add
JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361887610
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFJob.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+import org.apache.submarine.server.submitter.k8s.model.MLJob;
+
+/**
+ * It's the tf-operator's entry model.
+ */
+public class TFJob extends MLJob {
Review comment:
Again, how about "K8sTFJob"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361887714
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFJobSpec.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+
+import java.util.Map;
+
+/**
+ * The entry spec of TFJob. It contains some {@link TFReplicaSpec} which describe the info about
+ * the job task.
+ */
+public class TFJobSpec {
Review comment:
How about "K8sTFJobSpec"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361897549
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobSpec.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+import java.util.Map;
+
+/**
+ * The submarine job spec for submarine job server. It consist of name, JobLibrarySpec,
+ * JobSubmitterSpec, the tasks.
+ */
+public class JobSpec {
+ /**
+ * The user-specified job name. Such as: mnist
+ */
+ private String name;
+
+ /**
+ * The user-specified ML framework spec.
+ */
+ private JobLibrarySpec librarySpec;
+
+ /**
+ * The user-specified submitter spec.
+ */
+ private JobSubmitterSpec submitterSpec;
+
+ /**
+ * The tasks of the job.
+ * Such as:
+ * TensorFlow: Chief, Ps, Worker, Evaluator
+ * PyTorch: Master, Worker
+ */
+ private Map<String, JobTaskSpec> taskSpecs;
Review comment:
@tangzhankun The map's key is `Master` or `Worker` etc, I will add the more detailed comment for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361888563
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFReplicaSpec.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+
+/**
+ * The replica spec for TFJob. It contains replicas, restart policy and pod template.
+ * The template describe the running instance for task.
+ */
+public class TFReplicaSpec {
Review comment:
This spec seems cannot cover the distributed TF job spec. Check here: https://github.com/kubeflow/tf-operator/blob/master/examples/v1/dist-mnist/tf_job_mnist.yaml
The Spec should have the ability to describe both "worker" and "ps". And each "worker" or "ps" has the same pod spec.
I prefer to refer to the "Component.java" to get an idea and re-design this. Here's my idea:
"K8sTFReplicaSpec" could contain two "component" members(workerComponent and psComponent).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361890676
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sJobSubmitter.java
##########
@@ -29,29 +34,67 @@
import io.kubernetes.client.models.V1DeleteOptionsBuilder;
import io.kubernetes.client.util.ClientBuilder;
import io.kubernetes.client.util.KubeConfig;
+import org.apache.submarine.server.api.JobSubmitter;
+import org.apache.submarine.server.api.exception.UnsupportedJobTypeException;
+import org.apache.submarine.server.api.job.Job;
+import org.apache.submarine.server.api.spec.JobSpec;
import org.apache.submarine.server.submitter.k8s.model.CustomResourceJob;
import org.apache.submarine.server.submitter.k8s.model.CustomResourceJobList;
+import org.apache.submarine.server.submitter.k8s.model.MLJob;
+import org.apache.submarine.server.submitter.k8s.parser.JobSpecParser;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.FileReader;
-import java.io.IOException;
-
/**
* JobSubmitter for Kubernetes Cluster.
*/
-// TODO(jiwq): It should implement the JobSubmitter interface
-public class K8sJobSubmitter {
+public class K8sJobSubmitter implements JobSubmitter {
private final Logger LOG = LoggerFactory.getLogger(K8sJobSubmitter.class);
+ /**
+ * Key: kind of CRD, such as TFJob/PyTorchJob
+ * Value: the CRD api with version
+ */
+ private Map<String, String> supportedCRDMap;
+
public K8sJobSubmitter(String confPath) throws IOException {
ApiClient client =
ClientBuilder.kubeconfig(KubeConfig.loadKubeConfig(new FileReader(confPath))).build();
Configuration.setDefaultApiClient(client);
}
- public String submitJob(K8sJobRequest request) {
- return "job_id";
+ @Override
+ public void initialize() {
+ supportedCRDMap = new HashMap<>();
+ supportedCRDMap.put("TFJob", "kubeflow.org/v1");
+ }
+
+ @Override
+ public String getSubmitterType() {
+ return "k8s";
+ }
+
+ @Override
+ public Job submitJob(JobSpec jobSpec) throws UnsupportedJobTypeException {
+ if (!supportedCRDMap.containsKey(jobSpec.getSubmitterSpec().getType())) {
+ throw new UnsupportedJobTypeException();
+ }
+
+ Job job = new Job();
Review comment:
Why not using JobManager to submit this job? If I understand correctly, the job manager is the job submittion entry?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361890375
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobSpec.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+import java.util.Map;
+
+/**
+ * The submarine job spec for submarine job server. It consist of name, JobLibrarySpec,
+ * JobSubmitterSpec, the tasks.
+ */
+public class JobSpec {
+ /**
+ * The user-specified job name. Such as: mnist
+ */
+ private String name;
+
+ /**
+ * The user-specified ML framework spec.
+ */
+ private JobLibrarySpec librarySpec;
+
+ /**
+ * The user-specified submitter spec.
+ */
+ private JobSubmitterSpec submitterSpec;
+
+ /**
+ * The tasks of the job.
+ * Such as:
+ * TensorFlow: Chief, Ps, Worker, Evaluator
+ * PyTorch: Master, Worker
+ */
+ private Map<String, JobTaskSpec> taskSpecs;
Review comment:
I'm confusing here. What's the key and value of the taskSpecs? What's the key difference between JobTaskSpec and the deleted "component.java"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361890104
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobSubmitterSpec.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+/**
+ * The spec of the specified JobSubmitter
+ */
+public class JobSubmitterSpec {
+ /**
+ * The type of JobSubmitter which will be selected to submit job. Such as: yarn/yarnservice/k8s
+ */
+ private String type;
Review comment:
How about we use an Enum type here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361897723
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobSubmitterSpec.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+/**
+ * The spec of the specified JobSubmitter
+ */
+public class JobSubmitterSpec {
+ /**
+ * The type of JobSubmitter which will be selected to submit job. Such as: yarn/yarnservice/k8s
+ */
+ private String type;
Review comment:
@tangzhankun Due to the submitter implements is split with server, so the `Enum` is not suitable for this situation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361904344
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFReplicaSpec.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+
+/**
+ * The replica spec for TFJob. It contains replicas, restart policy and pod template.
+ * The template describe the running instance for task.
+ */
+public class TFReplicaSpec {
Review comment:
After offline sync with Wanqiang. I understand that the "name" field in JobTaskSpec will indicate the "ps" or "worker"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361885576
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/MLJob.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1ObjectMeta;
+
+/**
+ * The abstract machine learning job for the CRD job
+ */
+public abstract class MLJob {
Review comment:
Can we rename it to "K8sMLJob"? The "MLJob" is vague to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361904479
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobTaskSpec.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The task spec for job, a job consists of tasks.
+ */
+public class JobTaskSpec {
+ /**
+ * The job task name, the range is [Chief, Ps, Worker, Evaluator, Master]
+ */
+ private String name;
Review comment:
@jiwq How about we make this an Enum type?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361897549
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobSpec.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+import java.util.Map;
+
+/**
+ * The submarine job spec for submarine job server. It consist of name, JobLibrarySpec,
+ * JobSubmitterSpec, the tasks.
+ */
+public class JobSpec {
+ /**
+ * The user-specified job name. Such as: mnist
+ */
+ private String name;
+
+ /**
+ * The user-specified ML framework spec.
+ */
+ private JobLibrarySpec librarySpec;
+
+ /**
+ * The user-specified submitter spec.
+ */
+ private JobSubmitterSpec submitterSpec;
+
+ /**
+ * The tasks of the job.
+ * Such as:
+ * TensorFlow: Chief, Ps, Worker, Evaluator
+ * PyTorch: Master, Worker
+ */
+ private Map<String, JobTaskSpec> taskSpecs;
Review comment:
@tangzhankun The map's key is `Master` or `Worker` etc, I will add the more detailed comment for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361885576
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/MLJob.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1ObjectMeta;
+
+/**
+ * The abstract machine learning job for the CRD job
+ */
+public abstract class MLJob {
Review comment:
Can we rename it to "K8sMLJob"? The "MLJob" is vague to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361907361
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobTaskSpec.java
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The task spec for job, a job consists of tasks.
+ */
+public class JobTaskSpec {
+ /**
+ * The job task name, the range is [Chief, Ps, Worker, Evaluator, Master]
+ */
+ private String name;
Review comment:
never mind, let's skip this for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361893371
##########
File path: submarine-commons/commons-utils/src/main/java/org/apache/submarine/commons/utils/SubmarineConfiguration.java
##########
@@ -484,7 +523,10 @@ public void updateConfiguration(String name, String value) {
"workbench.websocket.max.text.message.size", "1024000"),
WORKBENCH_WEB_WAR("workbench.web.war", "submarine-workbench/workbench-web/dist"),
SUBMARINE_RUNTIME_CLASS("submarine.runtime.class",
- "org.apache.submarine.server.submitter.yarn.YarnRuntimeFactory");
+ "org.apache.submarine.server.submitter.yarn.YarnRuntimeFactory"),
+ SUBMARINE_SUBMITTERS("submarine.submitters", ""),
+ SUBMARINE_SUBMITTERS_ENTRY("submarine.submitters.%s.class", ""),
Review comment:
Is it better to use
SUBMARINE_SUBMITTERS_ENTRY("submarine.submitters.%s.entry", "")
or
SUBMARINE_SUBMITTERS_CLASS("submarine.submitters.%s.class", "") ?
"entry" and "class" is not match
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361898128
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFJobSpec.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+
+import java.util.Map;
+
+/**
+ * The entry spec of TFJob. It contains some {@link TFReplicaSpec} which describe the info about
+ * the job task.
+ */
+public class TFJobSpec {
Review comment:
I think it's okay, because it's owned by this package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361887714
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFJobSpec.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+
+import java.util.Map;
+
+/**
+ * The entry spec of TFJob. It contains some {@link TFReplicaSpec} which describe the info about
+ * the job task.
+ */
+public class TFJobSpec {
Review comment:
How about "K8sTFJobSpec"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361890104
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobSubmitterSpec.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+/**
+ * The spec of the specified JobSubmitter
+ */
+public class JobSubmitterSpec {
+ /**
+ * The type of JobSubmitter which will be selected to submit job. Such as: yarn/yarnservice/k8s
+ */
+ private String type;
Review comment:
How about we use an Enum type here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361904526
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFReplicaSpec.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+
+/**
+ * The replica spec for TFJob. It contains replicas, restart policy and pod template.
+ * The template describe the running instance for task.
+ */
+public class TFReplicaSpec {
Review comment:
Deleting this comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361897723
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/JobSubmitterSpec.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec;
+
+/**
+ * The spec of the specified JobSubmitter
+ */
+public class JobSubmitterSpec {
+ /**
+ * The type of JobSubmitter which will be selected to submit job. Such as: yarn/yarnservice/k8s
+ */
+ private String type;
Review comment:
@tangzhankun Due to the submitter implements is split with server, so the `Enum` is not suitable for this situation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361898349
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFReplicaSpec.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+import io.kubernetes.client.models.V1PodTemplateSpec;
+
+/**
+ * The replica spec for TFJob. It contains replicas, restart policy and pod template.
+ * The template describe the running instance for task.
+ */
+public class TFReplicaSpec {
Review comment:
@tangzhankun More info see `TFJobTest.java` please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361887610
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFJob.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+import org.apache.submarine.server.submitter.k8s.model.MLJob;
+
+/**
+ * It's the tf-operator's entry model.
+ */
+public class TFJob extends MLJob {
Review comment:
Again, how about "K8sTFJob"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361890676
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sJobSubmitter.java
##########
@@ -29,29 +34,67 @@
import io.kubernetes.client.models.V1DeleteOptionsBuilder;
import io.kubernetes.client.util.ClientBuilder;
import io.kubernetes.client.util.KubeConfig;
+import org.apache.submarine.server.api.JobSubmitter;
+import org.apache.submarine.server.api.exception.UnsupportedJobTypeException;
+import org.apache.submarine.server.api.job.Job;
+import org.apache.submarine.server.api.spec.JobSpec;
import org.apache.submarine.server.submitter.k8s.model.CustomResourceJob;
import org.apache.submarine.server.submitter.k8s.model.CustomResourceJobList;
+import org.apache.submarine.server.submitter.k8s.model.MLJob;
+import org.apache.submarine.server.submitter.k8s.parser.JobSpecParser;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.FileReader;
-import java.io.IOException;
-
/**
* JobSubmitter for Kubernetes Cluster.
*/
-// TODO(jiwq): It should implement the JobSubmitter interface
-public class K8sJobSubmitter {
+public class K8sJobSubmitter implements JobSubmitter {
private final Logger LOG = LoggerFactory.getLogger(K8sJobSubmitter.class);
+ /**
+ * Key: kind of CRD, such as TFJob/PyTorchJob
+ * Value: the CRD api with version
+ */
+ private Map<String, String> supportedCRDMap;
+
public K8sJobSubmitter(String confPath) throws IOException {
ApiClient client =
ClientBuilder.kubeconfig(KubeConfig.loadKubeConfig(new FileReader(confPath))).build();
Configuration.setDefaultApiClient(client);
}
- public String submitJob(K8sJobRequest request) {
- return "job_id";
+ @Override
+ public void initialize() {
+ supportedCRDMap = new HashMap<>();
+ supportedCRDMap.put("TFJob", "kubeflow.org/v1");
+ }
+
+ @Override
+ public String getSubmitterType() {
+ return "k8s";
+ }
+
+ @Override
+ public Job submitJob(JobSpec jobSpec) throws UnsupportedJobTypeException {
+ if (!supportedCRDMap.containsKey(jobSpec.getSubmitterSpec().getType())) {
+ throw new UnsupportedJobTypeException();
+ }
+
+ Job job = new Job();
Review comment:
Why not using JobManager to submit this job? If I understand correctly, the job manager is the job submittion entry?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] yuanzac commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
yuanzac commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361893655
##########
File path: submarine-server/server-api/pom.xml
##########
@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>submarine-server</artifactId>
+ <groupId>org.apache.submarine</groupId>
+ <version>0.3.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>server-api</artifactId>
+ <name>Submarine: Server API</name>
+
+</project>
Review comment:
Add a new line at end of file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #135:
SUBMARINE-321. Add JobManager and SubmitterManager components
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #135: SUBMARINE-321. Add JobManager and SubmitterManager components
URL: https://github.com/apache/submarine/pull/135#discussion_r361898128
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/model/tfjob/TFJobSpec.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.model.tfjob;
+
+import com.google.gson.annotations.SerializedName;
+
+import java.util.Map;
+
+/**
+ * The entry spec of TFJob. It contains some {@link TFReplicaSpec} which describe the info about
+ * the job task.
+ */
+public class TFJobSpec {
Review comment:
I think it's okay, because it's owned by this package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org