You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@submarine.apache.org by GitBox <gi...@apache.org> on 2020/07/26 17:03:31 UTC
[GitHub] [submarine] manirajv06 opened a new pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
manirajv06 opened a new pull request #362:
URL: https://github.com/apache/submarine/pull/362
### What is this PR for?
https://issues.apache.org/jira/browse/SUBMARINE-570
### What type of PR is it?
Feature
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-570
### How should this be tested?
* First time? Setup Travis CI as described on https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? Yes/No
* Is there breaking changes for older versions? Yes/No
* Does this needs documentation? Yes/No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r464096843
##########
File path: docs/userdocs/k8s/api/experiment.md
##########
@@ -167,6 +167,79 @@ Above example assume environment "my-submarine-env" already exists in Submarine.
}
```
+## Create Experiment (Using Pre-defined/Stored Environment)
Review comment:
This change shouldn't be part of this PR. It is part of https://github.com/apache/submarine/pull/348. Will check again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] liuxunorg commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r461312054
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/CodeSpec.java
##########
@@ -0,0 +1,24 @@
+package org.apache.submarine.server.api.spec;
Review comment:
Please add ASF License head.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465789700
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
##########
@@ -168,8 +171,35 @@ private static V1PodTemplateSpec parseTemplateSpec(
container.setResources(resources);
container.setEnv(parseEnvVars(taskSpec, experimentSpec.getMeta().getEnvVars()));
containers.add(container);
- podSpec.setContainers(containers);
+ /**
+ * Init Git localize Container
+ */
+ if (experimentSpec.getCode() != null) {
+ CodeLocalizer localizer = AbstractCodeLocalizer.getCodeLocalizer(
+ experimentSpec.getCode().getSyncMode(),
+ experimentSpec.getCode().getUrl());
+ localizer.localize(podSpec);
+
+ if (podSpec.getInitContainers() != null
+ && podSpec.getInitContainers().size() > 0) {
+ String volumeName = podSpec.getInitContainers().get(0).getVolumeMounts()
Review comment:
(Yes, pod containers may mount multiple volumes in future for various reasons which can be pat of other init container process). But at any point of time, this "code localizer" init container would have only 1 volume mount. However, open to change if you think otherwise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465784457
##########
File path: docs/userdocs/k8s/api/experiment.md
##########
@@ -167,6 +167,79 @@ Above example assume environment "my-submarine-env" already exists in Submarine.
}
```
+## Create Experiment (Using Pre-defined/Stored Environment)
Review comment:
Taken care
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465786293
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
Review comment:
As of today, it seem like there are no differences. But when we get into "git ssh mode" implementation, we will need this separation. For ex, GIT_SYNC_SSH needs to be set as "true" only for git ssh mode" implementation
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
Review comment:
As of today, it seem like there are no differences. But when we get into "git ssh mode" implementation, we will need this separation. For ex, GIT_SYNC_SSH needs to be set as "true" only for "git ssh mode" implementation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on pull request #362:
URL: https://github.com/apache/submarine/pull/362#issuecomment-668054649
@wangdatan @liuxunorg Forgot to mention that code has been tested using e2e tests. Was able to see repo code has been synced to /code directory inside k8 container.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] tangzhankun commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r468302034
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public abstract class GitCodeLocalizer extends AbstractCodeLocalizer {
Review comment:
The credential seems should be created in the workbench and couple with the user account. And we may also think about the refresh of the secure token. I'd prefer to not do the secure localizer for now.
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.exception.InvalidSpecException;
+
+import io.kubernetes.client.models.V1EmptyDirVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1Volume;
+
+public abstract class AbstractCodeLocalizer implements CodeLocalizer {
+
+ public static final String CODE_LOCALIZER_PATH = "/code";
Review comment:
Do we need to set this environment variable to the other containers so that the user can set command like
"python $CODE_PATH/src/train.py"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] wangdatan commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
wangdatan commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r464788450
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
Review comment:
@manirajv06 , it looks like HTTP and SSH Git localizer pretty same, I'm wondering why we need the two implementations. And why `localize` of `SSHGitCodeLocalizer` is empty?
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
##########
@@ -168,8 +171,35 @@ private static V1PodTemplateSpec parseTemplateSpec(
container.setResources(resources);
container.setEnv(parseEnvVars(taskSpec, experimentSpec.getMeta().getEnvVars()));
containers.add(container);
- podSpec.setContainers(containers);
+ /**
+ * Init Git localize Container
+ */
+ if (experimentSpec.getCode() != null) {
+ CodeLocalizer localizer = AbstractCodeLocalizer.getCodeLocalizer(
+ experimentSpec.getCode().getSyncMode(),
+ experimentSpec.getCode().getUrl());
+ localizer.localize(podSpec);
+
+ if (podSpec.getInitContainers() != null
+ && podSpec.getInitContainers().size() > 0) {
+ String volumeName = podSpec.getInitContainers().get(0).getVolumeMounts()
Review comment:
Is this part of logic generic enough to support multiple volumes? (It is possible in the future we want to support mount other volumes to the pod, such as an NFS volume. Instead of get(0), can we get the volume by dir name?
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.exception.InvalidSpecException;
+
+import io.kubernetes.client.models.V1EmptyDirVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1Volume;
+
+public abstract class AbstractCodeLocalizer implements CodeLocalizer {
+
+ private String url;
+
+ public AbstractCodeLocalizer(String url) {
+ this.url = url;
+ }
+
+ /**
+ * @return the url
+ */
+ public String getUrl() {
+ return url;
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+ V1Volume volume = new V1Volume();
+ volume.setName("code-dir");
Review comment:
can we make `code-dir` to be a constant and can be referred in other places (like test)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] asfgit closed pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #362:
URL: https://github.com/apache/submarine/pull/362
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] tangzhankun commented on pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
tangzhankun commented on pull request #362:
URL: https://github.com/apache/submarine/pull/362#issuecomment-675195467
LGTM. Thanks @manirajv06 . Will merge it if no more comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] liuxunorg commented on pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
liuxunorg commented on pull request #362:
URL: https://github.com/apache/submarine/pull/362#issuecomment-672811038
This PR has been too long, we can merge it in first, and then improve if there are problems.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465849576
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class SSHGitCodeLocalizer extends GitCodeLocalizer {
+
+ public SSHGitCodeLocalizer(String url) {
+ super(url);
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+ }
Review comment:
@jiwq Scope of this PR/JIRA is to provide support only for http based git sync mode. Added this class now itself to get clear picture of how interfaces and its impl look like. More over, we will be adding support for ssh very soon. Will create a jira for the same. cc @wangdatan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on pull request #362:
URL: https://github.com/apache/submarine/pull/362#issuecomment-673546967
@tangzhankun I am also having similar thoughts and to take care of private repo's in separate jira. Will create a jira.
@liuxunorg Made final commit for setting env var to bring logical conclusion. It is in good shape.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465786293
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
Review comment:
As of today, it seem like there are no differences. But when we get into "git ssh mode" implementation, we will need this separation. For ex, GIT_SYNC_SSH needs to be set as "true".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465867558
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class SSHGitCodeLocalizer extends GitCodeLocalizer {
+
+ public SSHGitCodeLocalizer(String url) {
+ super(url);
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+ }
Review comment:
@jiwq Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on pull request #362:
URL: https://github.com/apache/submarine/pull/362#issuecomment-670826288
@wangdatan
Since PR has taken a good shape and to bring conclusion, writing down plans to close gaps:
1. How do we handle private git repo to sync the code?
First cut:
Can we do the following in experiment spec?
spec {
..
code: {
"mode": "git",
"url": "..",
"username": "..",
"password": ".."
}
..
}
and use the credentials to sync the code from git. It carries the risk of showing above info in pod description (kubectl describe pod ..), in addition to expecting username and password in plain text as part of experiment spec.
Next Phase:
Creating a secret in K8 and using the same for code sync is much cleaner and better solution. However, we need username/password info at least one time to create a "secret" in K8 internally. Once this is done, we can use the same secret when same repo has been used in any experiment and pass those credentials to "code localizer" init container to sync the code. This way, there is no need of passing the username/password again and again.
We can bring this change in intuitive way for Note Book users as it is Web UI driven. Where as, users coming through API's would find it bit confusing. For example, Users pass username/password as part of experiment spec in first run and don't bother to remove those creds from second run onwards. I think we can debate more on this before concluding the approach.
Thoughts?
2. How localized code would be used?
My understanding is, using commands mentioned in "cmd" of "meta" spec.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] wangdatan commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
wangdatan commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r463853959
##########
File path: docs/userdocs/k8s/api/experiment.md
##########
@@ -167,6 +167,79 @@ Above example assume environment "my-submarine-env" already exists in Submarine.
}
```
+## Create Experiment (Using Pre-defined/Stored Environment)
Review comment:
I'm not sure why we make this change, I thought it will include /code init spec, but it didn't. Do you plan to add it later?
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public abstract class GitCodeLocalizer extends AbstractCodeLocalizer {
+
+ public void localize(V1PodSpec podSpec) {
+ super.localize(podSpec);
+ }
+
+ public GitCodeLocalizer(ExperimentSpec experimentSpec) {
+ super(experimentSpec);
+ }
+
+ public static CodeLocalizer getGitCodeLocalizer(
Review comment:
Instead of get CodeInitalizer from ExperimentSpec, it's better to use URL to get the CodeLocalizer.
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1Container;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1VolumeMount;
+
+public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
+
+ public HTTPGitCodeLocalizer(ExperimentSpec experimentSpec) {
+ super(experimentSpec);
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+
+ V1Container container = new V1Container();
+
+ container.setName("git-localizer");
+ container.setImage("git-sync");
+
+ V1VolumeMount mount = new V1VolumeMount();
Review comment:
Can we (or is it possible) make the volume mount logic common to different code initializers?
And SSHGitCode.. can share this implementation, correct?
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1EmptyDirVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1Volume;
+
+public abstract class AbstractCodeLocalizer implements CodeLocalizer {
Review comment:
I think we should remove all experimentSpec from CodeLocalizer (and its implementations).
One important purpose is to allow use CodeSpec in both ExperimentSpec and Notebook. Even Notebook is not ready yet, I think the implementation can general enough to support both.
Regarding the implementation, we should have no problem to remove experimentSpec from implementation, what we only need to do is use code URL to init codeLocalizer.
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public abstract class GitCodeLocalizer extends AbstractCodeLocalizer {
Review comment:
How we handle Git pull credentials, should it part of ExperimentSpec, or part of Namespace secret.
@manirajv06, @liuxunorg , @tangzhankun: any ideas here?
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public abstract class GitCodeLocalizer extends AbstractCodeLocalizer {
+
+ public void localize(V1PodSpec podSpec) {
+ super.localize(podSpec);
+ }
+
+ public GitCodeLocalizer(ExperimentSpec experimentSpec) {
+ super(experimentSpec);
+ }
+
+ public static CodeLocalizer getGitCodeLocalizer(
+ ExperimentSpec experimentSpec) {
+
+ String url = experimentSpec.getCode().getUrl();
Review comment:
We should parse URL properly and check the protocol instead of using .contains:
https://docs.oracle.com/javase/tutorial/networking/urls/urlInfo.html
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public abstract class GitCodeLocalizer extends AbstractCodeLocalizer {
+
+ public void localize(V1PodSpec podSpec) {
+ super.localize(podSpec);
+ }
+
+ public GitCodeLocalizer(ExperimentSpec experimentSpec) {
+ super(experimentSpec);
+ }
+
+ public static CodeLocalizer getGitCodeLocalizer(
+ ExperimentSpec experimentSpec) {
+
+ String url = experimentSpec.getCode().getUrl();
Review comment:
And we should check URL for code spec is valid, if it is not valid, we should fail submission of the experiment/notebook.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465828414
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class SSHGitCodeLocalizer extends GitCodeLocalizer {
+
+ public SSHGitCodeLocalizer(String url) {
+ super(url);
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+ }
Review comment:
@manirajv06 Thanks for your great contribution. Can you mind add TODO comment in here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] wangdatan commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
wangdatan commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465854563
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
##########
@@ -168,8 +171,35 @@ private static V1PodTemplateSpec parseTemplateSpec(
container.setResources(resources);
container.setEnv(parseEnvVars(taskSpec, experimentSpec.getMeta().getEnvVars()));
containers.add(container);
- podSpec.setContainers(containers);
+ /**
+ * Init Git localize Container
+ */
+ if (experimentSpec.getCode() != null) {
+ CodeLocalizer localizer = AbstractCodeLocalizer.getCodeLocalizer(
+ experimentSpec.getCode().getSyncMode(),
+ experimentSpec.getCode().getUrl());
+ localizer.localize(podSpec);
+
+ if (podSpec.getInitContainers() != null
+ && podSpec.getInitContainers().size() > 0) {
+ String volumeName = podSpec.getInitContainers().get(0).getVolumeMounts()
Review comment:
It makes sense to me. We can revisit this once we want to support multiple volumes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on pull request #362:
URL: https://github.com/apache/submarine/pull/362#issuecomment-664014329
@wangdatan First cut implementation. Can you please take a look?
Need to do integration testing to ensure code has been synced from git repos etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465849576
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class SSHGitCodeLocalizer extends GitCodeLocalizer {
+
+ public SSHGitCodeLocalizer(String url) {
+ super(url);
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+ }
Review comment:
@jiwq Scope of this PR/JIRA is to provide support http based git sync mode. Added this class now to get clear picture of how interfaces and its impl look like. More over, we will be adding support for ssh very soon. Will create a jira for the same. cc @wangdatan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465789884
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.exception.InvalidSpecException;
+
+import io.kubernetes.client.models.V1EmptyDirVolumeSource;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1Volume;
+
+public abstract class AbstractCodeLocalizer implements CodeLocalizer {
+
+ private String url;
+
+ public AbstractCodeLocalizer(String url) {
+ this.url = url;
+ }
+
+ /**
+ * @return the url
+ */
+ public String getUrl() {
+ return url;
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+ V1Volume volume = new V1Volume();
+ volume.setName("code-dir");
Review comment:
Taken care.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] wangdatan commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
wangdatan commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r465853911
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
Review comment:
gotcha, make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r464097544
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1PodSpec;
+
+public abstract class GitCodeLocalizer extends AbstractCodeLocalizer {
Review comment:
Probably, one time activity based approach is preferable as it doesn't change for every experiment. It remains same for repo's.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] manirajv06 commented on a change in pull request #362: SUBMARINE-570. Support run experiment/notebook with synced code
Posted by GitBox <gi...@apache.org>.
manirajv06 commented on a change in pull request #362:
URL: https://github.com/apache/submarine/pull/362#discussion_r464096650
##########
File path: submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.submarine.server.api.spec.ExperimentSpec;
+
+import io.kubernetes.client.models.V1Container;
+import io.kubernetes.client.models.V1PodSpec;
+import io.kubernetes.client.models.V1VolumeMount;
+
+public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
+
+ public HTTPGitCodeLocalizer(ExperimentSpec experimentSpec) {
+ super(experimentSpec);
+ }
+
+ @Override
+ public void localize(V1PodSpec podSpec) {
+
+ V1Container container = new V1Container();
+
+ container.setName("git-localizer");
+ container.setImage("git-sync");
+
+ V1VolumeMount mount = new V1VolumeMount();
Review comment:
Ah ! Have followed the same approach for VolumeDevice but missed this. Taken care.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org