You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@submarine.apache.org by GitBox <gi...@apache.org> on 2021/06/29 12:56:01 UTC
[GitHub] [submarine] kevin85421 opened a new pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
kevin85421 opened a new pull request #622:
URL: https://github.com/apache/submarine/pull/622
### What is this PR for?
We cannot run two experiments with the same name. Take experimentIT.java as an example, the frontend E2E testcase will create an experiment named "experiment-e2e-test". If we run this testcase twice, the workbench just shows an "experiment-e2e-test" rather than two.
The root cause is that the K8sSubmitter will create two PODs for an "experiment-e2e-test". The names of these two PODs are "experiment-e2e-test-ps-0" and "experiment-e2e-test-worker-0". Hence, when we try to create the second "experiment-e2e-test", K8sSubmitter will throw an exception due to duplicate POD names.
To reproduce the bug:
```bash
# Step1: Run workbench on port 8080
# Step2:
cd submarine-cloud-v2
# Step3: Create "experiment-e2e-test" twice
./hack/run_frontend_e2e.sh experimentIT
./hack/run_frontend_e2e.sh experimentIT
# Step4: Check Workbench: only an "experiment-e2e-test" exists => BUG!
```
* My solution: I append `${experimentCounter}` ([ExperimentManager.java#L58](https://github.com/apache/submarine/blob/f53e3564a270e57690edf0e58a8137a78d9eca3e/submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java#L58)) to the experiment name. With the solution, the first "experiment-e2e-test" will create two PODs:
```
experiment-e2e-test-0001-ps-0
experiment-e2e-test-0001-worker-0
```
### What type of PR is it?
[Bug Fix]
### Todos
* Check "_" (Reference: [Link](https://blog.csdn.net/airangrong6572/article/details/101273245))
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-880
### How should this be tested?
```bash
# Step1: Run workbench on port 8080
# Step2:
cd submarine-cloud-v2
# Step3: Create "experiment-e2e-test" twice
./hack/run_frontend_e2e.sh experimentIT
./hack/run_frontend_e2e.sh experimentIT
# Step4: Check Workbench: two"experiment-e2e-test" exists (Correct)
```
### Screenshots (if appropriate)
<img width="1440" alt="ζͺε 2021-06-29 δΈε8 24 49" src="https://user-images.githubusercontent.com/20109646/123800896-5434f300-d91c-11eb-8991-d4740c4287ce.png">
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #622:
URL: https://github.com/apache/submarine/pull/622#discussion_r668387580
##########
File path: submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
##########
@@ -108,8 +108,10 @@ public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeE
spec.getMeta().getEnvVars().put(RestConstants.SUBMARINE_TRACKING_URI, url);
spec.getMeta().getEnvVars().put(RestConstants.LOG_DIR_KEY, RestConstants.LOG_DIR_VALUE);
- String lowerName = spec.getMeta().getName().toLowerCase();
+ String lowerName = spec.getMeta().getName().toLowerCase();
spec.getMeta().setName(lowerName);
+ spec.getMeta().setExpID(id.toString().replaceAll("_", "-"));
Review comment:
Yes. But the label value could contain dashes (-), underscores (_), dots (.), and alphanumerics between.
I think we should keep it, so that we can't convert the concept when use label selector. Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-877811592
@jiwq I am sorry to bother you. Do you have any other recommendations about this patch? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] ByronHsu edited a comment on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
ByronHsu edited a comment on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-870717365
I agree with @kevin85421. Changing the job name each time for the same task is quite annoying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-876272702
@jiwq I have fixed it. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-871534450
@jiwq Thank you for your recommendations!
I have some questions:
Q1: Does "job name" mean "experiment name"?
Q2: Does "pod label" mean "pod name"?
The following points assumed that the answers to these two questions are YES. In my opinion, with this patch, users can still retrieve their experiments via experiment names easily because the experiment (with the modified name) will be returned. Hence, people can get the experiment easily from the workbench.
In addition, because the rename rule is appending ${experimentCounter} to the original experiment name, users can still retrieve their experiment with the original experiment name easily.
For example, a user creates an experiment "mnist-exp".
Experiment name: "mnist-exp-0001" (same as the experiment name on the workbench)
POD name: "mnist-exp-0001"
When a user wants to retrieve the experiment, they can use both "mnist-exp-0001" or "mnist-exp".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #622:
URL: https://github.com/apache/submarine/pull/622#discussion_r668397755
##########
File path: submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
##########
@@ -108,8 +108,10 @@ public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeE
spec.getMeta().getEnvVars().put(RestConstants.SUBMARINE_TRACKING_URI, url);
spec.getMeta().getEnvVars().put(RestConstants.LOG_DIR_KEY, RestConstants.LOG_DIR_VALUE);
- String lowerName = spec.getMeta().getName().toLowerCase();
+ String lowerName = spec.getMeta().getName().toLowerCase();
spec.getMeta().setName(lowerName);
+ spec.getMeta().setExpID(id.toString().replaceAll("_", "-"));
Review comment:
+1 After discussed offline. Thanks @kevin85421 for the future design, I expected the refactor of experiment feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] ByronHsu edited a comment on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
ByronHsu edited a comment on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-870717365
I agree with @kevin85421. Trying the job name each time for the same task is quite annoying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on a change in pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on a change in pull request #622:
URL: https://github.com/apache/submarine/pull/622#discussion_r666356084
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java
##########
@@ -51,6 +55,22 @@ public void setName(String name) {
this.name = name;
}
+ /**
+ * Get the experiment id which is unique within a namespace.
+ * @return experiment id
+ */
+ public String getExpID() {
Review comment:
I have fixed it.
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java
##########
@@ -51,6 +55,22 @@ public void setName(String name) {
this.name = name;
}
+ /**
+ * Get the experiment id which is unique within a namespace.
+ * @return experiment id
+ */
+ public String getExpID() {
+ return experimentId;
+ }
+
+ /**
+ * experiment id must be unique within a namespace. Is required when creating experiment.
+ * @param experimentId experiment id
+ */
+ public void setExpID(String experimentId) {
Review comment:
I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on a change in pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq commented on a change in pull request #622:
URL: https://github.com/apache/submarine/pull/622#discussion_r666287760
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java
##########
@@ -51,6 +55,22 @@ public void setName(String name) {
this.name = name;
}
+ /**
+ * Get the experiment id which is unique within a namespace.
+ * @return experiment id
+ */
+ public String getExpID() {
+ return experimentId;
+ }
+
+ /**
+ * experiment id must be unique within a namespace. Is required when creating experiment.
+ * @param experimentId experiment id
+ */
+ public void setExpID(String experimentId) {
Review comment:
```suggestion
public void setExperimentId(String experimentId) {
```
##########
File path: submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentMeta.java
##########
@@ -51,6 +55,22 @@ public void setName(String name) {
this.name = name;
}
+ /**
+ * Get the experiment id which is unique within a namespace.
+ * @return experiment id
+ */
+ public String getExpID() {
Review comment:
Disallow to abridge the name, because it will reduce readability.
```suggestion
public String getExperimentId() {
```
##########
File path: submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
##########
@@ -108,8 +108,10 @@ public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeE
spec.getMeta().getEnvVars().put(RestConstants.SUBMARINE_TRACKING_URI, url);
spec.getMeta().getEnvVars().put(RestConstants.LOG_DIR_KEY, RestConstants.LOG_DIR_VALUE);
- String lowerName = spec.getMeta().getName().toLowerCase();
+ String lowerName = spec.getMeta().getName().toLowerCase();
spec.getMeta().setName(lowerName);
+ spec.getMeta().setExpID(id.toString().replaceAll("_", "-"));
Review comment:
What's the reason of considering '-' rather than '_'?
Supported character set of K8s: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-880643852
Thank @jiwq for your recommendations! I will propose a new design to enable experiments to be grouped by tags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq edited a comment on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq edited a comment on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-871934404
@kevin85421
> Q1: Does "job name" mean "experiment name"?
Yes, it is experiment spec name field.
Because we defined the name in experiment spec, so I think we should keep this concept and user should not to know the real pod name.
> Q2: Does "pod label" mean "pod name"?
NO. More info see https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
By example, user defined the mnist tensorflow experiment spec in submarine and submit twice, we can found mnist-1 and mnist-2 jobs. If user want to query the minist experiment, he/her only should use the mnist as the query param to retrieve the list of experiments. And if want get more info should use the specific name we can call it instance name, which is mnist-1 or mnist-2 etc.
Simplify, I think we should two tasks to do.
1. Fill the K8s object labels use experiment name field.
2. Append the counter after the job(tfjob/pytorch job)'s name which is defined in experiment spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on a change in pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on a change in pull request #622:
URL: https://github.com/apache/submarine/pull/622#discussion_r666354267
##########
File path: submarine-server/server-core/src/main/java/org/apache/submarine/server/experiment/ExperimentManager.java
##########
@@ -108,8 +108,10 @@ public Experiment createExperiment(ExperimentSpec spec) throws SubmarineRuntimeE
spec.getMeta().getEnvVars().put(RestConstants.SUBMARINE_TRACKING_URI, url);
spec.getMeta().getEnvVars().put(RestConstants.LOG_DIR_KEY, RestConstants.LOG_DIR_VALUE);
- String lowerName = spec.getMeta().getName().toLowerCase();
+ String lowerName = spec.getMeta().getName().toLowerCase();
spec.getMeta().setName(lowerName);
+ spec.getMeta().setExpID(id.toString().replaceAll("_", "-"));
Review comment:
#630 describes that why I replace "_" with "-".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] asfgit closed pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #622:
URL: https://github.com/apache/submarine/pull/622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-872220249
@kevin85421 Perfect :) I'm not sure whether the workbench supports the feature. If not we should develop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-870701150
@jiwq Thank you for your recommendation! In my opinion, data scientists need to run multiple experiments to achieve a single task.
For example, a data scientist wants to train a model with the MNIST dataset. At first, he creates an experiment "mnist-exp" with his `minist-ver1.py`. Next, because the model accuracy does not satisfy the scientist's expectation, the data scientist makes some modifications to the Python script (`mnist-ver1.py`) and creates an experiment "mnist-exp" again. The bug will occur seamlessly (That is, the data scientist does not know his experiment will not be created).
To avoid the condition, the data scientist needs to:
* Method1:
* Delete the first "mnist-exp"
* Create "mnist-exp" with the updated `mnist-ver1.py`.
* Cons:
* (1) Lose the data about the first "mnist-exp"
* (2) Delete "mnist-exp" manually
* Method2:
* Change the name of the experiment (ex: "mnist-exp-2")
* Cons: Data scientists usually need to run many experiments to achieve a single task. Thus, the repeat human effort is very annoying. (For example, "mnist-exp-2", "mnist-exp-3", "mnist-exp-4" ... "mnist-exp-100")
With my personal user experience, I usually use a script to help me create an experiment without entering the information, including experiment name, number of workers, and resource limitations, again and again. However, because of this bug, when I want to run my script to create an experiment again, I need a lot of human effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-873633274
@jiwq I have updated this PR. You can check the sections: "My solution" and "Screenshots".
To elaborate,
an user creates an experiment: ${original_exp_name} = "experiment-e2e-test"
The experiment name on the workbench: "experiment-e2e-test" => conclusion (1)
Names of PODs: "experiment-e2e-test-0001" => conclusion (2)
Name of CR: "experiment-e2e-test-0001"
Labels of CR: "experiment-e2e-test" => conclusion (4)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-874858903
@kevin85421 How about use the experiment id as the pod name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-871934404
> Q1: Does "job name" mean "experiment name"?
Yes, it is experiment spec name field.
Because we defined the name in experiment spec, so I think we should keep this concept and user should not to know the real pod name.
> Q2: Does "pod label" mean "pod name"?
NO. More info see https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
By example, user defined the mnist tensorflow experiment spec in submarine and submit twice, we can found mnist-1 and mnist-2 jobs. If user want to query the minist experiment, he/her only should use the mnist as the query param to retrieve the list of experiments. And if want get more info should use the specific name we can call it instance name, which is mnist-1 or mnist-2 etc.
Simplify, I think we should two tasks to do.
1. Fill the K8s object labels use experiment name field.
2. Append the counter after the job(tfjob/pytorch job)'s name which is defined in experiment spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] jiwq commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
jiwq commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-871496869
Thanks @kevin85421 for a detailed explanation. I agree, but I think we should set the job name as the pod label. So that the user can retrieve his/her experiments easily by name.
In this patch we should set the pod label. I accept add the new api later to retrieve the experiments by the user's experiment name, and we should fill a JIRA to track this feature. Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] kevin85421 commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
kevin85421 commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-872000440
@jiwq Got it. It makes sense.
To conclude,
(1) The experiment name on the workbench must be the same as the name specified by users.
(2) The names of the PODs (i.e. instance name) related to the experiment are `${original_exp_name} + "-" + ${experimentCounter}`
(3) Users can use ${original_exp_name} to retrieve their experiments.
(4) Set the labels of the PODs to ${original_exp_name}
For instance,
User wants to create an experiment: ${original_exp_name} = "mnist-exp"
The experiment name on the workbench: "mnist-exp" => conclusion (1)
Names of PODs: "mnist-exp-0001" => conclusion (2)
Labels of PODs: "mnist-exp" => conclusion (4)
Do I misunderstand anything? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [submarine] ByronHsu commented on pull request #622: SUBMARINE-880. Cannot run two experiments with the same name
Posted by GitBox <gi...@apache.org>.
ByronHsu commented on pull request #622:
URL: https://github.com/apache/submarine/pull/622#issuecomment-870717365
I agree with @kevin85421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org