[GitHub] [submarine] cdmikechen commented on a diff in pull request #989: SUBMARINE-1283. copy data for experiment before it running via distcp to minio

[GitBox <gi...@apache.org>]

cdmikechen commented on code in PR #989:
URL: https://github.com/apache/submarine/pull/989#discussion_r973002910


##########
.github/scripts/build-image-locally-v3.sh:
##########
@@ -17,12 +17,14 @@
 #
 
 SUBMARINE_VERSION="0.8.0-SNAPSHOT"

Review Comment:
   I am consulting on an unrelated matter:
   Can we get our version information directly from the `pom.xml` so that we don't need to do a lot of text replacement and version number validation each time we make a change to the version? For example
   ```shell
   awk '/<\/*version\/*>/{gsub(/[[:space:]]*<\/*version\/*>/,"");print $0}' pom.xml | head -n 1
   ```
   
   cc @pingsutw @xunliu 



##########
submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java:
##########
@@ -63,6 +64,14 @@ public CodeSpec getCode() {
   public void setCode(CodeSpec code) {
     this.code = code;
   }
+  
+  public Map<String, String> getExperimentHandlerSpec() {
+    return experimentHandlerSpec;
+  }
+
+  public void setExperimentHandlerSpec(Map<String, String> experimentHandlerSpec) {
+    this.experimentHandlerSpec = experimentHandlerSpec;
+  }

Review Comment:
   I think we could add some more test cases to the `submarine-server/server-submitter/submitter-k8s/src/test/java/org/apache/submarine/server/submitter/k8s/ExperimentSpecParserTest.java` to pass the codecov checks.
   



##########
submarine-experiment-prehandler/fs_prehandler/hdfs_prehandler.py:
##########
@@ -15,26 +15,43 @@
 

Review Comment:
   I tried to build the docker image and found that roughly this image has 2.8G in local. 
   I don't know if this will affect the overall user experience, after all, if the image is too large, it will take too much time to pull it. 
   Or is it possible to reduce it to a simple java code, so that python and the whole hadoop will not be installed , and the image will contain only the jars needed inside.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@submarine.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org