[GitHub] [dolphinscheduler] EricGao888 commented on a diff in pull request #10337: [Feature][Task Plugin]Enable users to switch and install conda env in jupyter task

[GitBox <gi...@apache.org>]

EricGao888 commented on code in PR #10337:
URL: https://github.com/apache/dolphinscheduler/pull/10337#discussion_r893242034


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-jupyter/src/main/java/org/apache/dolphinscheduler/plugin/task/jupyter/JupyterConstants.java:
##########
@@ -33,6 +33,18 @@ private JupyterConstants() {
      */
     public static final String CONDA_ACTIVATE = "conda activate";
 
+    /**
+     * create and activate conda env from tar
+     */
+    public static final String CREATE_ENV_FROM_TAR = "mkdir jupyter_env && " +
+            "tar -xzf %s -C jupyter_env && " +
+            "source jupyter_env/bin/activate";

Review Comment:
   Currently, `env tar` will be downloaded from `resource center` before task execution and removed when task execution completed. https://github.com/apache/dolphinscheduler/blob/f90f0f8490d8f4cd87d2368cd0264a4837918c7d/dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java#L141-L145
   https://github.com/apache/dolphinscheduler/blob/f90f0f8490d8f4cd87d2368cd0264a4837918c7d/dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java#L199-L203
   
   However, I think it is a good idea to give a choice to define the path and install the env on workers without getting removed after task execution. In this way, the same env will not need to be downloaded every time.
   
   WDYT @SbloodyS @zhongjiajie 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org