[GitHub] [dolphinscheduler] EricGao888 commented on issue #9942: [Feature][Task Plugin] Abstract python related tasks

[GitBox <gi...@apache.org>]

EricGao888 commented on issue #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942#issuecomment-1216202039

   > * If on a WORKER node I want to run two python nodes at the same time, and the two nodes require different python versions. So the source is not enough, is it necessary to write the absolute path to meet this requirement?
   > * Whether to alias the environment variable and configure it in the corresponding profile, because the same environment variable of each WORKER node may be in different paths. For example, the python2 of worker A is under a/xxx/python2, and the python2 of worker B is under b/xx/cc/python2。we can config it in every node's dolpscheduler.env file
   
   1. In `Jupyter Task Plugin`, we will source `conda.sh`, which is configurable, each time before running a task. Therefore, you could use different python virtual environments for different python task nodes even on the same worker, they will not affect one another.
   2. After sourcing `conda.sh`, we use `conda activate xxx` to activate the specific environment and execute python script with the python interpreter in that environment, therefore, users do not need to worry about the python path.
   
   
   
   https://github.com/apache/dolphinscheduler/blob/2a7e0bee917c0a75280e848ea40e5492a3302c76/dolphinscheduler-task-plugin/dolphinscheduler-task-jupyter/src/test/java/org/apache/dolphinscheduler/plugin/task/jupyter/JupyterTaskTest.java#L112-L131


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org