You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/05/07 11:36:08 UTC
[GitHub] [dolphinscheduler] EricGao888 opened a new issue, #9942: [Feature][Task Plugin] Enable environment switch for python task plugin
EricGao888 opened a new issue, #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
* Currently we cannot alter python virtual environment for python task plugin once ds starts. However, in practice, sometimes we need different python tasks to run in different virtual env.
* We might add a field `env` or something like this so that users could choose the specific virtual environments for their python tasks. To implement this, we could execute something like `conda activate xxx` before executing the python script in python task plugin.
### Use case
* Already described above.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #9942: [Feature][Task Plugin] Enable environment switch for python task plugin
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942#issuecomment-1120193559
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #9942: [Feature][Task Plugin] Enable environment switch for python task plugin
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942#issuecomment-1216190633
https://github.com/apache/dolphinscheduler/blob/63dfa45474752bfd514f1c8d4503e038b3b14e7c/dolphinscheduler-task-plugin/dolphinscheduler-task-pytorch/src/main/java/org/apache/dolphinscheduler/plugin/task/pytorch/PythonEnvManager.java#L1-L82
I write a PythonEnvManager in #11498, include the method below
- getBuildEnvCommand: build a python environment
- getSourceEnvCommand: after we build a environment, we can get source command to change python environment
- getPythonCommand: after we build a environment, we can get python command directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #9942: [Feature][Task Plugin] Abstract python related tasks
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942#issuecomment-1216202039
> * If on a WORKER node I want to run two python nodes at the same time, and the two nodes require different python versions. So the source is not enough, is it necessary to write the absolute path to meet this requirement?
> * Whether to alias the environment variable and configure it in the corresponding profile, because the same environment variable of each WORKER node may be in different paths. For example, the python2 of worker A is under a/xxx/python2, and the python2 of worker B is under b/xx/cc/python2。we can config it in every node's dolpscheduler.env file
1. In `Jupyter Task Plugin`, we will source `conda.sh`, which is configurable, each time before running a task. Therefore, you could use different python virtual environments for different python task nodes even on the same worker, they will not affect one another.
2. After sourcing `conda.sh`, we use `conda activate xxx` to activate the specific environment and execute python script with the python interpreter in that environment, therefore, users do not need to worry about the python path.
https://github.com/apache/dolphinscheduler/blob/2a7e0bee917c0a75280e848ea40e5492a3302c76/dolphinscheduler-task-plugin/dolphinscheduler-task-jupyter/src/test/java/org/apache/dolphinscheduler/plugin/task/jupyter/JupyterTaskTest.java#L112-L131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] fuchanghai commented on issue #9942: [Feature][Task Plugin] Abstract python related tasks
Posted by GitBox <gi...@apache.org>.
fuchanghai commented on issue #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942#issuecomment-1216194581
- If on a WORKER node I want to run two python nodes at the same time, and the two nodes require different python versions. So the source is not enough, is it necessary to write the absolute path to meet this requirement?
- Whether to alias the environment variable and configure it in the corresponding profile, because the same environment variable of each WORKER node may be in different paths. For example, the python2 of worker A is under a/xxx/python2, and the python2 of worker B is under b/xx/cc/python2。we can config it in every node's dolpscheduler.env file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #9942: [Feature][Task Plugin] Enable environment switch for python task plugin
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942#issuecomment-1175885210
Currently, we have supported several python dependency management approaches in `Jupyter Task`, which is a specific type of python task. Some related discussions: https://github.com/apache/dolphinscheduler/pull/10658#issuecomment-1172014440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #9942: [Feature][Task Plugin] Enable environment switch for python task plugin
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #9942:
URL: https://github.com/apache/dolphinscheduler/issues/9942#issuecomment-1175893413
There are some points to be discussed here:
- We could abstract python type task, other specific python related task could inherit it, together with its several dependency management approaches.
- Which one to use? Choose `Anaconda` or `venv` to manage virtual python environment. Or maybe we could provide both of them for users to choose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org