You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/06/23 03:36:28 UTC
[GitHub] [dolphinscheduler] wangmingzhiJohn opened a new issue #5685: [Question] Environment of Python Task is different from that of real worker container
wangmingzhiJohn opened a new issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685
My DS system is setup by Docker.
When I ran like this in a PythonTask, I found that env is different from what i got in the docker container of worker.
This is what env showed by the **Python Task.**
And this is what env showed by execute `os.system("env")` or `os.environ` **directly in the container of worker**.
I don't know why they are totally different, especially the Variable 'PATH'. I thought that docker container is exactly where python executes script. So i am confused
**Which version of DolphinScheduler:**
-[1.3.6-release]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866522541
@wangmingzhiJohn I guess the hadoop is just set up for special users, you can try to add it to your tenant user's PATH.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866496412
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] wangmingzhiJohn commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
wangmingzhiJohn commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866500610
@ruanwenjun You are right. But how can I make the ENV settings available when I use tenant to execute the python script. I can't access the hadoop because some Variable not in PATH
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866498578
@wangmingzhiJohn It is because of the tenant? It seems that you use the tenant `2` to execute the `PythonTask`. You can try to use user `2` to execute os.system("env") or os.environ directly in the container of worker, and then compare the result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] chengshiwen edited a comment on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866526851
@wangdazhong001 Under different user, the result of `env` is different. Regardless of whether it is in the docker container, the same is true in linux on the host machine. The richer environment variables directly in the container of worker occur under `roor` user. While the environment variables from the python task occur under the user whose id is 2.
Since the python script does not execute `source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, the python task cannot read the environment variables in this file.
If you want to access the environment variables from `/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, there are four solutions:
1. `open('/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh', 'r')` and parse it
2. Use `python-dotenv`
3. Use the following code:
```python
import os
import pprint
import shlex
import subprocess
command = shlex.split("env -i bash -c 'source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh && env'")
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
(key, _, value) = line.partition("=")
os.environ[key] = value
proc.communicate()
pprint.pprint(dict(os.environ))
```
4. Use shell task instead of python task
```shell
source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh
python /path/to/your_python_script.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] chengshiwen edited a comment on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866526851
@wangdazhong001 Under different user, the result of `env` is different. Regardless of whether it is in the docker container, the same thing happens in linux on the host machine. The richer environment variables directly in the container of worker occur under `roor` user. While the environment variables from the python task occur under the user whose id is 2.
Since the python script does not execute `source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, the python task cannot read the environment variables in this file.
If you want to access the environment variables from `/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, there are four solutions:
1. `open('/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh', 'r')` and parse it
2. Use `python-dotenv`
3. Use the following code:
```python
import os
import pprint
import shlex
import subprocess
command = shlex.split("env -i bash -c 'source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh && env'")
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
(key, _, value) = line.partition("=")
os.environ[key] = value
proc.communicate()
pprint.pprint(dict(os.environ))
```
4. Use shell task instead of python task
```shell
source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh
python /path/to/your_python_script.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] chengshiwen commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866526851
@wangdazhong001 Under different user, the result of 'env' is different. Regardless of whether it is in the docker container, the same is true in linux on the host machine. The richer environment variables directly in the container of worker occur under `roor` user. While the environment variables from the python task occur under the user whose id is 2.
Since the python script does not execute `source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, the python task cannot read the environment variables in this file.
If you want to access the environment variables from `/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, there are four solutions:
1. `open('/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh', 'r')` and parse it
2. Use `python-dotenv`
3. Use the following code:
```python
import os
import pprint
import shlex
import subprocess
command = shlex.split("env -i bash -c 'source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh && env'")
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
(key, _, value) = line.partition("=")
os.environ[key] = value
proc.communicate()
pprint.pprint(dict(os.environ))
```
4. Use shell task instead of python task
```shell
source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh
python /path/to/your_python_script.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] chengshiwen commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-872716623
Close since it has been replied
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] wangmingzhiJohn commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
wangmingzhiJohn commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866562128
@chengshiwen @ruanwenjun Thank for your advise, I will try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] chengshiwen closed issue #5685: [Question] Environment of Python Task is different from that of real worker container
Posted by GitBox <gi...@apache.org>.
chengshiwen closed issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org