[GitHub] [dolphinscheduler] chengshiwen commented on issue #5685: [Question] Environment of Python Task is different from that of real worker container

[GitBox <gi...@apache.org>]

chengshiwen commented on issue #5685:
URL: https://github.com/apache/dolphinscheduler/issues/5685#issuecomment-866526851


   @wangdazhong001 Under different user, the result of 'env' is different. Regardless of whether it is in the docker container, the same is true in linux on the host machine. The richer environment variables directly in the container of worker occur under `roor` user.  While the environment variables from the python task occur under the user whose id is 2.
   
   Since the python script does not execute `source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, the python task cannot read the environment variables in this file.
   
   If you want to access the environment variables from `/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh`, there are four solutions:
   
   1. `open('/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh', 'r')` and parse it
   2. Use `python-dotenv`
   3. Use the following code:
   ```python
   import os
   import pprint
   import shlex
   import subprocess
   
   command = shlex.split("env -i bash -c 'source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh && env'")
   proc = subprocess.Popen(command, stdout = subprocess.PIPE)
   for line in proc.stdout:
     (key, _, value) = line.partition("=")
     os.environ[key] = value
   proc.communicate()
   
   pprint.pprint(dict(os.environ))
   ```
   4. Use shell task instead of python task
   ```shell
   source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh
   python /path/to/your_python_script.py
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org