You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "wangbowen1024 (via GitHub)" <gi...@apache.org> on 2023/12/22 03:19:50 UTC
[I] [Bug] [task] can't get env when runing task [dolphinscheduler]
wangbowen1024 opened a new issue, #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
can't get env when runing task, just like SPARKSQL or PYTHON or DATAX,maybe more。
i set env info into （./bin/env/dolphinscheduler_env.sh）
and set user env info :
but get error when run python task .
i debug find ,the system env not set：
and user env also not set:
### What you expected to happen
not error
### How to reproduce
everytime
### Anything else
_No response_
### Version
3.2.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "zhongjiajie (via GitHub)" <gi...@apache.org>.
zhongjiajie commented on issue #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352#issuecomment-1926440387
Hi all, you need to create new `env` in security center with both `PYTHON_LAUNCHER ` and `DATAX_LAUNCHER `
```sh
export PYTHON_LAUNCHER=/Users/zhongjiajie/.pyenv/shims/python3
export DATAX_LAUNCHER=/Users/zhongjiajie/Documents/dist/datax/bin/datax.py
```
and then use the env in task, I test latest code and it work, I would like to close this issue, you can reopen it if you have further issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "zhongjiajie (via GitHub)" <gi...@apache.org>.
zhongjiajie closed issue #15352: [Bug] [task] can't get env when runing task
URL: https://github.com/apache/dolphinscheduler/issues/15352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "wangbowen1024 (via GitHub)" <gi...@apache.org>.
wangbowen1024 commented on issue #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352#issuecomment-1886770249
> > **I have the same issue with version 3.2** Here is my environment setup in dolphinscheduler_env.sh
> > ```shell
> > export SPARK_HOME=${SPARK_HOME:-/data/spark3}
> >
> > export PYTHON_LAUNCHER=${PYTHON_LAUNCHER:-/usr/local/python3/bin/python3}
> >
> > export DATAX_LAUNCHER=${DATAX_LAUNCHER:-/usr/local/datax/bin/datax.py}
> >
> > export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_LAUNCHER:$PATH
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > When I pass the task or the result of these variables
> > ```shell
> > echo " ------shell task 1 ------"
> > echo "The python launcher program is ${PYTHON_LAUNCHER} "
> > echo "The spark home is ${SPARK_HOME}"
> > echo "The datax launcher program is ${DATAX_LAUNCHER}"`
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > As shown in the log below, they are all null.
> > ```
> > ------shell task 1 ------
> > The python launcher program is
> > The spark home is
> > The datax launcher program is
> > ```
>
> Add environment variable parameters from the management interface environment management options Then associate the above variables during flow and project operations, but it still doesn't work.
you can set these env to /etc/profile to sloved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "cnfree0355 (via GitHub)" <gi...@apache.org>.
cnfree0355 commented on issue #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352#issuecomment-1886759964
**I have the same issue with version 3.2**
Here is my environment setup in dolphinscheduler_env.sh
```bash
export SPARK_HOME=${SPARK_HOME:-/data/spark3}
export PYTHON_LAUNCHER=${PYTHON_LAUNCHER:-/usr/local/python3/bin/python3}
export DATAX_LAUNCHER=${DATAX_LAUNCHER:-/usr/local/datax/bin/datax.py}
export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_LAUNCHER:$PATH
```
When I pass the task or the result of these variables
```bash
echo " ------shell task 1 ------"
echo "The python launcher program is ${PYTHON_LAUNCHER} "
echo "The spark home is ${SPARK_HOME}"
echo "The datax launcher program is ${DATAX_LAUNCHER}"`
```
As shown in the log below, they are all null.
```text
------shell task 1 ------
The python launcher program is
The spark home is
The datax launcher program is
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "wangbowen1024 (via GitHub)" <gi...@apache.org>.
wangbowen1024 commented on issue #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352#issuecomment-1869090508
> Try to import the env in your task. <img alt="image" width="592" src="https://private-user-images.githubusercontent.com/22415594/292517722-af2ea78a-ac80-44d5-bb6b-1b23e6bab4e4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDM1MzE5MDIsIm5iZiI6MTcwMzUzMTYwMiwicGF0aCI6Ii8yMjQxNTU5NC8yOTI1MTc3MjItYWYyZWE3OGEtYWM4MC00NGQ1LWJiNmItMWIyM2U2YmFiNGU0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjI1VDE5MTMyMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI4ZjA3NDhjOTQ5MjRjM2JiNDNhZWVkNTA5Mzg5MWJiZTM3YjZhMzNkYTMzZDVlZjQzOGE5ODQ5ODNhMDVjZWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.BjPx7xVesQQ6zwlAwZ906rslUnaMsrJOUJlxlWIHhmk">
I tried, but I failed. Now I write these environment variables to /etc/profile to resolve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "cnfree0355 (via GitHub)" <gi...@apache.org>.
cnfree0355 commented on issue #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352#issuecomment-1886764029
> **I have the same issue with version 3.2** Here is my environment setup in dolphinscheduler_env.sh
>
> ```shell
> export SPARK_HOME=${SPARK_HOME:-/data/spark3}
>
> export PYTHON_LAUNCHER=${PYTHON_LAUNCHER:-/usr/local/python3/bin/python3}
>
> export DATAX_LAUNCHER=${DATAX_LAUNCHER:-/usr/local/datax/bin/datax.py}
>
> export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_LAUNCHER:$PATH
> ```
>
> When I pass the task or the result of these variables
>
> ```shell
> echo " ------shell task 1 ------"
> echo "The python launcher program is ${PYTHON_LAUNCHER} "
> echo "The spark home is ${SPARK_HOME}"
> echo "The datax launcher program is ${DATAX_LAUNCHER}"`
> ```
>
> As shown in the log below, they are all null.
>
> ```
> ------shell task 1 ------
> The python launcher program is
> The spark home is
> The datax launcher program is
> ```
Add environment variable parameters from the management interface environment management options
Then associate the above variables during flow and project operations, but it still doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352#issuecomment-1867803018
Try to import the env in your task?
<img width="592" alt="image" src="https://github.com/apache/dolphinscheduler/assets/22415594/af2ea78a-ac80-44d5-bb6b-1b23e6bab4e4">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task] can't get env when runing task [dolphinscheduler]
Posted by "oicq1699 (via GitHub)" <gi...@apache.org>.
oicq1699 commented on issue #15352:
URL: https://github.com/apache/dolphinscheduler/issues/15352#issuecomment-1897724433
In the old version log, you can see that when generating the shell script, there will be a line source `/opt/dolphin/worker-server/conf/dolphinscheduler_env.sh` script, but in version 3.2, this line is not seen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org