You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/13 09:16:26 UTC
[GitHub] [dolphinscheduler] zhouhongyu888 opened a new issue, #10938: [Bug] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
zhouhongyu888 opened a new issue, #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
I hava deployed ds by docker-compose and mount the spark-client on localhost,when I submit a spark-task(the example task of sparkPi), the error below happends:
[INFO] 2022-07-13 08:30:49.424 +0000 [taskAppId=TASK-20220713-6178360895936_1-48-48] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[63] - -> /tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: 4: /tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: source: not found
/tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: 5: /tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: /bin/spark-submit: not found
### What you expected to happen
the task can run successfully
### How to reproduce
1、deployed dolphinscheduler by docker-compose
2、mount the directory /opt/soft on localhost
### Anything else
_No response_
### Version
3.0.0-beta-2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhouhongyu888 commented on issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
zhouhongyu888 commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1184041554
> It seems you lack of spark environment.
This is a bug,when execute the command, it seems to use /bin/spark-submit, not $SPARK_HOME2/bin/spark-submit。so I copy the scripts in the directory of $SPARK_HOME2/bin to the directory of /bin,the bug disappears
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10938: [Bug] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1183021282
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1184042765
> /tmp/dolphinscheduler/exec/process/6169061703072/6178360895936_1/48/48/48_48.command: source: not found
The reason for the failure is that the source environment variable failed. Please check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] black-06 commented on issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
black-06 commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1269309737
Unlucky, this bug was recurred in 3.1.0 :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1225618705
Can we use `#!/bin/bash` to replace `#!/bin/sh`? @SbloodyS @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1183348017
It seems you lack of spark environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhuangchong closed issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
zhuangchong closed issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
URL: https://github.com/apache/dolphinscheduler/issues/10938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yutianaiqingtian commented on issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found
Posted by GitBox <gi...@apache.org>.
yutianaiqingtian commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1213029754
I have the same issue when use the 3.0.0-release version. The log as brew.
```
[LOG-PATH]: /opt/dolphinscheduler/logs/20220812/6508614388448_7-10-14.log, [HOST]: Host{address='my-do-worker-1.my-do-worker-headless:1234', ip='my-do-worker-1.my-do-worker-headless', port=1234}
[INFO] 2022-08-12 19:12:36.086 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[69] - spark task params {"localParams":[],"rawScript":"","resourceList":[],"programType":"SCALA","mainClass":"org.apache.spark.examples.SparkPi","mainJar":{"id":2,"resourceName":"/spark-examples_2.12-3.2.1.jar","res":"spark-examples_2.12-3.2.1.jar"},"deployMode":"client","sparkVersion":"SPARK2","driverCores":1,"driverMemory":"512M","numExecutors":2,"executorMemory":"2G","executorCores":2}
[INFO] 2022-08-12 19:12:36.091 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[130] - spark task command: ${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default spark-examples_2.12-3.2.1.jar
[INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[85] - tenantCode user:telent01, task dir:10_14
[INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[90] - create command file:/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command
[INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[116] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/dolphinscheduler/conf/dolphinscheduler_env.sh
${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default spark-examples_2.12-3.2.1.jar
[INFO] 2022-08-12 19:12:36.096 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[290] - task run command: sudo -u telent01 sh /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command
[INFO] 2022-08-12 19:12:36.097 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[181] - process start, process id is: 7776
[INFO] 2022-08-12 19:12:36.102 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[205] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14, processId:7776 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
[INFO] 2022-08-12 19:12:37.098 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[63] - -> /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: 4: /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: source: not found
/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: 5: /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: /bin/spark-submit: not found
[INFO] 2022-08-12 19:12:37.099 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[57] - FINALIZE_SESSION
```
**The default script compiler is sh, not the bash**. and the #!/bin/sh not have the source command default. If you want to make the environment take effect. you can overwrite the environment on the workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org