[GitHub] [dolphinscheduler] yutianaiqingtian commented on issue #10938: [Question] [3.0.0-beta-2] /tmp/dolphinscheduler/exec/process.command: /bin/spark-submit: not found

[GitBox <gi...@apache.org>]

yutianaiqingtian commented on issue #10938:
URL: https://github.com/apache/dolphinscheduler/issues/10938#issuecomment-1213029754

   I have the same issue when use the 3.0.0-release version.  The log as brew.
   
   ```
   [LOG-PATH]: /opt/dolphinscheduler/logs/20220812/6508614388448_7-10-14.log, [HOST]:  Host{address='my-do-worker-1.my-do-worker-headless:1234', ip='my-do-worker-1.my-do-worker-headless', port=1234}
   [INFO] 2022-08-12 19:12:36.086 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[69] - spark task params {"localParams":[],"rawScript":"","resourceList":[],"programType":"SCALA","mainClass":"org.apache.spark.examples.SparkPi","mainJar":{"id":2,"resourceName":"/spark-examples_2.12-3.2.1.jar","res":"spark-examples_2.12-3.2.1.jar"},"deployMode":"client","sparkVersion":"SPARK2","driverCores":1,"driverMemory":"512M","numExecutors":2,"executorMemory":"2G","executorCores":2}
   [INFO] 2022-08-12 19:12:36.091 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[130] - spark task command: ${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default spark-examples_2.12-3.2.1.jar
   [INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[85] - tenantCode user:telent01, task dir:10_14
   [INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[90] - create command file:/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command
   [INFO] 2022-08-12 19:12:36.092 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[116] - command : #!/bin/sh
   BASEDIR=$(cd `dirname $0`; pwd)
   cd $BASEDIR
   source /opt/dolphinscheduler/conf/dolphinscheduler_env.sh
   ${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default spark-examples_2.12-3.2.1.jar
   [INFO] 2022-08-12 19:12:36.096 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[290] - task run command: sudo -u telent01 sh /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command
   [INFO] 2022-08-12 19:12:36.097 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[181] - process start, process id is: 7776
   [INFO] 2022-08-12 19:12:36.102 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[205] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14, processId:7776 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
   [INFO] 2022-08-12 19:12:37.098 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[63] -  -> /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: 4: /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: source: not found
   	/tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: 5: /tmp/dolphinscheduler/exec/process/6508601296608/6508614388448_7/10/14/10_14.command: /bin/spark-submit: not found
   [INFO] 2022-08-12 19:12:37.099 +0800 [taskAppId=TASK-20220812-6508614388448_7-10-14] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[57] - FINALIZE_SESSION
   
   ```
   
   **The default script compiler is sh, not the bash**. and the #!/bin/sh not have the source command default. If you want to make the environment take effect. you can overwrite the environment on the workflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org