You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/08/23 03:34:52 UTC
[GitHub] [dolphinscheduler] Tandoy opened a new issue #6021: [Question] Use a shell script to encapsulate the spark-submit command to submit the spark task to yarn and the task status is abnormal.
Tandoy opened a new issue #6021:
URL: https://github.com/apache/dolphinscheduler/issues/6021
**Steps to reproduce the behavior**
Use shell script to encapsulate spark-submit command to submit spark task to yarn
**Describe the question**
A shell script is used to encapsulate the spark-submit command to execute a spark task. Some tasks are executed successfully but the dolphinscheduler returns a status of failure. But there are also some tasks in the workflow that dolphinscheduler returns successfully. And using the same shell script to execute hive tasks does not exist.
**Environment Description:**
dolphinscheduler：1.3.2
Hadoop version : 2.6.0-cdh5.13.3
Hive version : 1.1.0-cdh5.13.3
yarn.resourcemanager：single
Storage (HDFS/S3/GCS..) : HDFS
Running on Docker? (yes/no) :no
**common.properties**
**shell script**
#!/bin/bash
sqlfile_name=$1
data_date=$2
curDay=`echo $data_date | cut -c 7-8`
curMonth=`echo $data_date | cut -c 5-6`
curYear=`echo $data_date |cut -c 1-4`
curCal=`cal $curMonth $curYear`
endDay=`echo $curCal | awk '{print $NF}'`
#echo $endDay
if [ "$curDay" -eq "$endDay" ]
then
echo "开始执行.sql文件"
spark2-submit \
--class com.wecreditlife.framework.SparkExecSqlResApp \
--master yarn \
--deploy-mode cluster \
--driver-memory 1G \
--num-executors 2 \
--executor-memory 1G \
--executor-cores 4 \
--conf spark.yarn.executor.memoryOverhead=512 \
--conf spark.yarn.driver.memoryOverhead=512 \
/opt/xysh/xysh-udf-project-1.0-SNAPSHOT.jar ${sqlfile_name} ${data_date}
echo "执行完毕"
else
echo "ignore"
fi
**Additional context:**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #6021: [Question] Use a shell script to encapsulate the spark-submit command to submit the spark task to yarn and the task status is abnormal.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6021:
URL: https://github.com/apache/dolphinscheduler/issues/6021#issuecomment-903416338
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #6021: [Question] Use a shell script to encapsulate the spark-submit command to submit the spark task to yarn and the task status is abnormal.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6021:
URL: https://github.com/apache/dolphinscheduler/issues/6021#issuecomment-903416279
**Steps to reproduce the behavior**
Use shell script to encapsulate spark-submit command to submit spark task to yarn
**Describe the question**
A shell script is used to encapsulate the spark-submit command to execute a spark task. Some tasks are executed successfully but the dolphinscheduler returns a status of failure. But there are also some tasks in the workflow that dolphinscheduler returns successfully. And using the same shell script to execute hive tasks does not exist.
**Environment Description:**
dolphinscheduler：1.3.2
Hadoop version : 2.6.0-cdh5.13.3
Hive version : 1.1.0-cdh5.13.3
yarn.resourcemanager：single
Storage (HDFS/S3/GCS..) : HDFS
Running on Docker? (yes/no) :no
**common.properties**
**shell script**
#!/bin/bash
sqlfile_name=$1
data_date=$2
curDay=`echo $data_date | cut -c 7-8`
curMonth=`echo $data_date | cut -c 5-6`
curYear=`echo $data_date |cut -c 1-4`
curCal=`cal $curMonth $curYear`
endDay=`echo $curCal | awk '{print $NF}'`
#echo $endDay
if [ "$curDay" -eq "$endDay" ]
then
echo "开始执行.sql文件"
spark2-submit \
--class com.wecreditlife.framework.SparkExecSqlResApp \
--master yarn \
--deploy-mode cluster \
--driver-memory 1G \
--num-executors 2 \
--executor-memory 1G \
--executor-cores 4 \
--conf spark.yarn.executor.memoryOverhead=512 \
--conf spark.yarn.driver.memoryOverhead=512 \
/opt/xysh/xysh-udf-project-1.0-SNAPSHOT.jar ${sqlfile_name} ${data_date}
echo "执行完毕"
else
echo "ignore"
fi
**Additional context:**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org