You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/04/16 10:46:53 UTC
[GitHub] [incubator-dolphinscheduler] lslone opened a new issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
lslone opened a new issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445
这个问题如何解决？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone commented on issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
lslone commented on issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-615031192
Problem solving：Yarn configuration problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone removed a comment on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
Posted by GitBox <gi...@apache.org>.
lslone removed a comment on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614574036
#!/bin/bash
APP=gmall
hive=/bin/hive
if [ -n "$1" ] ;then
do_date=$1
else
do_date=`date -d "-1 day" +%F`
fi
sql="
set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table "$APP".dwd_start_log
PARTITION (dt='$do_date')
select
get_json_object(data,'$.mid') mid_id,
get_json_object(data,'$.uid') user_id,
get_json_object(data,'$.vc') version_code,
get_json_object(data,'$.vn') version_name,
get_json_object(data,'$.l') lang,
get_json_object(data,'$.sr') source,
get_json_object(data,'$.os') os,
get_json_object(data,'$.ar') area,
get_json_object(data,'$.md') model,
get_json_object(data,'$.ba') brand,
get_json_object(data,'$.sv') sdk_version,
get_json_object(data,'$.g') gmail,
get_json_object(data,'$.hw') height_width,
get_json_object(data,'$.t') app_time,
get_json_object(data,'$.nw') network,
get_json_object(data,'$.ln') lng,
get_json_object(data,'$.la') lat,
get_json_object(data,'$.entry') entry,
get_json_object(data,'$.open_ad_type') open_ad_type,
get_json_object(data,'$.action') action,
get_json_object(data,'$.loading_time') loading_time,
get_json_object(data,'$.detail') detail,
get_json_object(data,'$.extend1') extend1
from "$APP".ods_start_log
where dt='$do_date';
"
$hive -e "$sql"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] xingchun-chen commented on issue #2445: [BUG] shell调度hive脚本显示执行失败！！！
Posted by GitBox <gi...@apache.org>.
xingchun-chen commented on issue #2445: [BUG] shell调度hive脚本显示执行失败！！！
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614978771
please update title in english
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone removed a comment on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
Posted by GitBox <gi...@apache.org>.
lslone removed a comment on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614573649
#!/bin/bash
# 定义变量方便修改
APP=gmall
hive=/bin/hive
# 如果是输入的日期按照取输入日期；如果没输入日期取当前时间的前一天
if [ -n "$1" ] ;then
do_date=$1
else
do_date=`date -d "-1 day" +%F`
fi
sql="
set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table "$APP".dwd_start_log
PARTITION (dt='$do_date')
select
get_json_object(data,'$.mid') mid_id,
get_json_object(data,'$.uid') user_id,
get_json_object(data,'$.vc') version_code,
get_json_object(data,'$.vn') version_name,
get_json_object(data,'$.l') lang,
get_json_object(data,'$.sr') source,
get_json_object(data,'$.os') os,
get_json_object(data,'$.ar') area,
get_json_object(data,'$.md') model,
get_json_object(data,'$.ba') brand,
get_json_object(data,'$.sv') sdk_version,
get_json_object(data,'$.g') gmail,
get_json_object(data,'$.hw') height_width,
get_json_object(data,'$.t') app_time,
get_json_object(data,'$.nw') network,
get_json_object(data,'$.ln') lng,
get_json_object(data,'$.la') lat,
get_json_object(data,'$.entry') entry,
get_json_object(data,'$.open_ad_type') open_ad_type,
get_json_object(data,'$.action') action,
get_json_object(data,'$.loading_time') loading_time,
get_json_object(data,'$.detail') detail,
get_json_object(data,'$.extend1') extend1
from "$APP".ods_start_log
where dt='$do_date';
"
$hive -e "$sql"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone edited a comment on issue #2445: [BUG] shell调度hive脚本显示执行失败！！！
Posted by GitBox <gi...@apache.org>.
lslone edited a comment on issue #2445: [BUG] shell调度hive脚本显示执行失败！！！
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614574235
实际是执行成功，但界面显示执行失败
hive脚本如下
#!/bin/bash
APP=gmall
hive=/bin/hive
if [ -n "$1" ] ;then
do_date=$1
else
do_date=date -d "-1 day" +%F
fi
sql="
set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table "$APP".dwd_start_log
PARTITION (dt='$do_date')
select
get_json_object(data,'$.mid') mid_id,
get_json_object(data,'$.uid') user_id,
get_json_object(data,'$.vc') version_code,
get_json_object(data,'$.vn') version_name,
get_json_object(data,'$.l') lang,
get_json_object(data,'$.sr') source,
get_json_object(data,'$.os') os,
get_json_object(data,'$.ar') area,
get_json_object(data,'$.md') model,
get_json_object(data,'$.ba') brand,
get_json_object(data,'$.sv') sdk_version,
get_json_object(data,'$.g') gmail,
get_json_object(data,'$.hw') height_width,
get_json_object(data,'$.t') app_time,
get_json_object(data,'$.nw') network,
get_json_object(data,'$.ln') lng,
get_json_object(data,'$.la') lat,
get_json_object(data,'$.entry') entry,
get_json_object(data,'$.open_ad_type') open_ad_type,
get_json_object(data,'$.action') action,
get_json_object(data,'$.loading_time') loading_time,
get_json_object(data,'$.detail') detail,
get_json_object(data,'$.extend1') extend1
from "$APP".ods_start_log
where dt='$do_date';
"
$hive -e "$sql"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone opened a new issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
lslone opened a new issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445
这个问题如何解决？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone commented on issue #2445: The
shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
lslone commented on issue #2445: The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-615008485
> @qiaozhanwei
详细日志
[INFO] 2020-04-17 10:47:07.675 org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl:[211] - consume tasks: [2_28_1_29_2886865333,2886865330,2886865331],there still have 0 tasks need to be executed
[INFO] 2020-04-17 10:47:07.695 org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread:[208] - worker fetch taskId : 29 from queue
[INFO] 2020-04-17 10:47:07.696 org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread:[218] - task instance local execute path : /tmp/dolphinscheduler/exec/process/1/1/28/29
[INFO] 2020-04-17 10:47:07.697 org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread:[229] - task : 29 ready to submit to task scheduler thread
[INFO] 2020-04-17 10:47:07.697 org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl:[278] - consume task /dolphinscheduler/tasks_queue/2_28_1_29_2886865333,2886865330,2886865331
[INFO] 2020-04-17 10:47:07.705 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[93] - script path : /tmp/dolphinscheduler/exec/process/1/1/28/29
[INFO] 2020-04-17 10:47:07.708 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[313] - get resource file from hdfs :/dolphinscheduler/root/resources/ods_to_dwd_log.sh
[INFO] 2020-04-17 10:47:07.722 - [taskAppId=TASK-1-28-29]:[92] - shell task params {"rawScript":"sh ods_to_dwd_log.sh $(date \"+%Y-%m-%d\")\n","localParams":[],"resourceList":[{"res":"ods_to_dwd_log.sh"}]}
[INFO] 2020-04-17 10:47:07.722 - [taskAppId=TASK-1-28-29]:[150] - raw script : sh ods_to_dwd_log.sh $(date "+%Y-%m-%d")
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[151] - task dir : /tmp/dolphinscheduler/exec/process/1/1/28/29
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[99] - tenantCode user:lsl, task dir:1_28_29
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[103] - create command file:/tmp/dolphinscheduler/exec/process/1/1/28/29/1_28_29.command
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[116] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/dolphinscheduler-backend/conf/env/.dolphinscheduler_env.sh
/tmp/dolphinscheduler/exec/process/1/1/28/29/1_28_29_node.sh
[INFO] 2020-04-17 10:47:07.724 - [taskAppId=TASK-1-28-29]:[329] - task run command:
sudo -u lsl sh /tmp/dolphinscheduler/exec/process/1/1/28/29/1_28_29.command
[INFO] 2020-04-17 10:47:07.739 - [taskAppId=TASK-1-28-29]:[158] - process start, process id is: 22765
[INFO] 2020-04-17 10:47:29.733 - [taskAppId=TASK-1-28-29]:[168] - process has exited, work dir:/tmp/dolphinscheduler/exec/process/1/1/28/29, pid:22765 ,exitStatusCode:0
[INFO] 2020-04-17 10:47:29.737 - [taskAppId=TASK-1-28-29]:[231] - process id is 22765
[INFO] 2020-04-17 10:47:29.737 - [taskAppId=TASK-1-28-29]:[432] - find app id: application_1586920707751_0019
[INFO] 2020-04-17 10:47:29.737 - [taskAppId=TASK-1-28-29]:[236] - yarn log url:application_1586920707751_0019
[ERROR] 2020-04-17 10:47:29.753 org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known
java.net.UnknownHostException: null: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
Posted by GitBox <gi...@apache.org>.
lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614573649
#!/bin/bash
# 定义变量方便修改
APP=gmall
hive=/bin/hive
# 如果是输入的日期按照取输入日期；如果没输入日期取当前时间的前一天
if [ -n "$1" ] ;then
do_date=$1
else
do_date=`date -d "-1 day" +%F`
fi
sql="
set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table "$APP".dwd_start_log
PARTITION (dt='$do_date')
select
get_json_object(data,'$.mid') mid_id,
get_json_object(data,'$.uid') user_id,
get_json_object(data,'$.vc') version_code,
get_json_object(data,'$.vn') version_name,
get_json_object(data,'$.l') lang,
get_json_object(data,'$.sr') source,
get_json_object(data,'$.os') os,
get_json_object(data,'$.ar') area,
get_json_object(data,'$.md') model,
get_json_object(data,'$.ba') brand,
get_json_object(data,'$.sv') sdk_version,
get_json_object(data,'$.g') gmail,
get_json_object(data,'$.hw') height_width,
get_json_object(data,'$.t') app_time,
get_json_object(data,'$.nw') network,
get_json_object(data,'$.ln') lng,
get_json_object(data,'$.la') lat,
get_json_object(data,'$.entry') entry,
get_json_object(data,'$.open_ad_type') open_ad_type,
get_json_object(data,'$.action') action,
get_json_object(data,'$.loading_time') loading_time,
get_json_object(data,'$.detail') detail,
get_json_object(data,'$.extend1') extend1
from "$APP".ods_start_log
where dt='$do_date';
"
$hive -e "$sql"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] xingchun-chen commented on issue
#2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
xingchun-chen commented on issue #2445: The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614996169
@qiaozhanwei
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone commented on issue #2445: The
shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
lslone commented on issue #2445: The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614995214
> 你能检查工作日志是否错误
What's the problem？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone closed issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
lslone closed issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
Posted by GitBox <gi...@apache.org>.
lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614574684
调度方法
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] xingchun-chen commented on issue #2445: [BUG] shell调度hive脚本显示执行失败！！！
Posted by GitBox <gi...@apache.org>.
xingchun-chen commented on issue #2445: [BUG] shell调度hive脚本显示执行失败！！！
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614978531
can you check worker log is error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone edited a comment on issue
#2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
lslone edited a comment on issue #2445: The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-615008485
> @qiaozhanwei
Detailed log！！！！
[INFO] 2020-04-17 10:47:07.675 org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl:[211] - consume tasks: [2_28_1_29_2886865333,2886865330,2886865331],there still have 0 tasks need to be executed
[INFO] 2020-04-17 10:47:07.695 org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread:[208] - worker fetch taskId : 29 from queue
[INFO] 2020-04-17 10:47:07.696 org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread:[218] - task instance local execute path : /tmp/dolphinscheduler/exec/process/1/1/28/29
[INFO] 2020-04-17 10:47:07.697 org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread:[229] - task : 29 ready to submit to task scheduler thread
[INFO] 2020-04-17 10:47:07.697 org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl:[278] - consume task /dolphinscheduler/tasks_queue/2_28_1_29_2886865333,2886865330,2886865331
[INFO] 2020-04-17 10:47:07.705 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[93] - script path : /tmp/dolphinscheduler/exec/process/1/1/28/29
[INFO] 2020-04-17 10:47:07.708 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[313] - get resource file from hdfs :/dolphinscheduler/root/resources/ods_to_dwd_log.sh
[INFO] 2020-04-17 10:47:07.722 - [taskAppId=TASK-1-28-29]:[92] - shell task params {"rawScript":"sh ods_to_dwd_log.sh $(date \"+%Y-%m-%d\")\n","localParams":[],"resourceList":[{"res":"ods_to_dwd_log.sh"}]}
[INFO] 2020-04-17 10:47:07.722 - [taskAppId=TASK-1-28-29]:[150] - raw script : sh ods_to_dwd_log.sh $(date "+%Y-%m-%d")
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[151] - task dir : /tmp/dolphinscheduler/exec/process/1/1/28/29
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[99] - tenantCode user:lsl, task dir:1_28_29
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[103] - create command file:/tmp/dolphinscheduler/exec/process/1/1/28/29/1_28_29.command
[INFO] 2020-04-17 10:47:07.723 - [taskAppId=TASK-1-28-29]:[116] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/dolphinscheduler-backend/conf/env/.dolphinscheduler_env.sh
/tmp/dolphinscheduler/exec/process/1/1/28/29/1_28_29_node.sh
[INFO] 2020-04-17 10:47:07.724 - [taskAppId=TASK-1-28-29]:[329] - task run command:
sudo -u lsl sh /tmp/dolphinscheduler/exec/process/1/1/28/29/1_28_29.command
[INFO] 2020-04-17 10:47:07.739 - [taskAppId=TASK-1-28-29]:[158] - process start, process id is: 22765
[INFO] 2020-04-17 10:47:29.733 - [taskAppId=TASK-1-28-29]:[168] - process has exited, work dir:/tmp/dolphinscheduler/exec/process/1/1/28/29, pid:22765 ,exitStatusCode:0
[INFO] 2020-04-17 10:47:29.737 - [taskAppId=TASK-1-28-29]:[231] - process id is 22765
[INFO] 2020-04-17 10:47:29.737 - [taskAppId=TASK-1-28-29]:[432] - find app id: application_1586920707751_0019
[INFO] 2020-04-17 10:47:29.737 - [taskAppId=TASK-1-28-29]:[236] - yarn log url:application_1586920707751_0019
[ERROR] 2020-04-17 10:47:29.753 org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known
java.net.UnknownHostException: null: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
Posted by GitBox <gi...@apache.org>.
lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614574235
hive脚本如下
#!/bin/bash
APP=gmall
hive=/bin/hive
if [ -n "$1" ] ;then
do_date=$1
else
do_date=date -d "-1 day" +%F
fi
sql="
set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table "$APP".dwd_start_log
PARTITION (dt='$do_date')
select
get_json_object(data,'$.mid') mid_id,
get_json_object(data,'$.uid') user_id,
get_json_object(data,'$.vc') version_code,
get_json_object(data,'$.vn') version_name,
get_json_object(data,'$.l') lang,
get_json_object(data,'$.sr') source,
get_json_object(data,'$.os') os,
get_json_object(data,'$.ar') area,
get_json_object(data,'$.md') model,
get_json_object(data,'$.ba') brand,
get_json_object(data,'$.sv') sdk_version,
get_json_object(data,'$.g') gmail,
get_json_object(data,'$.hw') height_width,
get_json_object(data,'$.t') app_time,
get_json_object(data,'$.nw') network,
get_json_object(data,'$.ln') lng,
get_json_object(data,'$.la') lat,
get_json_object(data,'$.entry') entry,
get_json_object(data,'$.open_ad_type') open_ad_type,
get_json_object(data,'$.action') action,
get_json_object(data,'$.loading_time') loading_time,
get_json_object(data,'$.detail') detail,
get_json_object(data,'$.extend1') extend1
from "$APP".ods_start_log
where dt='$do_date';
"
$hive -e "$sql"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
Posted by GitBox <gi...@apache.org>.
lslone commented on issue #2445: shell调度hive脚本，调度界面显示失败但是yarn是执行成功的
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-614574036
#!/bin/bash
APP=gmall
hive=/bin/hive
if [ -n "$1" ] ;then
do_date=$1
else
do_date=`date -d "-1 day" +%F`
fi
sql="
set hive.exec.dynamic.partition.mode=nonstrict;
insert overwrite table "$APP".dwd_start_log
PARTITION (dt='$do_date')
select
get_json_object(data,'$.mid') mid_id,
get_json_object(data,'$.uid') user_id,
get_json_object(data,'$.vc') version_code,
get_json_object(data,'$.vn') version_name,
get_json_object(data,'$.l') lang,
get_json_object(data,'$.sr') source,
get_json_object(data,'$.os') os,
get_json_object(data,'$.ar') area,
get_json_object(data,'$.md') model,
get_json_object(data,'$.ba') brand,
get_json_object(data,'$.sv') sdk_version,
get_json_object(data,'$.g') gmail,
get_json_object(data,'$.hw') height_width,
get_json_object(data,'$.t') app_time,
get_json_object(data,'$.nw') network,
get_json_object(data,'$.ln') lng,
get_json_object(data,'$.la') lat,
get_json_object(data,'$.entry') entry,
get_json_object(data,'$.open_ad_type') open_ad_type,
get_json_object(data,'$.action') action,
get_json_object(data,'$.loading_time') loading_time,
get_json_object(data,'$.detail') detail,
get_json_object(data,'$.extend1') extend1
from "$APP".ods_start_log
where dt='$do_date';
"
$hive -e "$sql"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] lslone closed issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
lslone closed issue #2445: 【BUG】The shell dispatching hive script failed to execute!!
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] gaogaoshan commented on issue #2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
gaogaoshan commented on issue #2445:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-630688743
我是如下配置的但是还是报错.dolphinscheduler.common.utils.HadoopUtils:[85] - null
java.lang.NullPointerException: null，方便贴一下你的配置吗
vi dsInstallPath/dolphinscheduler/conf/common.properties
yarn.resourcemanager.ha.rm.ids=
yarn.application.status.address=10.59.66.11 #我的resourcemanager的IP
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaogaoshan commented on issue #2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
gaogaoshan commented on issue #2445:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-630710827
我重新install以后可以了
install.sh
yarnHaIps=""
singleYarnIp="10.59.xxx.xxx"
===》重新安装后会如下
common.properties
yarn.resourcemanager.ha.rm.ids=
yarn.application.status.address=http://10.59.xxx.xxx:8088/ws/v1/cluster/apps/%s
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 removed a comment on issue #2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
743294668 removed a comment on issue #2445:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-617732596
@lslone
我现在涉及提交yarn的shell任务也是因为这个错误失败了，想问下你是怎么解决的呢？谢谢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 commented on issue #2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
743294668 commented on issue #2445:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-617732596
@lslone
我现在涉及提交yarn的shell人也是因为这个错误失败了，想问下你是怎么解决的呢？谢谢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 edited a comment on issue #2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
743294668 edited a comment on issue #2445:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-617732596
@lslone
我现在涉及提交yarn的shell任务也是因为这个错误失败了，想问下你是怎么解决的呢？谢谢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaogaoshan commented on issue #2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
gaogaoshan commented on issue #2445:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-630675596
hi lslone 我也是这个问题，请问yarn 我没有做ha 应该怎么配置？
yarn.application.status.address=ResourceManager的IP吗
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 commented on issue #2445: The shell dispatching hive script failed to execute!!
Posted by GitBox <gi...@apache.org>.
743294668 commented on issue #2445:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2445#issuecomment-630677966
> hi lslone 我也是这个问题，请问yarn 我没有做ha 应该怎么配置？
> yarn.application.status.address=ResourceManager的IP吗
Just replace the IP in yarn.application.status.address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org