You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by lk_hadoop <lk...@163.com> on 2016/10/27 03:09:24 UTC
may I run distcp by using shellaction
hi,all:
I want to use distcp tp copy ftp files to hdfs . I tried distcp action ,but I got error with:java.lang.IllegalArgumentException: begin > end in range (begin, end): (1415382327807, 1415382308114)
but it was succeed with command:
hadoop distcp ftp://youeftp:youeftp@192.168.0.232/biz/biz_20160803.zip hdfs://dev0:9000/sourcedata/biz/201608/biz_20160803.zip
so I write the command into a sh file and try to use oozie shell action:
but the job run succeed with out running the command. my workflow.xml like this:
<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-distcp">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>distcp.sh</exec>
<!--
<argument>my_output=Hello Oozie</argument>
-->
<file>/user/hadoop/examples/apps/youedata/weixin/distcp.sh#distcp.sh</file>
<capture-output/>
</shell>
<ok to="check-output"/>
<error to="fail"/>
</action>
<decision name="check-output">
<switch>
<case to="end">
${wf:actionData('shell-node')['my_output'] eq 'Hello Oozie'}
</case>
<default to="fail-output"/>
</switch>
</decision>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>
</workflow-app>
--------------------------------------------------------------------------------------------------------------
and distcp.sh:
#!/bin/bash
hadoop distcp ftp://youeftp:youeftp@192.168.0.232/weixin/biz/biz_20161025.zip hdfs://dev0:9000/sourcedata/weixin/biz/biz/biz_20161025.zip
echo "my_output=Hello Oozie"
2016-10-27
lk_hadoop
Re: RE: may I run distcp by using shellaction
Posted by lk_hadoop <lk...@163.com>.
thanks a lot ganesh raman . I've resolved the distcp action failure. it's because of nodes of cluster didn't synchronize datetime.
but I really don't understand why sell action not work
2016-10-27
lk_hadoop
发件人：ganesh raman <ga...@thedatateam.in>
发送时间：2016-10-27 17:25
主题：RE: may I run distcp by using shellaction
收件人："user"<us...@oozie.apache.org>
抄送：
Hi
Can you help post the fuller stack trace for that distcp action failure?
May be we could add some conf param to silence something towards completion?
-----Original Message-----
From: "lk_hadoop" <lk...@163.com>
Sent: 10/27/2016 8:39 AM
To: "user" <us...@oozie.apache.org>
Subject: may I run distcp by using shellaction
hi,all:
I want to use distcp tp copy ftp files to hdfs . I tried distcp action ,but I got error with:java.lang.IllegalArgumentException: begin > end in range (begin, end): (1415382327807, 1415382308114)
but it was succeed with command:
hadoop distcp ftp://youeftp:youeftp@192.168.0.232/biz/biz_20160803.zip hdfs://dev0:9000/sourcedata/biz/201608/biz_20160803.zip
so I write the command into a sh file and try to use oozie shell action:
but the job run succeed with out running the command. my workflow.xml like this:
<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-distcp">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>distcp.sh</exec>
<!--
<argument>my_output=Hello Oozie</argument>
-->
<file>/user/hadoop/examples/apps/youedata/weixin/distcp.sh#distcp.sh</file>
<capture-output/>
</shell>
<ok to="check-output"/>
<error to="fail"/>
</action>
<decision name="check-output">
<switch>
<case to="end">
${wf:actionData('shell-node')['my_output'] eq 'Hello Oozie'}
</case>
<default to="fail-output"/>
</switch>
</decision>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>
</workflow-app>
--------------------------------------------------------------------------------------------------------------
and distcp.sh:
#!/bin/bash
hadoop distcp ftp://youeftp:youeftp@192.168.0.232/weixin/biz/biz_20161025.zip hdfs://dev0:9000/sourcedata/weixin/biz/biz/biz_20161025.zip
echo "my_output=Hello Oozie"
2016-10-27
lk_hadoop
RE: may I run distcp by using shellaction
Posted by ganesh raman <ga...@thedatateam.in>.
Hi
Can you help post the fuller stack trace for that distcp action failure?
May be we could add some conf param to silence something towards completion?
-----Original Message-----
From: "lk_hadoop" <lk...@163.com>
Sent: 10/27/2016 8:39 AM
To: "user" <us...@oozie.apache.org>
Subject: may I run distcp by using shellaction
hi,all:
I want to use distcp tp copy ftp files to hdfs . I tried distcp action ,but I got error with:java.lang.IllegalArgumentException: begin > end in range (begin, end): (1415382327807, 1415382308114)
but it was succeed with command:
hadoop distcp ftp://youeftp:youeftp@192.168.0.232/biz/biz_20160803.zip hdfs://dev0:9000/sourcedata/biz/201608/biz_20160803.zip
so I write the command into a sh file and try to use oozie shell action:
but the job run succeed with out running the command. my workflow.xml like this:
<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-distcp">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>distcp.sh</exec>
<!--
<argument>my_output=Hello Oozie</argument>
-->
<file>/user/hadoop/examples/apps/youedata/weixin/distcp.sh#distcp.sh</file>
<capture-output/>
</shell>
<ok to="check-output"/>
<error to="fail"/>
</action>
<decision name="check-output">
<switch>
<case to="end">
${wf:actionData('shell-node')['my_output'] eq 'Hello Oozie'}
</case>
<default to="fail-output"/>
</switch>
</decision>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<kill name="fail-output">
<message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
</kill>
<end name="end"/>
</workflow-app>
--------------------------------------------------------------------------------------------------------------
and distcp.sh:
#!/bin/bash
hadoop distcp ftp://youeftp:youeftp@192.168.0.232/weixin/biz/biz_20161025.zip hdfs://dev0:9000/sourcedata/weixin/biz/biz/biz_20161025.zip
echo "my_output=Hello Oozie"
2016-10-27
lk_hadoop