You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Something Something <ma...@gmail.com> on 2012/11/02 18:43:16 UTC
Running Python script under Oozie
Hello,
We have a Python script that we run as follows:
pig xyz.py –arg1 value1 arg2 value2
Basically, this runs under Jython.
Anyway, under Oozie I have set the workflow as follows:
<workflow-app xmlns='uri:oozie:workflow:0.3' name='my-wf'>
<start to='job1' />
<action name='job1'>
<pig>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
</configuration>
<script>${myscript}</script>
</pig>
<ok to="end" />
<error to="fail" />
</action>
<kill name="fail">
<message>Pig failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name='end' />
</workflow-app>
I have made the script very simple. It now has only one line:
print 'hello world from Oozie'
When I run it, I get this message:
before resolve [Pig failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]], after resolve [Pig
failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain],
exit code [2]]]
I also see this WARN message:
E1100: Command precondition does not hold before execution, [, coord action
is null], Error Code: E1100
Anyway, has anyone run a Python script via Pig & Oozie? I also need to
pass command line arguments to this script. Any samples would be greatly
appreciated. Thanks.
Re: Running Python script under Oozie
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi,
For standalone python script, you can use the "shell" action.
Looks like your example py script is used with pig.
Did you go to the hadoop launcher job log. That task log might have details reason.
If you got that, please send the log to debug.
Regards,
Mohammad
________________________________
From: Something Something <ma...@gmail.com>
To: user@oozie.apache.org
Sent: Friday, November 2, 2012 4:48 PM
Subject: Re: Running Python script under Oozie
I made some progress by adding jython.jar to the /lib directory, but it's
not working as expected.
It seems to me like Oozie is still not very developer friendly as yet:
1) The only error message I see is this:
[Pig failed, error message[Main class
[org.apache.oozie.action.hadoop.PigMain], exit code [2]]]
I just guessed that it might be looking for the jython.jar etc.
2) The interim output directory gets deleted. I tried -nocleanup with no
luck. In other words, it's really hard to debug issues.
Anyway, if anyone has run Python script via Oozie, would love to see how
you did it. Thanks.
On Fri, Nov 2, 2012 at 10:43 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Hello,
>
> We have a Python script that we run as follows:
>
> pig xyz.py –arg1 value1 arg2 value2
>
> Basically, this runs under Jython.
>
> Anyway, under Oozie I have set the workflow as follows:
>
> <workflow-app xmlns='uri:oozie:workflow:0.3' name='my-wf'>
> <start to='job1' />
> <action name='job1'>
> <pig>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> </property>
> <property>
> <name>mapred.job.queue.name</name>
> <value>default</value>
> </property>
> </configuration>
> <script>${myscript}</script>
> </pig>
> <ok to="end" />
> <error to="fail" />
> </action>
> <kill name="fail">
> <message>Pig failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name='end' />
> </workflow-app>
>
>
>
> I have made the script very simple. It now has only one line:
>
> print 'hello world from Oozie'
>
> When I run it, I get this message:
>
> before resolve [Pig failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]], after resolve [Pig
> failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain],
> exit code [2]]]
>
>
> I also see this WARN message:
>
> E1100: Command precondition does not hold before execution, [, coord
> action is null], Error Code: E1100
>
> Anyway, has anyone run a Python script via Pig & Oozie? I also need to
> pass command line arguments to this script. Any samples would be greatly
> appreciated. Thanks.
>
>
Re: Running Python script under Oozie
Posted by Something Something <ma...@gmail.com>.
I made some progress by adding jython.jar to the /lib directory, but it's
not working as expected.
It seems to me like Oozie is still not very developer friendly as yet:
1) The only error message I see is this:
[Pig failed, error message[Main class
[org.apache.oozie.action.hadoop.PigMain], exit code [2]]]
I just guessed that it might be looking for the jython.jar etc.
2) The interim output directory gets deleted. I tried -nocleanup with no
luck. In other words, it's really hard to debug issues.
Anyway, if anyone has run Python script via Oozie, would love to see how
you did it. Thanks.
On Fri, Nov 2, 2012 at 10:43 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Hello,
>
> We have a Python script that we run as follows:
>
> pig xyz.py –arg1 value1 arg2 value2
>
> Basically, this runs under Jython.
>
> Anyway, under Oozie I have set the workflow as follows:
>
> <workflow-app xmlns='uri:oozie:workflow:0.3' name='my-wf'>
> <start to='job1' />
> <action name='job1'>
> <pig>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> </property>
> <property>
> <name>mapred.job.queue.name</name>
> <value>default</value>
> </property>
> </configuration>
> <script>${myscript}</script>
> </pig>
> <ok to="end" />
> <error to="fail" />
> </action>
> <kill name="fail">
> <message>Pig failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name='end' />
> </workflow-app>
>
>
>
> I have made the script very simple. It now has only one line:
>
> print 'hello world from Oozie'
>
> When I run it, I get this message:
>
> before resolve [Pig failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]], after resolve [Pig
> failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain],
> exit code [2]]]
>
>
> I also see this WARN message:
>
> E1100: Command precondition does not hold before execution, [, coord
> action is null], Error Code: E1100
>
> Anyway, has anyone run a Python script via Pig & Oozie? I also need to
> pass command line arguments to this script. Any samples would be greatly
> appreciated. Thanks.
>
>