You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Arsalan Bilal <ch...@gmail.com> on 2012/03/09 14:51:43 UTC
Information about job priorities
Hi User,
I am using oozie to Manage my job. Now my requirement is to set job
priorities. i want to run job according to priorities.
if job fail then priority will be low, if all job come together , then it
should decide which job run first after seeing priorities and condition in
work-flow.
Any one Know , how can i set job priorities using oozie? Any informative
link or tutorial?
--
Best Regards,
Arsalan Bilal
Re: Information about job priorities
Posted by Arsalan Bilal <ch...@gmail.com>.
Hi Mona
i want to run fail or killed coordinator action again. how can i define
properties to re run fail coordinator action again.?
my coordinator.xml
<coordinator-app name="Pfo" frequency="10" start="2012-03-13T07:52Z"
> end="2012-04-13T18:00Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.1">
> <controls>
> <timeout>0</timeout>
> <concurrency>1</concurrency>
> <execution>LIFO</execution>
> </controls>
>
> <action>
> <workflow>
> <app-path>${workflowPath}</app-path>
> <configuration>
> <property>
> <name>jobTracker</name>
> <value>${jobTracker}</value>
> </property>
> <property>
> <name>nameNode</name>
> <value>${nameNode}</value>
> </property>
> <property>
> <name>queueName</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> </workflow>
> </action>
> </coordinator-app>
>
my workflow.xml
<workflow-app xmlns='uri:oozie:workflow:0.1' name='java-main-wf'>
> <start to='java1' />
> <action name='java1'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <prepare>
> <delete path="hdfs://localhost:8020/user/root/URI/op" />
> </prepare>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>default</value>
> </property>
> <property>
> <name>oozie.launcher.mapred.job.priority</name>
> <value>VERY_HIGH</value>
> </property>
> <property>
> <name>mapred.job.priority</name>
> <value>VERY_HIGH</value>
> </property>
>
> </configuration>
> <main-class>pof.usage.Main</main-class>
> </java>
> <ok to="end" />
> <error to="fail" />
> </action>
> <kill name="fail">
> <message>Java failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name='end' />
> </workflow-app>
>
On Sat, Mar 10, 2012 at 12:33 AM, Mona Chitnis <ch...@yahoo-inc.com>wrote:
> Hello,
>
> Oozie does not handle job priorities in its own scheduler. As soon as a
> workflow job is ready to do a transition, Oozie will trigger the transition.
>
> The remote system Oozie is submitting the job to (Hadoop), can be passed
> the appropriate configuration for prioritizing your job. Oozie uses a
> launcher (single map task, no reduce tasks) to configure your Hadoop job.
> You can include the following configuration parameters in your Oozie
> workflow.xml for your launcher and job.
>
> Option 1: setting ‘mapred.job.priority’
>
> <property>
> <name>oozie.launcher.mapred.job.priority</name>
> <value>${priority}</value>
> </property>
> <property>
> <name>mapred.job.priority</name>
> <value>${priority}</value>
> </property>
>
> Referring to Hadoop documentation on job priorities,
> http://hadoop.apache.org/common/docs/current/commands_manual.html#job
> Where priority value can be one of VERY_HIGH | HIGH | NORMAL | LOW |
> VERY_LOW
>
> Option 2: setting ‘mapred.job.queue.name’
> Different queues may have pre-determined priorities on the Hadoop cluster.
> Leveraging that, you can configure to run your Oozie launcher and job on
> specified queues.
>
> Are you interested in prioritizing Oozie workflow or coordinator jobs? In
> case of co-ordinator job specification, there is a <controls> block that
> can help order your execution priority beforehand.
>
> <coordinator-app name="[NAME]" frequency="[FREQUENCY]"
> start="[DATETIME]" end="[DATETIME]"
> timezone="[TIMEZONE]"
> xmlns="uri:oozie:coordinator:0.1">
> <controls>
> <concurrency>[CONCURRENCY]</concurrency>
> <execution>[EXECUTION_STRATEGY]</execution> <!-- value can be FIFO
> | LIFO | LAST_ONLY -->
>
> The documentation for the above is here:
>
> http://yahoo.github.com/oozie/releases/3.0.0/CoordinatorFunctionalSpec.html#a6.3._Synchronous_Coordinator_Application_Definition
>
>
> Let me know if you have more questions.
>
> Thanks,
>
> Mona
>
>
>
> On 3/9/12 5:51 AM, "Arsalan Bilal" <ch...@gmail.com> wrote:
>
> Hi User,
>
> I am using oozie to Manage my job. Now my requirement is to set job
> priorities. i want to run job according to priorities.
> if job fail then priority will be low, if all job come together , then it
> should decide which job run first after seeing priorities and condition in
> work-flow.
> Any one Know , how can i set job priorities using oozie? Any informative
> link or tutorial?
>
>
>
--
Best Regards,
Arsalan Bilal
Re: Information about job priorities
Posted by Mona Chitnis <ch...@yahoo-inc.com>.
Hello,
Oozie does not handle job priorities in its own scheduler. As soon as a workflow job is ready to do a transition, Oozie will trigger the transition.
The remote system Oozie is submitting the job to (Hadoop), can be passed the appropriate configuration for prioritizing your job. Oozie uses a launcher (single map task, no reduce tasks) to configure your Hadoop job. You can include the following configuration parameters in your Oozie workflow.xml for your launcher and job.
Option 1: setting ‘mapred.job.priority’
<property>
<name>oozie.launcher.mapred.job.priority</name>
<value>${priority}</value>
</property>
<property>
<name>mapred.job.priority</name>
<value>${priority}</value>
</property>
Referring to Hadoop documentation on job priorities, http://hadoop.apache.org/common/docs/current/commands_manual.html#job
Where priority value can be one of VERY_HIGH | HIGH | NORMAL | LOW | VERY_LOW
Option 2: setting ‘mapred.job.queue.name’
Different queues may have pre-determined priorities on the Hadoop cluster. Leveraging that, you can configure to run your Oozie launcher and job on specified queues.
Are you interested in prioritizing Oozie workflow or coordinator jobs? In case of co-ordinator job specification, there is a <controls> block that can help order your execution priority beforehand.
<coordinator-app name="[NAME]" frequency="[FREQUENCY]"
start="[DATETIME]" end="[DATETIME]" timezone="[TIMEZONE]"
xmlns="uri:oozie:coordinator:0.1">
<controls>
<concurrency>[CONCURRENCY]</concurrency>
<execution>[EXECUTION_STRATEGY]</execution> <!-- value can be FIFO | LIFO | LAST_ONLY -->
The documentation for the above is here:
http://yahoo.github.com/oozie/releases/3.0.0/CoordinatorFunctionalSpec.html#a6.3._Synchronous_Coordinator_Application_Definition
Let me know if you have more questions.
Thanks,
Mona
On 3/9/12 5:51 AM, "Arsalan Bilal" <ch...@gmail.com> wrote:
Hi User,
I am using oozie to Manage my job. Now my requirement is to set job
priorities. i want to run job according to priorities.
if job fail then priority will be low, if all job come together , then it
should decide which job run first after seeing priorities and condition in
work-flow.
Any one Know , how can i set job priorities using oozie? Any informative
link or tutorial?