You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Florin Dinu <fl...@cs.rice.edu> on 2012/07/17 05:45:42 UTC
Does Oozie work under compute node failures?
Hello,
I am trying to use Oozie to run multi-stage Hadoop jobs.
I have written a 5-stage chain job (one job takes as input the output
of the previous one) and it works well under normal conditions.
The reason I am using Oozie is that I hope it can manage the
re-computations that can appear under compute node failures.
So here's what I did today. When my job reached the 4th stage,
I killed the TaskTracker and DataNode on one of the machines.
An important detail is that I need to set the job output to be
replicated only once (dfs.replication=1).
My hope was that in this situation Oozie would be able to handle
the re-computation even if it cascaded all the way to the beginning
of the job. For example, I was hoping that once it detected the failure,
Oozie would realize that some of the blocks are lost and would trigger
a re-run of the necessary parts of previous jobs to re-compute those
blocks that are missing. What I see is the opposite. The job fails
because the re-execution of some mappers is attempted too many times
(the mapred.map.max.attempts limit is reached),but
Oozie does not react in any way to the failures I injected. It does
nothing more than let Hadoop manage this the best way it can.
Let me give an example. Assume M1 is a local mapper that is running
on the TaskTracker that I fail. I also fail the DataNode on the same
node. Since there is only one replica for each data block, M1 cannot
be re-executed no matter how many attempts Hadoop makes. The input
for M1 is lost. After detecting the failures, I was hoping Oozie would
understand that to compute M1 its input block needs to be re-generated.
For this I was hoping Oozie would restart necessary mappers and
reducers from previous jobs to re-compute the input block for M1. Of
course, restarting the entire previous job is a solution but is not the
most efficient. In the general case the re-computation may cascade back
multiple stages.
Can Oozie help me here?
If not is there a way to manage these re-computations automatically?
Can Cascading to this (I am not familiar with it as of yet)?
Thank you so much
Florin
Re: Does Oozie work under compute node failures?
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Florin,
As Alejandro mentioned, Oozie can't help much in this use-case.
On a related note, you can configure oozie WF to *auto-retry* any action from oozie level.
You can also configure the intra-retry interval. More details at: http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctionalSpec.html#a18_User-Retry_for_Workflow_Actions_since_Oozie_3.1
Regards,
Mohammad
________________________________
From: Alejandro Abdelnur <tu...@cloudera.com>
To: oozie-users@incubator.apache.org
Sent: Monday, July 16, 2012 8:58 PM
Subject: Re: Does Oozie work under compute node failures?
Florin,
I'm afraid Oozie won't help you in your scenario.
Once Oozie submits a job successfully to Hadoop (Oozie retries a few times
if the submission fails due to transient errors), Oozie assumes Hadoop will
do all the recovery of the job.
Thanks.
On Mon, Jul 16, 2012 at 8:45 PM, Florin Dinu <fl...@cs.rice.edu>wrote:
> Hello,
>
> I am trying to use Oozie to run multi-stage Hadoop jobs.
> I have written a 5-stage chain job (one job takes as input the output
> of the previous one) and it works well under normal conditions.
>
> The reason I am using Oozie is that I hope it can manage the
> re-computations that can appear under compute node failures.
>
> So here's what I did today. When my job reached the 4th stage,
> I killed the TaskTracker and DataNode on one of the machines.
> An important detail is that I need to set the job output to be
> replicated only once (dfs.replication=1).
>
> My hope was that in this situation Oozie would be able to handle
> the re-computation even if it cascaded all the way to the beginning
> of the job. For example, I was hoping that once it detected the failure,
> Oozie would realize that some of the blocks are lost and would trigger
> a re-run of the necessary parts of previous jobs to re-compute those
> blocks that are missing. What I see is the opposite. The job fails
> because the re-execution of some mappers is attempted too many times
> (the mapred.map.max.attempts limit is reached),but
> Oozie does not react in any way to the failures I injected. It does
> nothing more than let Hadoop manage this the best way it can.
>
> Let me give an example. Assume M1 is a local mapper that is running
> on the TaskTracker that I fail. I also fail the DataNode on the same
> node. Since there is only one replica for each data block, M1 cannot
> be re-executed no matter how many attempts Hadoop makes. The input
> for M1 is lost. After detecting the failures, I was hoping Oozie would
> understand that to compute M1 its input block needs to be re-generated.
> For this I was hoping Oozie would restart necessary mappers and
> reducers from previous jobs to re-compute the input block for M1. Of
> course, restarting the entire previous job is a solution but is not the
> most efficient. In the general case the re-computation may cascade back
> multiple stages.
>
> Can Oozie help me here?
> If not is there a way to manage these re-computations automatically?
> Can Cascading to this (I am not familiar with it as of yet)?
>
>
> Thank you so much
> Florin
>
>
>
>
>
>
>
>
--
Alejandro
Re: Does Oozie work under compute node failures?
Posted by Florin Dinu <fd...@rice.edu>.
Alejandro,
Thank you for the quick answer.
I see :|. Is there some way to do what I want automatically
so that I don't have to re-implement it myself in Hadoop?
Florin
On Mon, 16 Jul 2012 20:58:16 -0700
Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Florin,
>
> I'm afraid Oozie won't help you in your scenario.
>
> Once Oozie submits a job successfully to Hadoop (Oozie retries a few
> times if the submission fails due to transient errors), Oozie assumes
> Hadoop will do all the recovery of the job.
>
> Thanks.
>
>
> On Mon, Jul 16, 2012 at 8:45 PM, Florin Dinu
> <fl...@cs.rice.edu>wrote:
>
> > Hello,
> >
> > I am trying to use Oozie to run multi-stage Hadoop jobs.
> > I have written a 5-stage chain job (one job takes as input the
> > output of the previous one) and it works well under normal
> > conditions.
> >
> > The reason I am using Oozie is that I hope it can manage the
> > re-computations that can appear under compute node failures.
> >
> > So here's what I did today. When my job reached the 4th stage,
> > I killed the TaskTracker and DataNode on one of the machines.
> > An important detail is that I need to set the job output to be
> > replicated only once (dfs.replication=1).
> >
> > My hope was that in this situation Oozie would be able to handle
> > the re-computation even if it cascaded all the way to the beginning
> > of the job. For example, I was hoping that once it detected the
> > failure, Oozie would realize that some of the blocks are lost and
> > would trigger a re-run of the necessary parts of previous jobs to
> > re-compute those blocks that are missing. What I see is the
> > opposite. The job fails because the re-execution of some mappers is
> > attempted too many times (the mapred.map.max.attempts limit is
> > reached),but Oozie does not react in any way to the failures I
> > injected. It does nothing more than let Hadoop manage this the best
> > way it can.
> >
> > Let me give an example. Assume M1 is a local mapper that is running
> > on the TaskTracker that I fail. I also fail the DataNode on the same
> > node. Since there is only one replica for each data block, M1 cannot
> > be re-executed no matter how many attempts Hadoop makes. The input
> > for M1 is lost. After detecting the failures, I was hoping Oozie
> > would understand that to compute M1 its input block needs to be
> > re-generated. For this I was hoping Oozie would restart necessary
> > mappers and reducers from previous jobs to re-compute the input
> > block for M1. Of course, restarting the entire previous job is a
> > solution but is not the most efficient. In the general case the
> > re-computation may cascade back multiple stages.
> >
> > Can Oozie help me here?
> > If not is there a way to manage these re-computations automatically?
> > Can Cascading to this (I am not familiar with it as of yet)?
> >
> >
> > Thank you so much
> > Florin
> >
> >
> >
> >
> >
> >
> >
> >
>
>
Re: Does Oozie work under compute node failures?
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Florin,
I'm afraid Oozie won't help you in your scenario.
Once Oozie submits a job successfully to Hadoop (Oozie retries a few times
if the submission fails due to transient errors), Oozie assumes Hadoop will
do all the recovery of the job.
Thanks.
On Mon, Jul 16, 2012 at 8:45 PM, Florin Dinu <fl...@cs.rice.edu>wrote:
> Hello,
>
> I am trying to use Oozie to run multi-stage Hadoop jobs.
> I have written a 5-stage chain job (one job takes as input the output
> of the previous one) and it works well under normal conditions.
>
> The reason I am using Oozie is that I hope it can manage the
> re-computations that can appear under compute node failures.
>
> So here's what I did today. When my job reached the 4th stage,
> I killed the TaskTracker and DataNode on one of the machines.
> An important detail is that I need to set the job output to be
> replicated only once (dfs.replication=1).
>
> My hope was that in this situation Oozie would be able to handle
> the re-computation even if it cascaded all the way to the beginning
> of the job. For example, I was hoping that once it detected the failure,
> Oozie would realize that some of the blocks are lost and would trigger
> a re-run of the necessary parts of previous jobs to re-compute those
> blocks that are missing. What I see is the opposite. The job fails
> because the re-execution of some mappers is attempted too many times
> (the mapred.map.max.attempts limit is reached),but
> Oozie does not react in any way to the failures I injected. It does
> nothing more than let Hadoop manage this the best way it can.
>
> Let me give an example. Assume M1 is a local mapper that is running
> on the TaskTracker that I fail. I also fail the DataNode on the same
> node. Since there is only one replica for each data block, M1 cannot
> be re-executed no matter how many attempts Hadoop makes. The input
> for M1 is lost. After detecting the failures, I was hoping Oozie would
> understand that to compute M1 its input block needs to be re-generated.
> For this I was hoping Oozie would restart necessary mappers and
> reducers from previous jobs to re-compute the input block for M1. Of
> course, restarting the entire previous job is a solution but is not the
> most efficient. In the general case the re-computation may cascade back
> multiple stages.
>
> Can Oozie help me here?
> If not is there a way to manage these re-computations automatically?
> Can Cascading to this (I am not familiar with it as of yet)?
>
>
> Thank you so much
> Florin
>
>
>
>
>
>
>
>
--
Alejandro