You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Etienne Dumoulin <et...@idiro.com> on 2012/11/16 17:40:47 UTC
Nested Fork and Join & how to make an error silent
All,
I am trying to create an Oozie xml file from a DAG data structure.
My data structure (and actions to run) would look like that:
start----->A1,A2
A1----> A7
A2----->A7,A3,A4
A3----->A6
A7,A6,A4---->end
Now if the actions output are only temporary, I would like to delete them
asap,
I name D the delete actions:
start----->A1,A2
A1----> A7
A2----->A7,A3,A4
A3----->A6,[A3,A4,A7]
A7----->D1,[A3,A4,A7]
A4----->[A3,A4,A7]
A3,A4,A7---->D2
A6 ----> D3
D1----->join_end
D2------>join_end
D3------>join_end
join_end ----->end
I have two questions on forks/joins:
The documentation says that I need to have fork and join in pair, is that
still true?
Or is it only better, I just read a post from Virag the 19th of July:
"To me, the nested forks option you are considering looks good. Its also
better to have the join in pair."
I would like that my workflow finishes even through one branch fails, let
say that
A7 fails, I would like that A6 and A4 proceed. For this type of behaviour
can
I link all my error transitions to join_end?
It will not be possible if I have to pair forks and joins.
Regards,
Étienne
Re: Nested Fork and Join & how to make an error silent
Posted by Etienne Dumoulin <et...@idiro.com>.
Hi Robert,
Thank you for your reply it is of great help. I will finally go for killing
my
workflow if there is an error and implement the fork by pair. I cannot
afford
to change the oozie installation parameter.
I don't think I can use the copy-paste technique her, I use join because A7
has
two dependencies that needs to be respected, sorry for the clumsy
explanation
on the first post.
Thanks again,
Étienne
On 19 November 2012 17:23, Robert Kanter <rk...@cloudera.com> wrote:
> Hi Etienne,
>
> I'm not sure what will happen if all of your error transitions go to
> join_end. Alternatively, I know its not as "clean" and is more work, but
> you can usually make most workflows follow the restrictions on fork/join by
> duplicating (i.e. copy-paste) some of your actions and changing their ok
> and error transitions. For example, it looks like you have multiple ways
> of going to A7, you could copy A7 so that each way has it own copy; this
> would make it easier to use the joins.
>
> The oozie.validate.ForkJoin property goes in oozie-site.xml, and disabled
> fork/join checking for all of your workflows. You may have to restart
> Oozie for it to take effect.
>
> - Robert
>
>
>
>
> On Sat, Nov 17, 2012 at 12:47 AM, Etienne Dumoulin <
> etienne.dumoulin@idiro.com> wrote:
>
> > Hi Robert,
> >
> > Thank you for your reply.
> > The brackets indicate joins, when one action has several dependencies.
> > For example after forking in order to delete I need to wait that all the
> > direct
> > actions are finished.
> >
> > In the above example, if all my error transitions go to join_end, does
> the
> > workflow finished, or is it in
> > a dead lock?
> >
> > Can I specify the oozie.validate.ForkJoin to false in my job.properties
> > file, or
> > when I start the job in java?
> >
> > Thanks,
> >
> > Étienne
> >
> >
> > On 16 November 2012 21:59, Robert Kanter <rk...@cloudera.com> wrote:
> >
> > > Hi Etienne,
> > >
> > >
> > > I'm not sure I follow exactly your notation; what do the [ and ]
> brackets
> > > indicate?
> > >
> > >
> > > You can have multiple joins in your workflow, so instead of having
> > > everything go to join_end, you can have a hierarchy of joins such that
> > each
> > > fork in your workflow is in pair with a join. There is likely a way to
> > > re-work your workflow to be like this.
> > >
> > >
> > > Oozie does some extra checking when you use a fork (e.g. fork and join
> in
> > > pair, etc). I'm pretty sure that this is the only place where Oozie
> will
> > > enforce these restrictions. In other words, if Oozie were to actually
> > > start executing your workflow, it wouldn't complain if they're not in a
> > > pair. You can disable this extra checking by setting
> > > oozie.validate.ForkJoin to false in your oozie-site.xml. This may be
> > risky
> > > though and something could go wrong, so do not do this in a production
> > > cluster.
> > >
> > >
> > > - Robert
> > >
> > >
> > > On Fri, Nov 16, 2012 at 8:40 AM, Etienne Dumoulin <
> > > etienne.dumoulin@idiro.com> wrote:
> > >
> > > > All,
> > > >
> > > > I am trying to create an Oozie xml file from a DAG data structure.
> > > >
> > > > My data structure (and actions to run) would look like that:
> > > > start----->A1,A2
> > > > A1----> A7
> > > > A2----->A7,A3,A4
> > > > A3----->A6
> > > > A7,A6,A4---->end
> > > >
> > > >
> > > > Now if the actions output are only temporary, I would like to delete
> > them
> > > > asap,
> > > > I name D the delete actions:
> > > > start----->A1,A2
> > > > A1----> A7
> > > > A2----->A7,A3,A4
> > > > A3----->A6,[A3,A4,A7]
> > > > A7----->D1,[A3,A4,A7]
> > > > A4----->[A3,A4,A7]
> > > > A3,A4,A7---->D2
> > > > A6 ----> D3
> > > > D1----->join_end
> > > > D2------>join_end
> > > > D3------>join_end
> > > > join_end ----->end
> > > >
> > > > I have two questions on forks/joins:
> > > > The documentation says that I need to have fork and join in pair, is
> > that
> > > > still true?
> > > > Or is it only better, I just read a post from Virag the 19th of July:
> > > > "To me, the nested forks option you are considering looks good. Its
> > also
> > > > better to have the join in pair."
> > > >
> > > > I would like that my workflow finishes even through one branch fails,
> > let
> > > > say that
> > > > A7 fails, I would like that A6 and A4 proceed. For this type of
> > behaviour
> > > > can
> > > > I link all my error transitions to join_end?
> > > > It will not be possible if I have to pair forks and joins.
> > > >
> > > > Regards,
> > > >
> > > > Étienne
> > > >
> > >
> > <et...@idiro.com>
> >
>
--
Étienne Dumoulin
Head of product development
Idiro Technologies
Clarendon House,
34-37 Clarendon St,
Dublin 2
Ireland
M: +353 (0)17030410
Email: etienne.dumoulin@idiro.com
Re: Nested Fork and Join & how to make an error silent
Posted by Robert Kanter <rk...@cloudera.com>.
Hi Etienne,
I'm not sure what will happen if all of your error transitions go to
join_end. Alternatively, I know its not as "clean" and is more work, but
you can usually make most workflows follow the restrictions on fork/join by
duplicating (i.e. copy-paste) some of your actions and changing their ok
and error transitions. For example, it looks like you have multiple ways
of going to A7, you could copy A7 so that each way has it own copy; this
would make it easier to use the joins.
The oozie.validate.ForkJoin property goes in oozie-site.xml, and disabled
fork/join checking for all of your workflows. You may have to restart
Oozie for it to take effect.
- Robert
On Sat, Nov 17, 2012 at 12:47 AM, Etienne Dumoulin <
etienne.dumoulin@idiro.com> wrote:
> Hi Robert,
>
> Thank you for your reply.
> The brackets indicate joins, when one action has several dependencies.
> For example after forking in order to delete I need to wait that all the
> direct
> actions are finished.
>
> In the above example, if all my error transitions go to join_end, does the
> workflow finished, or is it in
> a dead lock?
>
> Can I specify the oozie.validate.ForkJoin to false in my job.properties
> file, or
> when I start the job in java?
>
> Thanks,
>
> Étienne
>
>
> On 16 November 2012 21:59, Robert Kanter <rk...@cloudera.com> wrote:
>
> > Hi Etienne,
> >
> >
> > I'm not sure I follow exactly your notation; what do the [ and ] brackets
> > indicate?
> >
> >
> > You can have multiple joins in your workflow, so instead of having
> > everything go to join_end, you can have a hierarchy of joins such that
> each
> > fork in your workflow is in pair with a join. There is likely a way to
> > re-work your workflow to be like this.
> >
> >
> > Oozie does some extra checking when you use a fork (e.g. fork and join in
> > pair, etc). I'm pretty sure that this is the only place where Oozie will
> > enforce these restrictions. In other words, if Oozie were to actually
> > start executing your workflow, it wouldn't complain if they're not in a
> > pair. You can disable this extra checking by setting
> > oozie.validate.ForkJoin to false in your oozie-site.xml. This may be
> risky
> > though and something could go wrong, so do not do this in a production
> > cluster.
> >
> >
> > - Robert
> >
> >
> > On Fri, Nov 16, 2012 at 8:40 AM, Etienne Dumoulin <
> > etienne.dumoulin@idiro.com> wrote:
> >
> > > All,
> > >
> > > I am trying to create an Oozie xml file from a DAG data structure.
> > >
> > > My data structure (and actions to run) would look like that:
> > > start----->A1,A2
> > > A1----> A7
> > > A2----->A7,A3,A4
> > > A3----->A6
> > > A7,A6,A4---->end
> > >
> > >
> > > Now if the actions output are only temporary, I would like to delete
> them
> > > asap,
> > > I name D the delete actions:
> > > start----->A1,A2
> > > A1----> A7
> > > A2----->A7,A3,A4
> > > A3----->A6,[A3,A4,A7]
> > > A7----->D1,[A3,A4,A7]
> > > A4----->[A3,A4,A7]
> > > A3,A4,A7---->D2
> > > A6 ----> D3
> > > D1----->join_end
> > > D2------>join_end
> > > D3------>join_end
> > > join_end ----->end
> > >
> > > I have two questions on forks/joins:
> > > The documentation says that I need to have fork and join in pair, is
> that
> > > still true?
> > > Or is it only better, I just read a post from Virag the 19th of July:
> > > "To me, the nested forks option you are considering looks good. Its
> also
> > > better to have the join in pair."
> > >
> > > I would like that my workflow finishes even through one branch fails,
> let
> > > say that
> > > A7 fails, I would like that A6 and A4 proceed. For this type of
> behaviour
> > > can
> > > I link all my error transitions to join_end?
> > > It will not be possible if I have to pair forks and joins.
> > >
> > > Regards,
> > >
> > > Étienne
> > >
> >
> <et...@idiro.com>
>
Re: Nested Fork and Join & how to make an error silent
Posted by Etienne Dumoulin <et...@idiro.com>.
Hi Robert,
Thank you for your reply.
The brackets indicate joins, when one action has several dependencies.
For example after forking in order to delete I need to wait that all the
direct
actions are finished.
In the above example, if all my error transitions go to join_end, does the
workflow finished, or is it in
a dead lock?
Can I specify the oozie.validate.ForkJoin to false in my job.properties
file, or
when I start the job in java?
Thanks,
Étienne
On 16 November 2012 21:59, Robert Kanter <rk...@cloudera.com> wrote:
> Hi Etienne,
>
>
> I'm not sure I follow exactly your notation; what do the [ and ] brackets
> indicate?
>
>
> You can have multiple joins in your workflow, so instead of having
> everything go to join_end, you can have a hierarchy of joins such that each
> fork in your workflow is in pair with a join. There is likely a way to
> re-work your workflow to be like this.
>
>
> Oozie does some extra checking when you use a fork (e.g. fork and join in
> pair, etc). I'm pretty sure that this is the only place where Oozie will
> enforce these restrictions. In other words, if Oozie were to actually
> start executing your workflow, it wouldn't complain if they're not in a
> pair. You can disable this extra checking by setting
> oozie.validate.ForkJoin to false in your oozie-site.xml. This may be risky
> though and something could go wrong, so do not do this in a production
> cluster.
>
>
> - Robert
>
>
> On Fri, Nov 16, 2012 at 8:40 AM, Etienne Dumoulin <
> etienne.dumoulin@idiro.com> wrote:
>
> > All,
> >
> > I am trying to create an Oozie xml file from a DAG data structure.
> >
> > My data structure (and actions to run) would look like that:
> > start----->A1,A2
> > A1----> A7
> > A2----->A7,A3,A4
> > A3----->A6
> > A7,A6,A4---->end
> >
> >
> > Now if the actions output are only temporary, I would like to delete them
> > asap,
> > I name D the delete actions:
> > start----->A1,A2
> > A1----> A7
> > A2----->A7,A3,A4
> > A3----->A6,[A3,A4,A7]
> > A7----->D1,[A3,A4,A7]
> > A4----->[A3,A4,A7]
> > A3,A4,A7---->D2
> > A6 ----> D3
> > D1----->join_end
> > D2------>join_end
> > D3------>join_end
> > join_end ----->end
> >
> > I have two questions on forks/joins:
> > The documentation says that I need to have fork and join in pair, is that
> > still true?
> > Or is it only better, I just read a post from Virag the 19th of July:
> > "To me, the nested forks option you are considering looks good. Its also
> > better to have the join in pair."
> >
> > I would like that my workflow finishes even through one branch fails, let
> > say that
> > A7 fails, I would like that A6 and A4 proceed. For this type of behaviour
> > can
> > I link all my error transitions to join_end?
> > It will not be possible if I have to pair forks and joins.
> >
> > Regards,
> >
> > Étienne
> >
>
<et...@idiro.com>
Re: Nested Fork and Join & how to make an error silent
Posted by Robert Kanter <rk...@cloudera.com>.
Hi Etienne,
I'm not sure I follow exactly your notation; what do the [ and ] brackets
indicate?
You can have multiple joins in your workflow, so instead of having
everything go to join_end, you can have a hierarchy of joins such that each
fork in your workflow is in pair with a join. There is likely a way to
re-work your workflow to be like this.
Oozie does some extra checking when you use a fork (e.g. fork and join in
pair, etc). I'm pretty sure that this is the only place where Oozie will
enforce these restrictions. In other words, if Oozie were to actually
start executing your workflow, it wouldn't complain if they're not in a
pair. You can disable this extra checking by setting
oozie.validate.ForkJoin to false in your oozie-site.xml. This may be risky
though and something could go wrong, so do not do this in a production
cluster.
- Robert
On Fri, Nov 16, 2012 at 8:40 AM, Etienne Dumoulin <
etienne.dumoulin@idiro.com> wrote:
> All,
>
> I am trying to create an Oozie xml file from a DAG data structure.
>
> My data structure (and actions to run) would look like that:
> start----->A1,A2
> A1----> A7
> A2----->A7,A3,A4
> A3----->A6
> A7,A6,A4---->end
>
>
> Now if the actions output are only temporary, I would like to delete them
> asap,
> I name D the delete actions:
> start----->A1,A2
> A1----> A7
> A2----->A7,A3,A4
> A3----->A6,[A3,A4,A7]
> A7----->D1,[A3,A4,A7]
> A4----->[A3,A4,A7]
> A3,A4,A7---->D2
> A6 ----> D3
> D1----->join_end
> D2------>join_end
> D3------>join_end
> join_end ----->end
>
> I have two questions on forks/joins:
> The documentation says that I need to have fork and join in pair, is that
> still true?
> Or is it only better, I just read a post from Virag the 19th of July:
> "To me, the nested forks option you are considering looks good. Its also
> better to have the join in pair."
>
> I would like that my workflow finishes even through one branch fails, let
> say that
> A7 fails, I would like that A6 and A4 proceed. For this type of behaviour
> can
> I link all my error transitions to join_end?
> It will not be possible if I have to pair forks and joins.
>
> Regards,
>
> Étienne
>