You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Gaurav Aggarwal <ga...@linkedin.com.INVALID> on 2015/02/19 02:15:15 UTC
Issues with disabling ForkJoin Validation
Hi,
I have a workflow which fails ForkJoin validation in oozie but works fine if I set oozie.wf.validate.ForkJoin=false.
Is it safe to set oozie.wf.validate.ForkJoin=false ?
I want to understand potential runtime issues that can be created if I disable fork join validation.
Thanks,
Gaurav
Re: Issues with disabling ForkJoin Validation
Posted by Mona Chitnis <ch...@yahoo-inc.com.INVALID>.
Hi Gaurav,
That particular error - "E1100: Command precondition does not hold before execution," is most of the times harmless, as its only used in internal logic when to execute certain commands and when to fail-fast. It is _not_ harmless however, if your workflow is eventually failing.
For your other question, there are potentially very complex ways of defining different combinations of Fork-Join, and the Oozie validation is put in place to validate 90% of the cases. For the remaining, when users are confident about their fork-join structures and get consistently right outputs after disabling this validation, they should just go ahead with it. As Robert mentioned, it is an ongoing effort to implement handling for all the various complex combinations.
--Mona
On Thursday, February 19, 2015 4:25 PM, Gaurav Aggarwal <ga...@linkedin.com.INVALID> wrote:
Just noticed even though working.xml workflow works but I do get " E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100" in both working.xml and not-working.xml.
Thanks,
Gaurav
________________________________________
From: Gaurav Aggarwal
Sent: Thursday, February 19, 2015 2:21 PM
To: user@oozie.apache.org
Subject: RE: Issues with disabling ForkJoin Validation
Hi Robert,
Thanks for the pointing to OOZIE-1978, this will be helpful.
PFA my workflows. I have 2 workflows and both of them fails validation
with different error message :-
* not-workflow.xml fails due to "E0730: Fork/Joins not in pair². Node ³d²
never executes if I submit with disabled validation.
* working.xml which fails due to "E0742: No Fork for join (Join_f) to pair
with". This workflow works perfectly if I submit with disabled validation.
Can you please share more details regarding why do we need Fork/Join to be
in pair, not-workflow.xml have lesser Fork because of a 3-way fork at
starting. And why Oozie never executes node ³d² ?
Similarly why should we have paired Fork/Join in each path from start to
end node and how working.xml executes perfectly ? User¹s workflow may be
such that logically graph has unpaired fork/join ?
Thanks,
Gaurav
________________________________________
From: Robert Kanter [rkanter@cloudera.com]
Sent: Wednesday, February 18, 2015 6:33 PM
To: user@oozie.apache.org
Subject: Re: Issues with disabling ForkJoin Validation
Hi Gaurav,
IIRC, it should say why it's failing the ForkJoin validation. What does it
say?
There are two reasons why we allow disabling ForkJoin validation:
1. In case there's a bug in the ForkJoin validation code (it's very
complicated), there's a workaround to allow skipping it
2. In larger workflows, the ForkJoin validation code can take a very long
time to finish (on the order of hours or days), so there needs to be a way
to skip it. (OOZIE-1978 <https://issues.apache.org/jira/browse/OOZIE-1978>
is open to improve this)
As for whether or not it's safe to disable the validation, if the workflow
is working fine, then it's okay. The main purpose of the ForkJoin
validation is to fail incorrect workflows at submission time, rather than
during runtime. I believe that there are some technically incorrect
ForkJoin cases where Oozie can still work just fine, even though it's
wrong. You've either done that or found a bug in the ForkJoin validation
code :)
Can you post the reason and the workflow so I can see what it's complaining
about?
- Robert
On Wed, Feb 18, 2015 at 5:15 PM, Gaurav Aggarwal <
gaggarwal@linkedin.com.invalid> wrote:
> Hi,
>
> I have a workflow which fails ForkJoin validation in oozie but works fine
> if I set oozie.wf.validate.ForkJoin=false.
>
> Is it safe to set oozie.wf.validate.ForkJoin=false ?
>
> I want to understand potential runtime issues that can be created if I
> disable fork join validation.
>
> Thanks,
> Gaurav
>
RE: Issues with disabling ForkJoin Validation
Posted by Gaurav Aggarwal <ga...@linkedin.com.INVALID>.
Just noticed even though working.xml workflow works but I do get " E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100" in both working.xml and not-working.xml.
Thanks,
Gaurav
________________________________________
From: Gaurav Aggarwal
Sent: Thursday, February 19, 2015 2:21 PM
To: user@oozie.apache.org
Subject: RE: Issues with disabling ForkJoin Validation
Hi Robert,
Thanks for the pointing to OOZIE-1978, this will be helpful.
PFA my workflows. I have 2 workflows and both of them fails validation
with different error message :-
* not-workflow.xml fails due to "E0730: Fork/Joins not in pair². Node ³d²
never executes if I submit with disabled validation.
* working.xml which fails due to "E0742: No Fork for join (Join_f) to pair
with". This workflow works perfectly if I submit with disabled validation.
Can you please share more details regarding why do we need Fork/Join to be
in pair, not-workflow.xml have lesser Fork because of a 3-way fork at
starting. And why Oozie never executes node ³d² ?
Similarly why should we have paired Fork/Join in each path from start to
end node and how working.xml executes perfectly ? User¹s workflow may be
such that logically graph has unpaired fork/join ?
Thanks,
Gaurav
________________________________________
From: Robert Kanter [rkanter@cloudera.com]
Sent: Wednesday, February 18, 2015 6:33 PM
To: user@oozie.apache.org
Subject: Re: Issues with disabling ForkJoin Validation
Hi Gaurav,
IIRC, it should say why it's failing the ForkJoin validation. What does it
say?
There are two reasons why we allow disabling ForkJoin validation:
1. In case there's a bug in the ForkJoin validation code (it's very
complicated), there's a workaround to allow skipping it
2. In larger workflows, the ForkJoin validation code can take a very long
time to finish (on the order of hours or days), so there needs to be a way
to skip it. (OOZIE-1978 <https://issues.apache.org/jira/browse/OOZIE-1978>
is open to improve this)
As for whether or not it's safe to disable the validation, if the workflow
is working fine, then it's okay. The main purpose of the ForkJoin
validation is to fail incorrect workflows at submission time, rather than
during runtime. I believe that there are some technically incorrect
ForkJoin cases where Oozie can still work just fine, even though it's
wrong. You've either done that or found a bug in the ForkJoin validation
code :)
Can you post the reason and the workflow so I can see what it's complaining
about?
- Robert
On Wed, Feb 18, 2015 at 5:15 PM, Gaurav Aggarwal <
gaggarwal@linkedin.com.invalid> wrote:
> Hi,
>
> I have a workflow which fails ForkJoin validation in oozie but works fine
> if I set oozie.wf.validate.ForkJoin=false.
>
> Is it safe to set oozie.wf.validate.ForkJoin=false ?
>
> I want to understand potential runtime issues that can be created if I
> disable fork join validation.
>
> Thanks,
> Gaurav
>
RE: Issues with disabling ForkJoin Validation
Posted by Gaurav Aggarwal <ga...@linkedin.com.INVALID>.
Hi Robert,
Thanks for the pointing to OOZIE-1978, this will be helpful.
PFA my workflows. I have 2 workflows and both of them fails validation
with different error message :-
* not-workflow.xml fails due to "E0730: Fork/Joins not in pair². Node ³d²
never executes if I submit with disabled validation.
* working.xml which fails due to "E0742: No Fork for join (Join_f) to pair
with". This workflow works perfectly if I submit with disabled validation.
Can you please share more details regarding why do we need Fork/Join to be
in pair, not-workflow.xml have lesser Fork because of a 3-way fork at
starting. And why Oozie never executes node ³d² ?
Similarly why should we have paired Fork/Join in each path from start to
end node and how working.xml executes perfectly ? User¹s workflow may be
such that logically graph has unpaired fork/join ?
Thanks,
Gaurav
________________________________________
From: Robert Kanter [rkanter@cloudera.com]
Sent: Wednesday, February 18, 2015 6:33 PM
To: user@oozie.apache.org
Subject: Re: Issues with disabling ForkJoin Validation
Hi Gaurav,
IIRC, it should say why it's failing the ForkJoin validation. What does it
say?
There are two reasons why we allow disabling ForkJoin validation:
1. In case there's a bug in the ForkJoin validation code (it's very
complicated), there's a workaround to allow skipping it
2. In larger workflows, the ForkJoin validation code can take a very long
time to finish (on the order of hours or days), so there needs to be a way
to skip it. (OOZIE-1978 <https://issues.apache.org/jira/browse/OOZIE-1978>
is open to improve this)
As for whether or not it's safe to disable the validation, if the workflow
is working fine, then it's okay. The main purpose of the ForkJoin
validation is to fail incorrect workflows at submission time, rather than
during runtime. I believe that there are some technically incorrect
ForkJoin cases where Oozie can still work just fine, even though it's
wrong. You've either done that or found a bug in the ForkJoin validation
code :)
Can you post the reason and the workflow so I can see what it's complaining
about?
- Robert
On Wed, Feb 18, 2015 at 5:15 PM, Gaurav Aggarwal <
gaggarwal@linkedin.com.invalid> wrote:
> Hi,
>
> I have a workflow which fails ForkJoin validation in oozie but works fine
> if I set oozie.wf.validate.ForkJoin=false.
>
> Is it safe to set oozie.wf.validate.ForkJoin=false ?
>
> I want to understand potential runtime issues that can be created if I
> disable fork join validation.
>
> Thanks,
> Gaurav
>
Re: Issues with disabling ForkJoin Validation
Posted by Robert Kanter <rk...@cloudera.com>.
Hi Gaurav,
IIRC, it should say why it's failing the ForkJoin validation. What does it
say?
There are two reasons why we allow disabling ForkJoin validation:
1. In case there's a bug in the ForkJoin validation code (it's very
complicated), there's a workaround to allow skipping it
2. In larger workflows, the ForkJoin validation code can take a very long
time to finish (on the order of hours or days), so there needs to be a way
to skip it. (OOZIE-1978 <https://issues.apache.org/jira/browse/OOZIE-1978>
is open to improve this)
As for whether or not it's safe to disable the validation, if the workflow
is working fine, then it's okay. The main purpose of the ForkJoin
validation is to fail incorrect workflows at submission time, rather than
during runtime. I believe that there are some technically incorrect
ForkJoin cases where Oozie can still work just fine, even though it's
wrong. You've either done that or found a bug in the ForkJoin validation
code :)
Can you post the reason and the workflow so I can see what it's complaining
about?
- Robert
On Wed, Feb 18, 2015 at 5:15 PM, Gaurav Aggarwal <
gaggarwal@linkedin.com.invalid> wrote:
> Hi,
>
> I have a workflow which fails ForkJoin validation in oozie but works fine
> if I set oozie.wf.validate.ForkJoin=false.
>
> Is it safe to set oozie.wf.validate.ForkJoin=false ?
>
> I want to understand potential runtime issues that can be created if I
> disable fork join validation.
>
> Thanks,
> Gaurav
>