You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Harsh J (JIRA)" <ji...@apache.org> on 2012/06/11 06:13:42 UTC
[jira] [Created] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Harsh J created OOZIE-865:
-----------------------------
Summary: ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
Key: OOZIE-865
URL: https://issues.apache.org/jira/browse/OOZIE-865
Project: Oozie
Issue Type: Bug
Reporter: Harsh J
Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Virag Kothari reassigned OOZIE-865:
-----------------------------------
Assignee: Robert Kanter
Correct
Assigning to you :)
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413964#comment-13413964 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
JBPM does not recommend unequal fork/join pairs. The current fork/join validation follows it. Also, if we allow unequal fork/join pairing, then we need to account for other cases. So I think fork/join should always be in pair.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated OOZIE-865:
--------------------------------
Attachment: OOZIE-865.patch
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: OOZIE-865.patch, workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413099#comment-13413099 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
Yes, I agree. Ordering of the nodes should not make a difference and thats a bug
But should the decision node directly be allowed to evaluate to join?
For e.g. (simplified version of the attached workflow)
fork-> decision, action1
action1-> join
decision -> action2, join
action2 -> join
join -> action3
Instead of this, shouldn't the decision node be out of fork?
decision -> action3, fork
fork -> (action1, action2)
action1 -> join
action2 -> join
join ->action3
The above looks cleaner and better design to me
Basically, if the decision goes to join directly, then what is the use of having the decision node within fork itself?
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292891#comment-13292891 ]
Harsh J commented on OOZIE-865:
-------------------------------
Indeed, my comment seems to be ambiguous, sorry. The core of the issue is that the fork join validator fails if the fork-to nodes are decision nodes, the fork join validator simply fails.
I believe it fails due to mismatch in lengths of each side of the list (that seems to be the only check) but I haven't looked deeper into how they're populated.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413365#comment-13413365 ]
Robert Kanter commented on OOZIE-865:
-------------------------------------
While I think it makes sense to allow going to a kill node during a fork path (from a decision node or otherwise), it wouldn't make sense to allow going to an end node, right? (If a forked path wants to "end" it should just go to a join node.)
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414200#comment-13414200 ]
Robert Kanter commented on OOZIE-865:
-------------------------------------
That's good; verifying unequal fork/join pairs would probably get really complicated.
(just to clarify: the patch I uploaded does not allow unequal fork/join pairs)
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: OOZIE-865.patch, workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413505#comment-13413505 ]
Alejandro Abdelnur commented on OOZIE-865:
------------------------------------------
mmhh, I think I disagree with allowing going to end, end is 'graceful' end, going from within a forked path to end is not 'graceful' as you'd be killing actions in other paths of the fork. Does this make sense?
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413386#comment-13413386 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
After discussion with Mohammad, I think we should allow both 'kill' and 'end'. Basically think of them as terminal states and the workflow should be allowed to jump directly to these states.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292887#comment-13292887 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
In that case, the wf should have a single fork to 4 decision nodes and each of them eventually ending in a single join.
In general, the number of forks and joins should be equal even with decision nodes as a decision node will evaluate to an action node.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413954#comment-13413954 ]
Robert Kanter commented on OOZIE-865:
-------------------------------------
Do we want to allow something like this?
fork -> action1, action2, action3
action1 -> join2
action2 -> join2
join2 -> action4
action3 -> join1
action4 -> join1
join1 -> end
In other words, allowing an earlier join of some of the forked paths (as long as that joined path eventually goes to the correct join later)
Or should we just say that you have to do the equivalent using fork-join pairs?
fork1 -> fork2, action3
fork2 -> action1, action2
action1 -> join2
action2 -> join2
join2 -> action4
action3 -> join1
action4 -> join1
join1 -> end
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413851#comment-13413851 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
Ok..makes sense
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated OOZIE-865:
--------------------------------
Attachment: workflow.xml
I did some testing with the sample workflow I just attached. It does a fork where each path goes to a decision and each decision can go to the join node directly or to an action and then the join node. The validation code blocks this even though it actually runs correctly.
If you slightly modify the fork in the workflow to go to "path-2-action" instead of "path-2-decision" (but leaving "path-1-decision" as is), then the validation code will actually allow this even though there is a decision that goes directly to the join. But, if you instead modify the fork to go to "path-1-action" instead of "path-1-decision" (but leave "path-2-decision" as is), then the validation code will not allow this. So, the ordering of the fork paths makes a difference, which is probably not a good thing...
It looks like the problem is in LiteWorkflowAppParser.validateFork() where it checks if the transitions list contains each of the transitions from the decision node and throws an exception. Perhaps adding a check for "joinNode" here would fix the problem.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292900#comment-13292900 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
I know some cases when it fails..when one of the paths in decision node is a kill node or if a path in decision node evaluates directly to a join. It would be great if you can attach the fork/join graph from the failed workflow.
However in normal scenarios, I think decision nodes are handled within fork-join. The test case for this is at: Line 434: TestLiteWorkflowAppParser
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413182#comment-13413182 ]
Robert Kanter commented on OOZIE-865:
-------------------------------------
In your first example, the possible execution paths are:
{code}
fork -> action1 -> join -> action3
-> action2 ->
{code}
and
{code}
fork -> action1 -> join -> action3
-> ->
{code}
But in the second example, they are:
{code}
fork -> action1 -> join -> action3
-> action2 ->
{code}
and
{code}
action3
{code}
They aren't entirely equivalent. To fix the second example, the decision node should be "decision -> action1, fork", but I'm not sure that is allowed (or if it is, if it should actually be allowed) because if the decision node decides to take action1 and not fork, then action1 will go to join without having been forked first. I think to make an equivalent example, we'd have to have "decision -> action4, fork" where action4 is exactly the same as action1 except that it goes to action3 instead of join. But having duplicate nodes isn't good either.
Even if there is a way to re-write every possible workflow to an equivalent one without a decision going to a join, it seems to me that we should try to make it as flexible as possible. Is there a specific reason why we shouldn't allow this?
Besides, you can easily emulate this by having the decision node go to a no-op action that then goes to the join. So, people determined to write a workflow with a decision node that goes to a join would still be able to do so anyway.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414198#comment-13414198 ]
Alejandro Abdelnur commented on OOZIE-865:
------------------------------------------
Agree with virag. forks/joins MUST always come in pairs.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: OOZIE-865.patch, workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413305#comment-13413305 ]
Robert Kanter commented on OOZIE-865:
-------------------------------------
Sure, I can take this.
For the decision node going to a kill node, you're talking about when the decision node is in a forked path, right?
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated OOZIE-865:
-------------------------------------
Affects Version/s: 3.2.0
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413275#comment-13413275 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
Robert, you are right. To make it equivalent, it should decision->action1, fork. Though that looks cleaner to me, I agree we should be flexible.
Then we should allow decision node directly going to join. Also, I saw some wf's where one of the paths in decision node is kill. So, perhaps we should have a check for kill node too.
Do you want to take this patch?
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412038#comment-13412038 ]
Robert Kanter commented on OOZIE-865:
-------------------------------------
Virag, did you have a solution already for this? Otherwise, I can start looking where you mentioned in TestLiteWorkflowAppParser.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412059#comment-13412059 ]
Virag Kothari commented on OOZIE-865:
-------------------------------------
As I mentioned in my previous comment, can someone attach the failing fork/join graph when there are decision nodes. In some cases it fails, but not in all.
There are some use cases I have seen like:
Within a fork node, one of the paths in decision node goes to kill without going to join. The current validation doesn't allow it.
I want to know if there are similar issues you have seen. Then, we can design the type of fork/join structure Oozie should allow
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413191#comment-13413191 ]
Alejandro Abdelnur commented on OOZIE-865:
------------------------------------------
We should keep in mind that the validation is to verify we'll formed WFs, even if they could be done in a different/better/cleaner way.
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Attachments: workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-865) ForkJoin validator checks total
lengths of forks vs. joins instead of actual paths
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414202#comment-13414202 ]
Alejandro Abdelnur commented on OOZIE-865:
------------------------------------------
+1
> ForkJoin validator checks total lengths of forks vs. joins instead of actual paths
> ----------------------------------------------------------------------------------
>
> Key: OOZIE-865
> URL: https://issues.apache.org/jira/browse/OOZIE-865
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 3.2.0
> Reporter: Harsh J
> Assignee: Robert Kanter
> Attachments: OOZIE-865.patch, workflow.xml
>
>
> Consider a WF that has four fork paths, each to a decision node, and each of these eventually in their further paths end at a single join node (thereby resulting in a valid DAG).
> When such a WF is passed to Oozie and fork join validator is enabled, the validation fails cause the numForks(4) > numJoins(1). This naive way appears to be wrong to compare, and we should ideally only compare true path based forks->joins lists, if possible.
> This causes a regression if the fork join validation is left enabled. Workaround for such workflows currently is to disable fork join validation via {{oozie.validate.ForkJoin}} set to {{false}} at the server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira