[jira] [Created] (OOZIE-2335) Create a Shard node

["Robert Kanter (JIRA)" <ji...@apache.org>]

Robert Kanter created OOZIE-2335:
------------------------------------

             Summary: Create a Shard node
                 Key: OOZIE-2335
                 URL: https://issues.apache.org/jira/browse/OOZIE-2335
             Project: Oozie
          Issue Type: New Feature
          Components: core
    Affects Versions: trunk
            Reporter: Robert Kanter


A Shard node has been discussed every so often, and it would be great to actually make it.  It's similar to a Fork node, but it multiplexes the same path instead of different paths.  For example, you can run the same action(s) multiple times.  

A config for concurrency would be good, to allow users to control how many concurrent multiplexed instances could run at the same time, similar to the concurrency control in a Coordinator.  This would also allow users to write loops (in a safe way, because the number of iterations is set to a number).

It would be good for a sharded path to support an EL function, say {{$\{shard:count()\}}}, which would return a the path count (in other words, each multiplexed instance of a sharded path would have it's own unique id).

We also probably need to have something similar to the ForkJoin checking that {{LiteWorkflowAppParser}} does.

I'm sure there's other features and things that would be good that I'm not thinking of right now.

It might be a good idea to take a look at [~tucu00]'s lite2 WiP patch at OOZIE-592, where he actually has a Shard/Unshard node.  I'm sure a lot of that can be brought in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)