[jira] [Commented] (OOZIE-1976) Specifying coordinator input datasets in more logical ways

["Purshotam Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101651#comment-14101651 ] 

Purshotam Shah commented on OOZIE-1976:
---------------------------------------

Looks good.
Is that possible to add wait time? Time will be offset from nominal time. For catchup jobs wait time can be ignored.

Few times user might want to wait for some time and want to process what ever input data are available.

1.Ex. <datain name=”DS1” dataset=”DS1” min=3 wait="10m">
This will wait for 10 min ( or all dependencies are match) and coord action will start running after 10 min, if input set >=3 is available.


2. Ex. <datain name=”DS1” dataset=”DS1” wait="10m">
This will wait for 10 min and process whatever data input dataset is  available.

3. Ex. <datain name=”DS1” dataset=”DS1” wait="30m">  
			<start-instance>${coord:current(-10)}</start-instance>
          	<end-instance>${coord:current(0)}</end-instance>
		</data-in>
If all dependencies are available before 30m, this doesn't need to wait for 30 min.


> Specifying coordinator input datasets in more logical ways
> ----------------------------------------------------------
>
>                 Key: OOZIE-1976
>                 URL: https://issues.apache.org/jira/browse/OOZIE-1976
>             Project: Oozie
>          Issue Type: New Feature
>          Components: coordinator
>    Affects Versions: trunk
>            Reporter: Mona Chitnis
>            Assignee: Mona Chitnis
>             Fix For: trunk
>
>         Attachments: OOZIE-1976-rough-design.pdf
>
>
> All dataset instances specified as input to coordinator, currently work on AND logic i.e. ALL of them should be available for workflow to start. We should enhance this to include more logical ways of specifying availability criteria e.g.
>  * OR between instances
>  * minimum N out of K instances
>  * delta datasets (process data incrementally)
> Use-cases for this:
>  * Different datasets are BCP, and workflow can run with either, whichever arrives earlier.
>  * Data is not guaranteed, and while $coord:latest allows skipping to available ones, workflow will never trigger unless mentioned number of instances are found.
>  * Workflow is like a ‘refining’ algorithm which should run after minimum required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)