[jira] [Updated] (AIRFLOW-4512) New trigger rule for tasks - At least one successful

["Bharath Palaksha (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Palaksha updated AIRFLOW-4512:
--------------------------------------
    Description: 
{{New trigger rule - *atleast_one_success*}}

{{Trigger rules help in defining better dependencies on up stream tasks. There are good number of rules currently defined which are as below}}
 * {{all_success}}: (default) all parents have succeeded

 * {{all_failed}}: all parents are in a {{failed}} or {{upstream_failed}} state

 * {{all_done}}: all parents are done with their execution

 * {{one_failed}}: fires as soon as at least one parent has failed, it does not wait for all parents to be done

 * {{one_success}}: fires as soon as at least one parent succeeds, it does not wait for all parents to be done

 * {{none_failed}}: all parents have not failed ({{failed}} or {{upstream_failed}}) i.e. all parents have succeeded or been skipped

 * {{none_skipped}}: no parent is in a {{skipped}} state, i.e. all parents are in a {{success}}, {{failed}}, or {{upstream_failed}} state

 * {{dummy}}: dependencies are just for show, trigger at will

 

There can be another rule added here which is *atleast_one_success* - This waits for all parent tasks to be complete and checks if at least one parent is successful and triggers current task. It differs from one_success as it waits for all parents to be done. 

Consider a very common scenario in data pipelines where you have a number of parallel tasks generating some data. As a downstream task to all these generate tasks, you have a task to collate all data into one collection which has to run if any of the upstream generate is successful and also has to wait for all of them to be done. one_success can't be used as it doesn't wait for other tasks.

  was:
{{New trigger rule - *atleast_one_success*}}

{{Trigger rules help in defining better dependencies on up stream tasks. There are good number of rules currently defined which are as below}}
 * {{all_success}}: (default) all parents have succeeded

 * {{all_failed}}: all parents are in a {{failed}} or {{upstream_failed}} state

 * {{all_done}}: all parents are done with their execution

 * {{one_failed}}: fires as soon as at least one parent has failed, it does not wait for all parents to be done

 * {{one_success}}: fires as soon as at least one parent succeeds, it does not wait for all parents to be done

 * {{none_failed}}: all parents have not failed ({{failed}} or {{upstream_failed}}) i.e. all parents have succeeded or been skipped

 * {{none_skipped}}: no parent is in a {{skipped}} state, i.e. all parents are in a {{success}}, {{failed}}, or {{upstream_failed}} state

 * {{dummy}}: dependencies are just for show, trigger at will

 

There can be another rule added here which is *atleast_one_success* - This waits for all parent tasks to be complete and checks if at least one parent is successful and triggers current task. It differs from one_success as it does not wait for all parents to be done. 

Consider a very common scenario in data pipelines where you have a number of parallel tasks generating some data. As a downstream task to all these generate tasks, you have a task to collate all data into one collection which has to run if any of the upstream generate is successful and also has to wait for all of them to be done. one_success can't be used as it doesn't wait for other tasks.


> New trigger rule for tasks - At least one successful
> ----------------------------------------------------
>
>                 Key: AIRFLOW-4512
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4512
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: core, dependencies
>    Affects Versions: 1.10.3
>            Reporter: Bharath Palaksha
>            Assignee: Bharath Palaksha
>            Priority: Minor
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> {{New trigger rule - *atleast_one_success*}}
> {{Trigger rules help in defining better dependencies on up stream tasks. There are good number of rules currently defined which are as below}}
>  * {{all_success}}: (default) all parents have succeeded
>  * {{all_failed}}: all parents are in a {{failed}} or {{upstream_failed}} state
>  * {{all_done}}: all parents are done with their execution
>  * {{one_failed}}: fires as soon as at least one parent has failed, it does not wait for all parents to be done
>  * {{one_success}}: fires as soon as at least one parent succeeds, it does not wait for all parents to be done
>  * {{none_failed}}: all parents have not failed ({{failed}} or {{upstream_failed}}) i.e. all parents have succeeded or been skipped
>  * {{none_skipped}}: no parent is in a {{skipped}} state, i.e. all parents are in a {{success}}, {{failed}}, or {{upstream_failed}} state
>  * {{dummy}}: dependencies are just for show, trigger at will
>  
> There can be another rule added here which is *atleast_one_success* - This waits for all parent tasks to be complete and checks if at least one parent is successful and triggers current task. It differs from one_success as it waits for all parents to be done. 
> Consider a very common scenario in data pipelines where you have a number of parallel tasks generating some data. As a downstream task to all these generate tasks, you have a task to collate all data into one collection which has to run if any of the upstream generate is successful and also has to wait for all of them to be done. one_success can't be used as it doesn't wait for other tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)