You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/06/22 08:18:00 UTC

[jira] [Commented] (AIRFLOW-4512) New trigger rule for tasks - At least one successful

    [ https://issues.apache.org/jira/browse/AIRFLOW-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141808#comment-17141808 ] 

ASF GitHub Bot commented on AIRFLOW-4512:
-----------------------------------------

jledru commented on pull request #5288:
URL: https://github.com/apache/airflow/pull/5288#issuecomment-647362242


   Hello,
   Is there any method to achieve the goal of atleast_one_success ?
   As I would need this behaviour in one of my DAG.
   Thanks,


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> New trigger rule for tasks - At least one successful
> ----------------------------------------------------
>
>                 Key: AIRFLOW-4512
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4512
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: core, dependencies
>    Affects Versions: 1.10.3
>            Reporter: Bharath Palaksha
>            Assignee: Bharath Palaksha
>            Priority: Minor
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> {{New trigger rule - *atleast_one_success*}}
> {{Trigger rules help in defining better dependencies on up stream tasks. There are good number of rules currently defined which are as below}}
>  * {{all_success}}: (default) all parents have succeeded
>  * {{all_failed}}: all parents are in a {{failed}} or {{upstream_failed}} state
>  * {{all_done}}: all parents are done with their execution
>  * {{one_failed}}: fires as soon as at least one parent has failed, it does not wait for all parents to be done
>  * {{one_success}}: fires as soon as at least one parent succeeds, it does not wait for all parents to be done
>  * {{none_failed}}: all parents have not failed ({{failed}} or {{upstream_failed}}) i.e. all parents have succeeded or been skipped
>  * {{none_skipped}}: no parent is in a {{skipped}} state, i.e. all parents are in a {{success}}, {{failed}}, or {{upstream_failed}} state
>  * {{dummy}}: dependencies are just for show, trigger at will
>  
> There can be another rule added here which is *atleast_one_success* - This waits for all parent tasks to be complete and checks if at least one parent is successful and triggers current task. It differs from one_success as it waits for all parents to be done. 
> Consider a very common scenario in data pipelines where you have a number of parallel tasks generating some data. As a downstream task to all these generate tasks, you have a task to collate all data into one collection which has to run if any of the upstream generate is successful and also has to wait for all of them to be done. one_success can't be used as it doesn't wait for other tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)