[jira] [Commented] (AIRFLOW-2680) Don't automatically percolate skipped state

["Andrew Harmon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688469#comment-16688469 ] 

Andrew Harmon commented on AIRFLOW-2680:
----------------------------------------

I'm looking for a solution to the same issue. I don't think you always want the SKIP status to propagate downstream. It would be nice to have some more control of this

> Don't automatically percolate skipped state
> -------------------------------------------
>
>                 Key: AIRFLOW-2680
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2680
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Andrei-Alin Popescu
>            Assignee: Andrei-Alin Popescu
>            Priority: Major
>
> Dear Airflow Maintainers,
>  
> As part of our workflow, we have cases where all the upstream of a certain task A can be skipped. In this case, airflow seems to automatically mark A as skipped.
> However, this does not quite work for us, since there are changes external to the DAG which A needs to process, regardless of whether its upstream ran or not. Additionally, we require A to get into an "upstream_failed" state and not run if any its upstream tasks failed.
> I don't see a trigger rule to cover this, so what would be the best way to achieve this? I was thinking we could attach a DummyOperator as an upstream to A, which in a way marks the fact that A depends on some external data and needs to run anyway, but this can get really ugly for big DAGs.
> I was also thinking we could have a new trigger_rule, e.g. "no_failure", which would only trigger tasks if no upstream has failed. It differs from "all_success" in that it will also trigger if all upstream has been skipped, rather than percolating the skipped state on.
> I'd really appreciate your feedback on this, and I'd like to know if in fact there is already a good way of doing this with airflow that I don't know of.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)