You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Wesley (Jira)" <ji...@apache.org> on 2019/09/13 10:06:00 UTC
[jira] [Commented] (AIRFLOW-2843) ExternalTaskSensor: Add option to
cease waiting immediately if the external DAG/task doesn't exist
[ https://issues.apache.org/jira/browse/AIRFLOW-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929083#comment-16929083 ]
Wesley commented on AIRFLOW-2843:
---------------------------------
Using the operator stumbled upon the case where the `external_dag_id` and `external_task_id` exist but the DAG is disabled, so the ExternalTaskSensor task instance keep polling for a DagRun that does not exist.
Perhaps the `check_existence` should check DagRun instead? or the name should be more specific like `check_dag_existence`/ `check_task_existence`?
What you guys think about this use case?
> ExternalTaskSensor: Add option to cease waiting immediately if the external DAG/task doesn't exist
> --------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-2843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2843
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Reporter: Xiaodong DENG
> Assignee: Xiaodong DENG
> Priority: Minor
> Fix For: 1.10.3
>
>
> h2. Background
> *ExternalTaskSensor* will keep waiting (given restrictions of retries, poke_interval, etc), even if the external DAG/task specified doesn't exist at all. In some cases, this waiting may still make sense as new DAG may backfill.
> But it may be good to provide an option to cease waiting immediately if the external DAG/task specified doesn't exist.
> h2. Proposal
> Provide an argument "check_existence". Set to *True* to check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist.
> The default value is set to *False* (no check or ceasing will happen) so it will not affect any existing DAGs or user expectation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)