[jira] [Commented] (AIRFLOW-2843) ExternalTaskSensor: Add option to cease waiting immediately if the external DAG/task doesn't exist

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745046#comment-16745046 ] 

ASF GitHub Bot commented on AIRFLOW-2843:
-----------------------------------------

XD-DENG commented on pull request #4547: [AIRFLOW-2843] Add flag in ExternalTaskSensor to check if external DAG/task exists
URL: https://github.com/apache/airflow/pull/4547
 
 
   ### Jira
   
     - https://issues.apache.org/jira/browse/AIRFLOW-2843
   
   ### Description
   
   #### Background
   `ExternalTaskSensor` will keep waiting even if the external DAG/task specified doesn't exist at all (this is making sense in some scenarios).
   
   But it may be good to provide an option to cease waiting immediately if the external task specified doesn't exist.
   
   #### Proposal
   Provide a flag `check_existence`. Set to `True` to check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist.
   
   - **Case - 1 Waiting for External DAG**: will check if the DAG exists in DB, AND whether the DAG file exists in file system
   - **Case - 2 Waiting for External Task**: the check to do for DAG will happen as well. In addition, re-scan the DAG and check if this task is in the DAG.
   
   **The default value is set to `False` (no check or ceasing will happen), so it will not affect any existing DAGs or user expectation.**
   
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> ExternalTaskSensor: Add option to cease waiting immediately if the external DAG/task doesn't exist
> --------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2843
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2843
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: operators
>            Reporter: Xiaodong DENG
>            Assignee: Xiaodong DENG
>            Priority: Minor
>
> h2. Background
> *ExternalTaskSensor* will keep waiting (given restrictions of retries, poke_interval, etc), even if the external DAG/task specified doesn't exist at all. In some cases, this waiting may still make sense as new DAG may backfill.
> But it may be good to provide an option to cease waiting immediately if the external DAG/task specified doesn't exist.
> h2. Proposal
> Provide an argument "check_existence". Set to *True* to check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist.
> The default value is set to *False* (no check or ceasing will happen) so it will not affect any existing DAGs or user expectation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)