[jira] [Updated] (AIRFLOW-6853) ExternalTaskSensor does not work with packaged DAGs

["Johannes Wienke (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johannes Wienke updated AIRFLOW-6853:
-------------------------------------
    Description: 
The current implementation of ExternalTaskSensor uses the file location of a DAG to determine if it still exists. This is realized by the following code fragment:

{code:python}
                if not os.path.exists(dag_to_wait.fileloc):
                    raise AirflowException('The external DAG '
                                           '{} was deleted.'.format(self.external_dag_id)
{code}

In case the DAG to check is part of a Zip archive, fileloc looks something like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can obviously not be checked with {{os.path.exists}} and therefore the sensor is currently unable to handle packaged DAGs at all.

This also contradicts the explicit note in the DAG model:
{noformat}
    # The location of the file containing the DAG object
    # Note: Do not depend on fileloc pointing to a file; in the case of a
    # packaged DAG, it will point to the subpath of the DAG within the
    # associated zip.
{noformat}

I suspect, the sensor should instead use {{is_active}}.

  was:
The current implementation of ExternalTaskSensor uses the file location of a DAG to determine if it still exists. This is realized by the following code fragment:

{code:python}
                if not os.path.exists(dag_to_wait.fileloc):
                    raise AirflowException('The external DAG '
                                           '{} was deleted.'.format(self.external_dag_id)
{code}

In case the DAG to check is part of a Zip archive, fileloc looks something like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can obviously not be checked with {{os.path.exists}} and therefore the sensor is currently unable to handle packaged DAGs at all.


> ExternalTaskSensor does not work with packaged DAGs
> ---------------------------------------------------
>
>                 Key: AIRFLOW-6853
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6853
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>    Affects Versions: 1.10.9
>            Reporter: Johannes Wienke
>            Priority: Major
>
> The current implementation of ExternalTaskSensor uses the file location of a DAG to determine if it still exists. This is realized by the following code fragment:
> {code:python}
>                 if not os.path.exists(dag_to_wait.fileloc):
>                     raise AirflowException('The external DAG '
>                                            '{} was deleted.'.format(self.external_dag_id)
> {code}
> In case the DAG to check is part of a Zip archive, fileloc looks something like: {{/usr/local/airflow/dags/analytics.zip/create_keyframes.py}}. This can obviously not be checked with {{os.path.exists}} and therefore the sensor is currently unable to handle packaged DAGs at all.
> This also contradicts the explicit note in the DAG model:
> {noformat}
>     # The location of the file containing the DAG object
>     # Note: Do not depend on fileloc pointing to a file; in the case of a
>     # packaged DAG, it will point to the subpath of the DAG within the
>     # associated zip.
> {noformat}
> I suspect, the sensor should instead use {{is_active}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)