[GitHub] [airflow] wongelz opened a new issue #14264: AirflowException: The external DAG was deleted when external_dag_id references zipped DAG

[GitBox <gi...@apache.org>]

wongelz opened a new issue #14264:
URL: https://github.com/apache/airflow/issues/14264


   **Apache Airflow version**: 2.0.0
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.16.3
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**:
   - **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
   - **Kernel** (e.g. `uname -a`): Linux airflow 3.10.0-1127.10.1.el7.x86_64 #1 SMP Tue May 26 15:05:43 EDT 2020 x86_64 GNU/Linux
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   ExternalTaskSensor with check_existence=True referencing an external DAG inside a .zip file raises the following exception:
   ```
   ERROR - The external DAG dag_a /opt/airflow-data/dags/my_dags.zip/dag_a.py was deleted.
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
       self._prepare_and_execute_task_with_callbacks(context, task)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
       result = self._execute_task(context, task_copy)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
       result = task_copy.execute(context=context)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/sensors/base.py", line 228, in execute
       while not self.poke(context):
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/sensors/external_task.py", line 159, in poke
       self._check_for_existence(session=session)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/sensors/external_task.py", line 184, in _check_for_existence
       raise AirflowException(f'The external DAG {self.external_dag_id} was deleted.')
   airflow.exceptions.AirflowException: The external DAG dag_a /opt/airflow-data/dags/my_dags.zip/dag_a.py was deleted.
   ```
   
   **What you expected to happen**:
   
   The existence check should PASS.
   
   **How to reproduce it**:
   
   1. Create a file *dag_a.py* with the following contents:
   ```
   from airflow import DAG
   from airflow.operators.dummy import DummyOperator
   from airflow.utils.timezone import datetime
   
   DEFAULT_DATE = datetime(2015, 1, 1)
   with DAG("dag_a", start_date=DEFAULT_DATE, schedule_interval="@daily") as dag:
       task_a = DummyOperator(task_id="task_a", dag=dag)
   ```
   
   2. Create a file *dag_b.py* with contents:
   ```
   from airflow import DAG
   from airflow.operators.dummy import DummyOperator
   from airflow.sensors.external_task import ExternalTaskSensor
   from airflow.utils.timezone import datetime
   
   DEFAULT_DATE = datetime(2015, 1, 1)
   
   with DAG("dag_b", start_date=DEFAULT_DATE, schedule_interval="@daily") as dag:
       sense_a = ExternalTaskSensor(
           task_id="sense_a",
           external_dag_id="dag_a",
           external_task_id="task_a",
           check_existence=True
       )
       task_b = DummyOperator(task_id="task_b", dag=dag)
       sense_a >> task_b
   ```
   
   3. `zip my_dags.zip dag_a.py dag_b.py`
   4. Load *my_dags.zip* into airflow and run *dag_b*
   5. Task *sense_a* will fail with exception above.
   
   **Anything else we need to know**:
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #14264: AirflowException: The external DAG was deleted when external_dag_id references zipped DAG

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #14264:
URL: https://github.com/apache/airflow/issues/14264#issuecomment-780202312


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org