[GitHub] [airflow] rkh-hash commented on issue #13047: Occasional "KeyError" in dag_processing

[GitBox <gi...@apache.org>]

rkh-hash commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-754661486


   Hi everyone,
   
   I encounter the same behaviour .
   
   My simple DAG:
   
   ```python
   """
       Example of DAG with only dummy operator.
   """
   
   from datetime import datetime
   
   from airflow.models import DAG
   from airflow.operators.dummy import DummyOperator
   
   DEFAULT_DATE = datetime(2020, 1, 4)
   
   default_args = {
       "owner": "airflow",
       "start_date": DEFAULT_DATE,
   }
   
   
   with DAG(
       dag_id="test_only_dummy_tasks",
       default_args=default_args,
       schedule_interval='*/30 * * * *'
   ) as dag:
   
       task_a = DummyOperator(task_id="test_task_a")
   
       task_b = DummyOperator(task_id="test_task_b")
   
       task_a >> task_b
   ```
   
   scheduler log: 
   ```
   Process ForkProcess-131:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
       self.run()
     File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
       self._target(*self._args, **self._kwargs)
     File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
       processor_manager.start()
     File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 596, in start
       return self._run_parsing_loop()
     File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
       self._processors.pop(processor.file_path)
   KeyError: '/opt/airflow/dags/example_dag_dummy_tasks.py'
   ```
   
   
   * os: `Ubuntu 18.04.5 LTS`
   * python version: `python3.7`
   * airflow version: `2.0.0`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org