[GitHub] [airflow] abhishekshenoy commented on issue #13047: Occasional "KeyError" in dag_processing

[GitBox <gi...@apache.org>]

abhishekshenoy commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-788039194


   Hi Team ,
      
      Our dags are failing to start with the below issue . We have not done any chages , had just restarted our scheduler and then onwards we are facing this issue. 
      
     ```
     [2021-03-01 15:27:34,622] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: wmt.hw.clinic.create_cluster.create_cluster 2021-03-01 15:06:37.473524+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
   [2021-03-01 15:27:36,366] {dagrun.py:429} ERROR - Marking run <DagRun wmt.hw.clinic.create_cluster @ 2021-03-01 15:06:37.473524+00:00: manual__2021-03-01T15:06:37.473524+00:00, externally triggered: True> failed
   Process ForkProcess-8:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
       self.run()
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
       processor_manager.start()
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 596, in start
       return self._run_parsing_loop()
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
       self._processors.pop(processor.file_path)
   KeyError: '/opt/airflow/dags/a0s03fc/rcm_clinic_create_cluster.py'
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org