You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/13 19:53:35 UTC
[GitHub] [airflow] potiuk opened a new issue #13047: Occasional "KeyError" in dag_processing
potiuk opened a new issue #13047:
URL: https://github.com/apache/airflow/issues/13047
**Apache Airflow version**: 2.0.0rc2
**Environment**: Breeze with example dags, Python 3.8 postgres.
- **OS** (e.g. from /etc/os-release): Linux
- **Kernel** (e.g. `uname -a`): Breeze CI image
- **Install tools**: Breeze:
```
./breeze start-airflow --backend postgres --load-example-dags --load-default-connections --install-airflow-version 2.0.0rc2 --skip-mounting-local-sources --python 3.8
```
**What happened**:
When testing airflow logging I occasionally stumble upon "KeyError' from `dag_procesing.py`. I am not sure exactly when it happens. It's not always reproducible but it looks like it is when I restart scheduler and trigger 'example_bash_operator.py" it happens rather randomly (1/10 times more or less). It does not happen always when I triggere task manually. DAG gets correctly executed after triggering, but the log is there and warniing printed in the logs right after the DAG finishes execution.
The error I see in scheduler's logs:
```
[2020-12-13 19:35:33,752] {dagbag.py:440} INFO - Filling up the DagBag from /usr/local/lib/python3.8/site-packages/airflow/example_dags/example_bash_operator.py
Running <TaskInstance: example_bash_operator.run_after_loop 2020-12-13T19:35:30.648020+00:00 [queued]> on host 6611da4b1a27
[2020-12-13 19:35:34,517] {dagrun.py:444} INFO - Marking run <DagRun example_bash_operator @ 2020-12-13 19:35:30.648020+00:00: manual__2020-12-13T19:35:30.648020+00:00, externally triggered: True> successful
[2020-12-13 19:35:34,523] {scheduler_job.py:1193} INFO - Executor reports execution of example_bash_operator.run_after_loop execution_date=2020-12-13 19:35:30.648020+00:00 exited with status success for try_number 1
Process ForkProcess-34:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
processor_manager.start()
File "/usr/local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 596, in start
return self._run_parsing_loop()
File "/usr/local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
self._processors.pop(processor.file_path)
KeyError: '/usr/local/lib/python3.8/site-packages/airflow/example_dags/example_bash_operator.py'
[2020-12-13 19:35:35,589] {dag_processing.py:396} WARNING - DagFileProcessorManager (PID=1029759) exited with exit code 1 - re-launching
```
**What you expected to happen**:
No error in logs.
**How to reproduce it**:
```
./breeze start-airflow --backend postgres --load-example-dags --load-default-connections --install-airflow-version 2.0.0rc2 --skip-mounting-local-sources --python 3.8
```
Login to the webserver, enable 'example_bash_operator", wait for it to execute. Trigger the example DAG several times (always wait for the end of execution. It happens randomly (for me around 1/10 tasks)
**Anything else we need to know**:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] abhishekshenoy commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
abhishekshenoy commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-788039194
Hi Team ,
Our dags are failing to start with the below issue . We have not done any chages , had just restarted our scheduler and then onwards we are facing this issue.
```
[2021-03-01 15:27:34,622] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: wmt.hw.clinic.create_cluster.create_cluster 2021-03-01 15:06:37.473524+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-03-01 15:27:36,366] {dagrun.py:429} ERROR - Marking run <DagRun wmt.hw.clinic.create_cluster @ 2021-03-01 15:06:37.473524+00:00: manual__2021-03-01T15:06:37.473524+00:00, externally triggered: True> failed
Process ForkProcess-8:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
processor_manager.start()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 596, in start
return self._run_parsing_loop()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
self._processors.pop(processor.file_path)
KeyError: '/opt/airflow/dags/a0s03fc/rcm_clinic_create_cluster.py'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744557173
Yeah I have seen this error - will need to dig in a bit -- but is a non-harmful one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744589243
> > Yeah I have seen this error - will need to dig in a bit -- but is a non-harmful one.
>
> Agree. Until we start having an influx of questions about all the warnings :)
Indeed, I will take a look at it soon'ish -- and hopefully we can merge a fix so we can tell them the fix is merged -- 2.0.1 will contain the fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744371348
I can confirm that this error happens quite often
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744059620
Summoning @ashb here. Not sure how serious it is, maybe this is only a warning and we can release without solving it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744062028
Can you check in all the other log files and see if we can an error anywhere
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744090357
Agree it's not fatal and likely is not a good reson for next RC.. I could not find any other related log errors. This looks like some race condition possibly in the new HA scheduler loop - my guts feeling tell me this is in the new "short-cirucuitt" part of the loop after the main loop is completed(). Havnt' looked at the code but I recalled there was some late addition here and that looks like this might be the reason (but if it is,then except the warning, it should be harmless)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744586482
> Yeah I have seen this error - will need to dig in a bit -- but is a non-harmful one.
Agree. Until we start having an influx of questions about all the warnings :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744425618
I think this is something to do with this -- we're somehow marking the dag run as sucessful multiple times!
```
[2020-12-14 13:02:38,752] 2393901 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1200}} INFO - Executor reports execution of example_branch_operator.branching execution_date=2020-12-13 00:00:00+00:00 exited with status queued for try_number 1
[2020-12-14 13:02:38,754] 2393901 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1220}} INFO - Setting external_id for <TaskInstance: example_branch_operator.branching 2020-12-13 00:00:00+00:00 [queued]> to eea37e5d-1280-45c5-8373-5c10338da84a
[2020-12-14 13:02:39,864] 2393901 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1200}} INFO - Executor reports execution of example_branch_operator.branching execution_date=2020-12-12 00:00:00+00:00 exited with status success for try_number 1
[2020-12-14 13:02:39,864] 2393901 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1200}} INFO - Executor reports execution of example_branch_operator.branching execution_date=2020-12-13 00:00:00+00:00 exited with status success for try_number 1
[2020-12-14 13:02:41,983] 2393901 MainProcess {{airflow.models.dagrun.DagRun dagrun.py:444}} INFO - Marking run <DagRun example_branch_operator @ 2020-12-13 00:00:00+00:00: scheduled__2020-12-13T00:00:00+00:00, externally triggered: False> successful
[2020-12-14 13:02:41,990] 2393901 MainProcess {{airflow.models.dagrun.DagRun dagrun.py:444}} INFO - Marking run <DagRun example_branch_operator @ 2020-12-12 00:00:00+00:00: scheduled__2020-12-12T00:00:00+00:00, externally triggered: False> successful
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rkh-hash commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
rkh-hash commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-754661486
Hi everyone,
I encounter the same behaviour .
My simple DAG:
```python
"""
Example of DAG with only dummy operator.
"""
from datetime import datetime
from airflow.models import DAG
from airflow.operators.dummy import DummyOperator
DEFAULT_DATE = datetime(2020, 1, 4)
default_args = {
"owner": "airflow",
"start_date": DEFAULT_DATE,
}
with DAG(
dag_id="test_only_dummy_tasks",
default_args=default_args,
schedule_interval='*/30 * * * *'
) as dag:
task_a = DummyOperator(task_id="test_task_a")
task_b = DummyOperator(task_id="test_task_b")
task_a >> task_b
```
scheduler log:
```
Process ForkProcess-131:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
processor_manager.start()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 596, in start
return self._run_parsing_loop()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
self._processors.pop(processor.file_path)
KeyError: '/opt/airflow/dags/example_dag_dummy_tasks.py'
```
* os: `Ubuntu 18.04.5 LTS`
* python version: `python3.7`
* airflow version: `2.0.0`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-788138438
> Hi Team ,
>
> Our dags are failing to start with the below issue . We have not done any chages , had just restarted our scheduler and then onwards we are facing this issue.
>
> ```
> [2021-03-01 15:27:34,622] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: wmt.hw.clinic.create_cluster.create_cluster 2021-03-01 15:06:37.473524+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
> [2021-03-01 15:27:36,366] {dagrun.py:429} ERROR - Marking run <DagRun wmt.hw.clinic.create_cluster @ 2021-03-01 15:06:37.473524+00:00: manual__2021-03-01T15:06:37.473524+00:00, externally triggered: True> failed
> Process ForkProcess-8:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
> self.run()
> File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
> self._target(*self._args, **self._kwargs)
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
> processor_manager.start()
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 596, in start
> return self._run_parsing_loop()
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
> self._processors.pop(processor.file_path)
> KeyError: '/opt/airflow/dags/a0s03fc/rcm_clinic_create_cluster.py'
> ```
You need to update to 2.0.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-748348948
https://github.com/apache/airflow/pull/13163 should significantly reduce this error.
apache/airflow#13651 fixes a bug introduced by my previous PR (apache/airflow#13163)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-748348948
https://github.com/apache/airflow/pull/13163 should significantly reduce this error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744442709
Oh I can't read -- those are two separate dates. So it makes sense that we'd set multiple runs to success :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744061897
From what I've read here I think it's non fatal - the dag processor is no longer "critical", and we can dig to the bottom of this for 2.0.x
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13047:
URL: https://github.com/apache/airflow/issues/13047
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744371348
I can confirm that this error happens quite often
> Can you check in all the other log files and see if we have any other errors recorded anywhere?
I saw no other errors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-748348948
https://github.com/apache/airflow/pull/13163 should significantly reduce this error:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744062028
Can you check in all the other log files and see if we have any other errors recorded anywhere?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13047: Occasional "KeyError" in dag_processing
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13047:
URL: https://github.com/apache/airflow/issues/13047#issuecomment-744090357
Agree it's not fatal and likely is not a good reson for next RC.. I could not find any other related log errors. This looks like some race condition possibly in the new HA scheduler loop - my guts feeling tell me this is in the new "short-cirucuitt" part of the loop after the main loop is gone. Havnt' looked at the code but I recalled there was some late addition here and that looks like this might be the reason (but if it is,then except the warning, it should be harmless)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org