You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/29 11:03:12 UTC
[GitHub] [airflow] thiagophx opened a new issue #11936: Huge increase on DagFileProcessorProcess
thiagophx opened a new issue #11936:
URL: https://github.com/apache/airflow/issues/11936
**Apache Airflow version**: 1.10.12
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.10
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch)
- **Kernel** (e.g. `uname -a`): 4.19.123-coreos
**What happened**:
After rolling out to Airflow 1.10.12, we've started to notice a huge increase on loading up DAG files. Some of our DAGs started timing out, and we see the following error on the logs:
```
Processor for /airflow/airflow-dags/dags/my_dag.py with PID 27954 started at 2020-10-29T02:21:04.630540+00:00 has timed out, killing it
Processor for /airflow/airflow-dags/dags/my_dag.py exited with return code -9
```
**What you expected to happen**:
We didn't have this issue on the 1.10.9. Our DAGs load time went from 1 second to over 40 seconds, and therefore going over the `dag_file_processor_timeout` setting.
**How to reproduce it**:
Any DAG with more than 200 tasks seem to be affected. On our case we have one DAG full of Sensors to other DAGs, but the issue doesn't seem to be related to a particular type of task, but rather the total size of the DAG.
**Anything else we need to know**:
The problem occurs every time the scheduled time is reached, so that the tasks are created, and the process times out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11936: Huge increase on DagFileProcessorProcess
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11936:
URL: https://github.com/apache/airflow/issues/11936#issuecomment-718762315
Is it relaated? https://github.com/apache/airflow/issues/11780 https://github.com/apache/airflow/pull/11010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thiagophx commented on issue #11936: Huge increase on DagFileProcessorProcess
Posted by GitBox <gi...@apache.org>.
thiagophx commented on issue #11936:
URL: https://github.com/apache/airflow/issues/11936#issuecomment-718768021
Hi @mik-laj , it seems definitely related.
So if I understand correctly, the fix will be merged on 1.10.13, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #11936: Huge increase on DagFileProcessorProcess
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #11936:
URL: https://github.com/apache/airflow/issues/11936
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #11936: Huge increase on DagFileProcessorProcess
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #11936:
URL: https://github.com/apache/airflow/issues/11936#issuecomment-718662360
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11936: Huge increase on DagFileProcessorProcess
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11936:
URL: https://github.com/apache/airflow/issues/11936#issuecomment-718771692
Yes. Fix will be released in the next release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thiagophx commented on issue #11936: Huge increase on DagFileProcessorProcess
Posted by GitBox <gi...@apache.org>.
thiagophx commented on issue #11936:
URL: https://github.com/apache/airflow/issues/11936#issuecomment-718776071
That's awesome, thanks so much for the quick response 🚀
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #11936: Huge increase on DagFileProcessorProcess
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #11936:
URL: https://github.com/apache/airflow/issues/11936#issuecomment-718762315
Is it related? https://github.com/apache/airflow/issues/11780 https://github.com/apache/airflow/pull/11010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org