You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/13 11:05:27 UTC
[GitHub] [airflow] twindebank opened a new issue #11494: DAG File Processor Slowdown in v1.10.12
twindebank opened a new issue #11494:
URL: https://github.com/apache/airflow/issues/11494
**Apache Airflow version**: 1.10.12
**Environment**: Mac / Docker (official image) for Mac / Docker (official image) deployed to GCE
## What happened
**There appears to be a 2-4x slowdown in DAG file processing times when comparing v1.10.11 to v1.10.12.**
We first noticed the issue when we upgraded our production deployment. We found that some DAGs were failing to schedule, and on inspecting the logs we worked out that the `AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT` limit was being hit. Looking through the `dag_processor_manager` logs, we found that DAGs were taking 3-4x longer to build in v1.10.12 compared to v1.10.11.
I set up a minimal test environment to replicate this issue locally with one randomly generated DAG containing 500 nodes. I first ran Airflow using the official docker images for quick results, then I ran Airflow directly on my machine once I confirmed the slowdown was present. The average `dag_processor_manager` timings were:
```
Docker/1.10.11: 9.94s
Docker/1.10.12: 27.84s
pyenv/1.10.11: 9.56s
pyenv/1.10.12: 24.67s
```
## How to reproduce it
The setup I used to test the issue is here: [airflow-slowdown-example.zip](https://github.com/apache/airflow/files/5370865/airflow-slowdown-example.zip).
It contains scripts to run Airflow via docker or directly, a randomly generated DAG, and a script to scrape and average the `dag_processor_manager` logs.
To run the tests:
Set the airflow version in the `Makefile` to `v1.10.11`.
`make build-pyenv` or `make build-docker`.
`make run-pyenv` or `make run-docker`.
Open Airflow UI and run the DAG.
Let it run for ~20mins.
`make kill-pyenv` or `make kill-docker`.
Repeat for `v1.10.12`.
Run the `summarise_logs.py` script to scrape the logs and calculate average `dag_processor_manager` timings.
**Anything else we need to know**:
In the test environment, the DAG processing time drops to 2s, for both versions, when the DAG has no active DAG run instance. I think both the webserver and the scheduler invoke the DAG processor manager by default, so I wonder if it's related to the scheduler only?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11494: DAG File Processor Slowdown in v1.10.12
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11494:
URL: https://github.com/apache/airflow/issues/11494#issuecomment-785829296
Massively re-worked dag parsing in Airflow 2.0 - closing this ticket now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #11494: DAG File Processor Slowdown in v1.10.12
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #11494:
URL: https://github.com/apache/airflow/issues/11494#issuecomment-707665853
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #11494: DAG File Processor Slowdown in v1.10.12
Posted by GitBox <gi...@apache.org>.
ashb closed issue #11494:
URL: https://github.com/apache/airflow/issues/11494
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11494: DAG File Processor Slowdown in v1.10.12
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11494:
URL: https://github.com/apache/airflow/issues/11494#issuecomment-722540920
Can you test it with current v1-10-test branch please?
https://github.com/apache/airflow/commit/cb750c11b1899a88cfdf25baa18cf9896c7cef8f and https://github.com/apache/airflow/commit/edae056edd6b6fae94533ca8a4cc220ebfbadc68 might fix it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org