You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jun-roh (via GitHub)" <gi...@apache.org> on 2025/11/06 05:04:35 UTC
[D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
GitHub user jun-roh created a discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I’m running Apache Airflow 3.1.1 using Docker Compose, and I’ve noticed that the Web UI is extremely slow when loading data — especially the main DAG list and DAG details pages.
We currently have about 90 DAGs, and everything else (scheduler, workers, etc.) seems to be functioning normally. However, the UI becomes noticeably sluggish, sometimes taking 20–30 seconds to render pages.
Here’s a simplified version of my setup:
`
version: "3.8"
networks:
airflow-net:
driver: bridge
volumes:
airflow-git:
driver: local
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME}
user: "${AIRFLOW_UID:-50000}:0"
restart: unless-stopped
environment:
# ---- Core ----
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__STORE_SERIALIZED_DAGS: True
AIRFLOW__CORE__LOAD_EXAMPLES: False
# ---- Database ----
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
AIRFLOW__CELERY__RESULT_BACKEND: "db+mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
# ---- Pooling ----
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE: 30
AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_TIMEOUT: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE: 1800
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING: True
# ---- Scheduler ----
AIRFLOW__SCHEDULER__PARSING_PROCESSES: 1
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 120
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 300
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: 64
# ---- Logging ----
AIRFLOW__LOGGING__REMOTE_LOGGING: True
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://example-bucket
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws_default
# ---- Celery ----
AIRFLOW__CELERY__BROKER_URL: redis://redis.example.net:6379/2
AIRFLOW__CELERY__TASK_TRACK_STARTED: True
AIRFLOW__CELERY__PREFETCH_MULTIPLIER: 1
TZ: Asia/Seoul
volumes:
- airflow-git:/opt/airflow/git
- ./airflow_local_settings.py:/opt/airflow/airflow_local_settings.py:ro,Z
networks: [airflow-net]
services:
airflow-apiserver:
<<: *airflow-common
command: >
bash -c 'exec airflow api-server --host 0.0.0.0 --port 8080 --workers 1 --apps all'
ports: ["8080:8080"]
depends_on:
airflow-gitsync: { condition: service_healthy }
airflow-init: { condition: service_completed_successfully }
`
Environment summary
• Airflow version: 3.1.1
• Executor: CeleryExecutor
• DB: MySQL on AWS RDS (db.r7g.large)
• Redis: hosted Redis cluster (for Celery broker)
• DAG count: ~90
• UI: AWS loadbalancer using HTTPS
• Serialized DAGs: enabled (STORE_SERIALIZED_DAGS=True)
The Airflow UI is slow:
• DAG list takes 10–20 seconds to load
• Clicking into a DAG detail or “Graph View” is also very sluggish
• Scheduler and workers seem healthy, tasks execute fine
• Database CPU/memory usage is normal
GitHub link: https://github.com/apache/airflow/discussions/57933
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
GitHub user marcosmartinezfco added a comment to the discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I upgraded to `3.1.4` and the ui feels way smoother now. The only bit that still feels quite slow is the switch to enable/disable a dag. It taks a solid 10 seconds to flip.
GitHub link: https://github.com/apache/airflow/discussions/57933#discussioncomment-15221762
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "jun-roh (via GitHub)" <gi...@apache.org>.
GitHub user jun-roh edited a discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I’m running Apache Airflow 3.1.1 using Docker Compose, and I’ve noticed that the Web UI is extremely slow when loading data — especially the main DAG list and DAG details pages.
We currently have about 90 DAGs, and everything else (scheduler, workers, etc.) seems to be functioning normally. However, the UI becomes noticeably sluggish, sometimes taking 20–30 seconds to render pages.
Here’s a simplified version of my setup:
`
version: "3.8"
networks:
airflow-net:
driver: bridge
volumes:
airflow-git:
driver: local
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME}
user: "${AIRFLOW_UID:-50000}:0"
restart: unless-stopped
environment:
# ---- Core ----
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__STORE_SERIALIZED_DAGS: True
AIRFLOW__CORE__LOAD_EXAMPLES: False
# ---- Database ----
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
AIRFLOW__CELERY__RESULT_BACKEND: "db+mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
# ---- Pooling ----
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE: 30
AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_TIMEOUT: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE: 1800
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING: True
# ---- Scheduler ----
AIRFLOW__SCHEDULER__PARSING_PROCESSES: 1
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 120
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 300
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: 64
# ---- Logging ----
AIRFLOW__LOGGING__REMOTE_LOGGING: True
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://example-bucket
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws_default
# ---- Celery ----
AIRFLOW__CELERY__BROKER_URL: redis://redis.example.net:6379/2
AIRFLOW__CELERY__TASK_TRACK_STARTED: True
AIRFLOW__CELERY__PREFETCH_MULTIPLIER: 1
TZ: Asia/Seoul
volumes:
- airflow-git:/opt/airflow/git
- ./airflow_local_settings.py:/opt/airflow/airflow_local_settings.py:ro,Z
networks: [airflow-net]
services:
airflow-apiserver:
<<: *airflow-common
command: >
bash -c 'exec airflow api-server --host 0.0.0.0 --port 8080 --workers 1 --apps all'
ports: ["8080:8080"]
depends_on:
airflow-gitsync: { condition: service_healthy }
airflow-init: { condition: service_completed_successfully }
`
Environment summary
• Airflow version: 3.1.1
• Executor: CeleryExecutor
• DB: MySQL on AWS RDS (db.r7g.large)
• Redis: hosted Redis cluster (for Celery broker)
• DAG count: ~90
• UI: AWS loadbalancer using HTTPS
• Serialized DAGs: enabled (STORE_SERIALIZED_DAGS=True)
The Airflow UI is slow:
• DAG list takes 10–20 seconds to load
• Clicking into a DAG detail or “Graph View” is also very sluggish
• Scheduler and workers seem healthy, tasks execute fine
• Database CPU/memory usage is normal
GitHub link: https://github.com/apache/airflow/discussions/57933
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
GitHub user marcosmartinezfco deleted a comment on the discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I upgraded to `3.1.4` and the ui feels way smoother now. The only bit that still feels quite slow is the switch to enable/disable a dag. It taks a solid 10 seconds to flip.
GitHub link: https://github.com/apache/airflow/discussions/57933#discussioncomment-15221755
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "jun-roh (via GitHub)" <gi...@apache.org>.
GitHub user jun-roh edited a discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I’m running Apache Airflow 3.1.1 using Docker Compose, and I’ve noticed that the Web UI is extremely slow when loading data — especially the main DAG list and DAG details pages.
We currently have about 90 DAGs, and everything else (scheduler, workers, etc.) seems to be functioning normally. However, the UI becomes noticeably sluggish, sometimes taking 20–30 seconds to render pages.
Here’s a simplified version of my setup:
```
version: "3.8"
networks:
airflow-net:
driver: bridge
volumes:
airflow-git:
driver: local
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME}
user: "${AIRFLOW_UID:-50000}:0"
restart: unless-stopped
environment:
# ---- Core ----
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__STORE_SERIALIZED_DAGS: True
AIRFLOW__CORE__LOAD_EXAMPLES: False
# ---- Database ----
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
AIRFLOW__CELERY__RESULT_BACKEND: "db+mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
# ---- Pooling ----
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE: 30
AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_TIMEOUT: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE: 1800
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING: True
# ---- Scheduler ----
AIRFLOW__SCHEDULER__PARSING_PROCESSES: 1
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 120
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 300
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: 64
# ---- Logging ----
AIRFLOW__LOGGING__REMOTE_LOGGING: True
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://example-bucket
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws_default
# ---- Celery ----
AIRFLOW__CELERY__BROKER_URL: redis://redis.example.net:6379/2
AIRFLOW__CELERY__TASK_TRACK_STARTED: True
AIRFLOW__CELERY__PREFETCH_MULTIPLIER: 1
TZ: Asia/Seoul
volumes:
- airflow-git:/opt/airflow/git
- ./airflow_local_settings.py:/opt/airflow/airflow_local_settings.py:ro
networks: [airflow-net]
services:
airflow-apiserver:
<<: *airflow-common
command: >
bash -c 'exec airflow api-server --host 0.0.0.0 --port 8080 --workers 1 --apps all'
ports: ["8080:8080"]
depends_on:
airflow-gitsync: { condition: service_healthy }
airflow-init: { condition: service_completed_successfully }
```
Environment summary
• Airflow version: 3.1.1
• Executor: CeleryExecutor
• DB: MySQL on AWS RDS (db.r7g.large)
• Redis: hosted Redis cluster (for Celery broker)
• DAG count: ~90
• UI: AWS loadbalancer using HTTPS
• Serialized DAGs: enabled (STORE_SERIALIZED_DAGS=True)
The Airflow UI is slow:
• DAG list takes 10–20 seconds to load
• Clicking into a DAG detail or “Graph View” is also very sluggish
• Scheduler and workers seem healthy, tasks execute fine
• Database CPU/memory usage is normal
GitHub link: https://github.com/apache/airflow/discussions/57933
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
GitHub user marcosmartinezfco added a comment to the discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
Same problem here, also with a docker compose setup and multiple celery workers spanning different nodes. Currently running on `3.1.0` so will try the proposed version.
GitHub link: https://github.com/apache/airflow/discussions/57933#discussioncomment-15220579
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "gitthangbaby (via GitHub)" <gi...@apache.org>.
GitHub user gitthangbaby added a comment to the discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I understand. We have hundreds of tasks rathern than dags, and i feel bad i am being pad to do scrolling all day. My worst UI glitch is that once I find a task and click on it, it scrolls the task list back to the beginning. Progressive reveal of tasks and logs, is so unfit here, i am still speechless. Good luck!
GitHub link: https://github.com/apache/airflow/discussions/57933#discussioncomment-19132009
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
Try 3.1.2 released yesterday. There were number of n+1 (many database queries when querying for dags) issues solved in 3.1.2 - see the changelog.
GitHub link: https://github.com/apache/airflow/discussions/57933#discussioncomment-14889590
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "jun-roh (via GitHub)" <gi...@apache.org>.
GitHub user jun-roh edited a discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I’m running Apache Airflow 3.1.1 using Docker Compose, and I’ve noticed that the Web UI is extremely slow when loading data — especially the main DAG list and DAG details pages.
We currently have about 90 DAGs, and everything else (scheduler, workers, etc.) seems to be functioning normally. However, the UI becomes noticeably sluggish, sometimes taking 20–30 seconds to render pages.
Here’s a simplified version of my setup:
`
version: "3.8"
networks:
airflow-net:
driver: bridge
volumes:
airflow-git:
driver: local
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME}
user: "${AIRFLOW_UID:-50000}:0"
restart: unless-stopped
environment:
# ---- Core ----
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__STORE_SERIALIZED_DAGS: True
AIRFLOW__CORE__LOAD_EXAMPLES: False
# ---- Database ----
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
AIRFLOW__CELERY__RESULT_BACKEND: "db+mysql+pymysql://${AIRFLOW_DB_USER}:${AIRFLOW_DB_PASSWORD_URL}@${AIRFLOW_DB_HOST}/${AIRFLOW_DB_NAME}"
# ---- Pooling ----
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE: 30
AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_TIMEOUT: 60
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE: 1800
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING: True
# ---- Scheduler ----
AIRFLOW__SCHEDULER__PARSING_PROCESSES: 1
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 120
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 300
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: 64
# ---- Logging ----
AIRFLOW__LOGGING__REMOTE_LOGGING: True
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://example-bucket
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws_default
# ---- Celery ----
AIRFLOW__CELERY__BROKER_URL: redis://redis.example.net:6379/2
AIRFLOW__CELERY__TASK_TRACK_STARTED: True
AIRFLOW__CELERY__PREFETCH_MULTIPLIER: 1
TZ: Asia/Seoul
volumes:
- airflow-git:/opt/airflow/git
- ./airflow_local_settings.py:/opt/airflow/airflow_local_settings.py:ro
networks: [airflow-net]
services:
airflow-apiserver:
<<: *airflow-common
command: >
bash -c 'exec airflow api-server --host 0.0.0.0 --port 8080 --workers 1 --apps all'
ports: ["8080:8080"]
depends_on:
airflow-gitsync: { condition: service_healthy }
airflow-init: { condition: service_completed_successfully }
`
Environment summary
• Airflow version: 3.1.1
• Executor: CeleryExecutor
• DB: MySQL on AWS RDS (db.r7g.large)
• Redis: hosted Redis cluster (for Celery broker)
• DAG count: ~90
• UI: AWS loadbalancer using HTTPS
• Serialized DAGs: enabled (STORE_SERIALIZED_DAGS=True)
The Airflow UI is slow:
• DAG list takes 10–20 seconds to load
• Clicking into a DAG detail or “Graph View” is also very sluggish
• Scheduler and workers seem healthy, tasks execute fine
• Database CPU/memory usage is normal
GitHub link: https://github.com/apache/airflow/discussions/57933
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "gitthangbaby (via GitHub)" <gi...@apache.org>.
GitHub user gitthangbaby added a comment to the discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
I have to work with it, Since version 3 it is unbelievably slow til current version 3.2.1. Together with the progressive scrolling, wow, so bad experience. Click -> 30 seconds -> Click -> 30 seconds -> scroll -> 10 seconds -> scroll etc. A simple Ctrl-F requires dozens of scrolls up and down to be able to search in the list of tasks and in the log.
GitHub link: https://github.com/apache/airflow/discussions/57933#discussioncomment-18189061
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose) [airflow]
Posted by "marcosmartinezfco (via GitHub)" <gi...@apache.org>.
GitHub user marcosmartinezfco added a comment to the discussion: Airflow 3.1.1 UI extremely slow when loading DAGs (running on Docker Compose)
yeah in my case we have around 700 dags and the search is quiete slow, plus any action on the main dag page. Once you click inside the dag it's fine.
Currently on `3.1.8` and we are not really planning to move so far as it's just a known pain but due our industry we don't want any trouble updating
GitHub link: https://github.com/apache/airflow/discussions/57933#discussioncomment-19130374
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org