You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pierrejeambrun (via GitHub)" <gi...@apache.org> on 2026/02/16 15:29:52 UTC
[I] API - Improve public get_task_instances endpoint [airflow]
pierrejeambrun opened a new issue, #62027:
URL: https://github.com/apache/airflow/issues/62027
### Body
Similarly to https://github.com/apache/airflow/issues/62025
On big installation the "get_task_instances" endpoint to list all the dagruns in the UI is taking a long time to return a response.
The critical part of the code is the actual db query. Most likely due to the number of joins creating rows explosion. We need to optimize the query, most likely by double checking the eager loading options and verify that there is no row explosion cause by a wrong option (joinedload vs selectinload). Also we can add `load_only` to limit the number of columns selected in each joins.
https://github.com/apache/airflow/blob/4e75919198be6b8e4d3ac52656a9b74e0e73b097/airflow-core/src/airflow/api_fastapi/core_api/routes/public/task_instances.py#L483-L533
Also needs investigation to check if an extra index could help.
https://github.com/user-attachments/assets/6cfc5f15-65d1-4a89-bd5c-5e1ff5cb5c99
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API - Improve public get_task_instances endpoint [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #62027:
URL: https://github.com/apache/airflow/issues/62027#issuecomment-3915403920
@kevinhongzl Please go ahead, you don't need to be assigned to start working on an issue or open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API - Improve public get_task_instances endpoint [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun closed issue #62027: API - Improve public get_task_instances endpoint
URL: https://github.com/apache/airflow/issues/62027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API - Improve public get_task_instances endpoint [airflow]
Posted by "Arunodoy18 (via GitHub)" <gi...@apache.org>.
Arunodoy18 commented on issue #62027:
URL: https://github.com/apache/airflow/issues/62027#issuecomment-3919037340
Hi! I’ve been looking into this issue and would like to work on it.
I’m planning to investigate:
- Reducing duplicate / nested joins in the TaskInstance query
- Improving SQLAlchemy query construction while preserving API behaviour
- Measuring performance impact before and after changes
Please let me know if there are any preferred directions or constraints I should keep in mind.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API - Improve public get_task_instances endpoint [airflow]
Posted by "kevinhongzl (via GitHub)" <gi...@apache.org>.
kevinhongzl commented on issue #62027:
URL: https://github.com/apache/airflow/issues/62027#issuecomment-3909658461
I'd like to work on this issue!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API - Improve public get_task_instances endpoint [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #62027:
URL: https://github.com/apache/airflow/issues/62027#issuecomment-3909285267
This will also fix https://github.com/apache/airflow/issues/61212 which is similar where are multiple joins to the same query. Some of the things we noticed in our testing but yet to find any solution in our setup using mysql 8.x.
1. `TaskInstance.start_date` is not indexed by default but the UI by default sorts by it on load `http://localhost:8000/api/v2/dags/~/dagRuns/~/taskInstances?limit=10&offset=0&order_by=-start_date&order_by=-run_after` .
2. `TaskInstance.id` is a primary key which is a UUID. UUID is not a native type in mysql but is stored by varchar(32). Postgres has native UUID support. Ref https://stackoverflow.com/questions/2365132/uuid-performance-in-mysql .
3. Removing the nested joins of same table and removing sorting had greatly improved performance but I was not able to update the relevant SQLAlchemy parts in the API endpoint to get there. Also sorting is an important element where users want to know the recent task instances which we are yet to tackle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API - Improve public get_task_instances endpoint [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #62027:
URL: https://github.com/apache/airflow/issues/62027#issuecomment-4153716945
Closing for now. The bottleneck is now on the Count(*) required for pagination. If we need to go further we will have to probably change the pagination to 'offset' pagination or do some bigger change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org