You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pierrejeambrun (via GitHub)" <gi...@apache.org> on 2025/04/03 21:48:30 UTC
[PR] Improve Grid endpoint response time [airflow]
pierrejeambrun opened a new pull request, #48771:
URL: https://github.com/apache/airflow/pull/48771
Improve the Grid endpoint response time. There are plenty of other improvements that can be made but this PR focuses on the main bottleneck of the endpoint which is `fill_task_instance_summaries`.
Deserializing the serialized dag from the DB as well as computing `get_task_group_map` for the serialized dag is pretty expensive. More importantly it is done on the serdag for each TI. The serdag is actually the same as long as the version do not change and therefore caching both results are helping.
On the dag `nested_groups` with more than 20 dag runs, the endpoint went from:
- 1s5-2s
- 0.5 - 0.7s
More importantly this will not scale linearly with the number of tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Grid endpoint response time [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #48771:
URL: https://github.com/apache/airflow/pull/48771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve Grid endpoint response time [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #48771:
URL: https://github.com/apache/airflow/pull/48771#discussion_r2027793291
##########
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py:
##########
@@ -115,9 +117,11 @@ def _fill_task_group_map(
task_nodes[task_node.task_id] = {
"is_group": False,
"parent_id": parent_node.node_id if parent_node else None,
- "task_count": task_nodes[parent_node.node_id]["task_count"]
- if _is_task_node_mapped_task_group(parent_node) and parent_node
- else [1],
+ "task_count": (
+ task_nodes[parent_node.node_id]["task_count"]
+ if _is_task_node_mapped_task_group(parent_node) and parent_node
+ else [1]
+ ),
Review Comment:
```suggestion
"task_count": task_nodes[parent_node.node_id]["task_count"]
if _is_task_node_mapped_task_group(parent_node) and parent_node
else [1],
```
##########
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py:
##########
@@ -237,9 +255,9 @@ def fill_task_instance_summaries(
queued_dttm=ti_queued_dttm,
child_states=child_states,
task_count=_get_total_task_count(run_id, task_node_map[task_id]["task_count"], session),
- state=TaskInstanceState[overall_ti_state.upper()]
- if overall_ti_state != "no_status"
- else None,
+ state=(
+ TaskInstanceState[overall_ti_state.upper()] if overall_ti_state != "no_status" else None
+ ),
Review Comment:
```suggestion
state=TaskInstanceState[overall_ti_state.upper()]
if overall_ti_state != "no_status"
else None,
```
##########
airflow-core/src/airflow/api_fastapi/core_api/services/ui/grid.py:
##########
@@ -137,12 +141,14 @@ def _get_total_task_count(
run_id: str, task_count: list[int | MappedTaskGroup | MappedOperator], session: SessionDep
) -> int:
return sum(
- node
- if isinstance(node, int)
- else (
- DBBaseOperator.get_mapped_ti_count(node, run_id=run_id, session=session) or 0
- if isinstance(node, (MappedTaskGroup, MappedOperator))
- else node
+ (
+ node
+ if isinstance(node, int)
+ else (
+ DBBaseOperator.get_mapped_ti_count(node, run_id=run_id, session=session) or 0
+ if isinstance(node, (MappedTaskGroup, MappedOperator))
+ else node
+ )
Review Comment:
```suggestion
node
if isinstance(node, int)
else (
DBBaseOperator.get_mapped_ti_count(node, run_id=run_id, session=session) or 0
if isinstance(node, (MappedTaskGroup, MappedOperator))
else node
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org