You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pierrejeambrun (via GitHub)" <gi...@apache.org> on 2025/12/03 15:16:15 UTC
[PR] Add UI api performance tips [airflow]
pierrejeambrun opened a new pull request, #59004:
URL: https://github.com/apache/airflow/pull/59004
Doc only portion of:
https://github.com/apache/airflow/pull/58814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add UI/API performance tips [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun merged PR #59004:
URL: https://github.com/apache/airflow/pull/59004
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add UI/API performance tips [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59004:
URL: https://github.com/apache/airflow/pull/59004#issuecomment-3612611312
### Backport failed to create: v3-1-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/19933001956'> Run details </a>
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-1-test</td>
<td><a href="https://github.com/apache/airflow/commit/0a1e798fb71520f5848583f63c006c97c6ea7065"><img src='https://img.shields.io/badge/Commit-0a1e798-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 0a1e798 v3-1-test
```
This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add UI/API performance tips [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #59004:
URL: https://github.com/apache/airflow/pull/59004#issuecomment-3613046022
Manual backport https://github.com/apache/airflow/pull/57719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add UI api performance tips [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #59004:
URL: https://github.com/apache/airflow/pull/59004#discussion_r2587622267
##########
airflow-core/docs/howto/performance.rst:
##########
@@ -0,0 +1,55 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Performance tuning (API and UI)
+===============================
+
+This guide collects pragmatic tips that improve Airflow performance for API and UI workloads.
+
+Custom metadata indexes
+-----------------------
+
+If you observe slowness in some API calls or specific UI views, you should inspect query plans and add indexes yourself
+that match your workload. Listing endpoints and UI table views with specific ordering criteria are likely
+to benefit from additional indexes if you have a large volume of metadata.
+
+When to use
+^^^^^^^^^^^
+
+- Slow API list/detail endpoints caused by frequent scans or lookups on columns like ``start_date``, timestamps (e.g. ``dttm``), or status fields.
+- UI pages that load large lists or perform heavy filtering on metadata tables.
+
+Guidance
+^^^^^^^^
+
+- Inspect the query planner (e.g., ``EXPLAIN``/``EXPLAIN ANALYZE``) for slow endpoints and identify missing indexes.
+- Prefer single or composite indexes that match your most common ordering logic, typically the ``order_by``
+ query parameter used in API calls. Composite indexes can cover multi criteria ordering.
+- Your optimal indexes depend on how you use the API and UI; there is no one-size-fits-all set we can ship by default.
+
+Upgrade considerations
+^^^^^^^^^^^^^^^^^^^^^^
+
+To avoid conflicts with Airflow database upgrades, delete your custom indexes before running an Airflow DB upgrade
+and re-apply them after the upgrade succeeds.
+
+Notes
+^^^^^
+
Review Comment:
Do we need to mention
```
Airflow database :doc:`../database-erd-ref`
```
here in the performance section as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add UI/API performance tips [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #59004:
URL: https://github.com/apache/airflow/pull/59004#discussion_r2589091861
##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -252,11 +252,6 @@ def to_orm(self, select: Select) -> Select:
if column is None:
column = getattr(self.model, lstriped_orderby)
- # MySQL does not support `nullslast`, and True/False ordering depends on the
- # database implementation.
- nullscheck = case((column.isnot(None), 0), else_=1)
-
- columns.append(nullscheck)
Review Comment:
Removed because that 'case statement' prevent usage of columns indexes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add UI/API performance tips [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #59004:
URL: https://github.com/apache/airflow/pull/59004#issuecomment-3622373424
Nice. This is definitely the best choice to leave things in the hands of users in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org