You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shivaam (via GitHub)" <gi...@apache.org> on 2026/10/20 16:36:02 UTC
[PR] Speed up trigger cleanup callback lookups [airflow]
shivaam opened a new pull request, #75228:
URL: https://github.com/apache/airflow/pull/75228
Trigger cleanup checks `callback.trigger_id` before deleting an unreferenced trigger. Unlike the
equivalent task-instance and asset-watcher columns, this callback column has no explicit Airflow
index on PostgreSQL or SQLite, so those checks can scan the callback table.
This is a follow-up to #70961, where the missing `callback.trigger_id` index was left as separate
work.
This adds a non-unique `idx_callback_trigger_id` index. PostgreSQL and SQLite use a partial index
containing only non-null trigger references. MySQL uses a full index because it does not support
equivalent partial indexes; InnoDB already creates an index for the foreign key, so the migration
standardizes its name without adding a duplicate index.
The column remains nullable and non-unique. Callback lifecycle and trigger-cleanup SQL are
unchanged.
In a local PostgreSQL benchmark with one million callback rows and one percent non-null references,
the cleanup query improved from 12.1 seconds without an index to about 7.9 milliseconds with either
indexed shape. The partial index was about 29 times smaller than the full index (240 KiB versus 6.8
MiB).
Testing performed:
- Upgrade, downgrade, and re-upgrade tests on SQLite, PostgreSQL 14, and MySQL 8.0
- SQLite cleanup-query plan regression test
- Migration stairway, schema synchronization, migration-head, and pending-upgrade checks
- Complete changed-file `prek` suite
Trade-offs:
- Maintaining the index adds the normal write cost when callback trigger references are inserted
or changed.
- The migration uses ordinary `CREATE INDEX`, consistent with existing Airflow migrations, so it
may temporarily block conflicting writes while the index is created.
related: #68243
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Codex (GPT-5) and Claude Code (Opus 5.5)
Generated-by: Codex (GPT-5) and Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add an Index on trigger_id to callback table to speed up trigger cleanup lookups [airflow]
Posted by "eladkal (via GitHub) via commits" <co...@airflow.apache.org>.
eladkal merged PR #75228:
URL: https://github.com/apache/airflow/pull/75228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add an Index on trigger_id to callback table to speed up trigger cleanup lookups [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #75228:
URL: https://github.com/apache/airflow/pull/75228#issuecomment-6315785458
Also paging experts on this @dstandish @seanmuth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org