You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "steveahnahn (via GitHub)" <gi...@apache.org> on 2026/08/09 02:08:10 UTC
[PR] Stop airflow db clean deleting triggers that are still in use [airflow]
steveahnahn opened a new pull request, #71339:
URL: https://github.com/apache/airflow/pull/71339
## Why
`airflow db clean` selects `trigger` rows by age alone, with no check for rows still in use.
The triggerer already removes every unreferenced trigger on each loop, so an old trigger row that survives is almost always still doing work. Three foreign keys point at `trigger`, and cleaning by age hits all three:
| referencing table | ON DELETE | effect of the delete |
| --- | --- | --- |
| `task_instance` | `CASCADE` | a running deferred task is deleted |
| `asset_watcher` | `CASCADE` | an event-driven watcher is deleted |
| `callback` | no rule | the delete fails and stops the command |
Only the `trigger` rows are copied to the archive table, so the cascade-deleted task instance reaches no archive and `db export-archived` cannot recover it. Asset watchers are created once and reused, so their `created_date` ages past any retention window and every clean removes them until the next Dag parse recreates them.
Verified on Postgres. Four old triggers, three of them in use, and the cleanup reports `Found 4 rows meeting deletion criteria`:
| | before | after |
| --- | --- | --- |
| deferred `task_instance` rows | 0 | 1 |
| `asset_watcher` rows | 0 | 1 |
`dag_version` already solves this with `skip_if_referenced` (#68339). This applies the same guard to `trigger`, so only genuinely orphaned rows are purged.
## MySQL
`trigger` is a reserved word on MySQL, and the archive step interpolated the table name into `CREATE TABLE ... LIKE` unquoted. That is a syntax error, so the `trigger` table could never be cleaned on MySQL at all. `ProgrammingError` is suppressed per table, so it surfaced only as a warning.
The two changes ship together because they share a code path: the archive table is always created, so the reference guard alone would still fail on MySQL.
## Tests
Both new tests fail without the change and pass with it. Full file: 74 pass on Postgres, 73 on MySQL, 72 on SQLite.
related: #56192
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Stop airflow db clean deleting triggers that are still in use [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #71339: Stop airflow db clean deleting triggers that are still in use
URL: https://github.com/apache/airflow/pull/71339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Stop airflow db clean deleting triggers that are still in use [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #71339:
URL: https://github.com/apache/airflow/pull/71339#issuecomment-5839281353
Hello @steveahnahn - thank you for your contributions to Apache Airflow!
The Airflow community has introduced a limit of **5 open pull requests at a time** for contributors without write access to the repository. You currently have 14 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
* #68595 - Bound the orphaned asset_state_store cleanup delete in the scheduler
* #69281 - Resolve dag versions once per scheduler loop
* #71339 - Stop airflow db clean deleting triggers that are still in use
* #71861 - Migrate Execution API responses that older Task SDK clients still read
* #71914 - Make LocalExecutor hold a parallelism slot per dispatched workload
* #72668 - Stop trigger rule evaluation removing valid mapped task instances
* #72669 - Fail the run visibly when a task vanishes from the Dag while unfinished
* #72672 - Stop the Execution API lazy-loading asset event relations per event
* #72979 - Stop airflow db downgrade corrupting deadline alerts it cannot convert
* #72980 - Convert team references to names when upgrading past the team_id removal
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
* #68423 - Reduce task group query work in Execution API
* #69518 - Resolve parse-batch run info with a constant number of queries
* #69586 - Cancel the Dataflow job when a user kills the deferred task
* #71862 - Keep asset-event Dag run start_date non-null for older Task SDK clients
This is **not** a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.
What we ask you to do is to make your **first prioritization decision**: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or `gh pr reopen <PR_NUMBER> --repo apache/airflow`. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.
While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.
Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org