You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "boqiang (via GitHub)" <gi...@apache.org> on 2026/10/14 19:48:14 UTC
[PR] Retry scheduled DagRun creation after database errors [airflow]
boqiang opened a new pull request, #74761:
URL: https://github.com/apache/airflow/pull/74761
Database errors while loading a serialized Dag or creating a scheduled DagRun are caught by per-Dag handlers. Continuing with the failed SQLAlchemy Session can turn the original retryable error into `PendingRollbackError` and interrupt scheduling.
Propagate `DBAPIError` from both handlers to the existing `_create_dagruns_for_dags` transaction retry boundary. That boundary rolls back before querying candidates again and reacquiring their row locks. Keep ordinary per-Dag error handling and `CommitProhibitorGuard` unchanged. Persistent database errors still propagate after the configured retry limit; this change does not provide per-Dag isolation for persistent failures.
Add regression coverage for transient errors and retry exhaustion at both interception points. Add a MySQL-only test using two real InnoDB transactions to cause error 1213 during pending Dag metadata flush (with autoflush both enabled and disabled), after one DagRun has already been created in the batch. Check that the batch re-queries locked candidates, produces exactly one run per scheduled Dag, and preserves the concurrent transaction's metadata updates.
Related: #59120
Validation on Python 3.11 through Breeze (Linux amd64 CI image on Apple Silicon):
- MySQL 8.4: all 6 new regression cases passed, including native InnoDB error 1213 with autoflush enabled and disabled.
- SQLite and PostgreSQL 14: 4 passed and 2 MySQL-only cases skipped on each backend.
- Reverting only the production change on the same upstream base: all 6 MySQL cases failed with the expected `PendingRollbackError` following the original constraint error or deadlock.
- Full selective DB checks: API 4291 passed/16 skipped/4 baseline failures; Other 967 passed/8 skipped/1 baseline failure. Non-DB checks: 5343 passed/11 skipped/2 xfailed/2 baseline failures.
- Full MySQL scheduler test file: 566 passed, 2 skipped. PostgreSQL Core group: 3560 passed, 32 skipped, 1 xfailed. PostgreSQL CLI group: 673 passed. PostgreSQL Serialization: 62 passed. PostgreSQL Always: 1918 passed, 6 skipped.
- Ruff, airflow-core mypy, and manual static checks passed.
- Broader selective checks found five plugin-count failures that also reproduce with both PR files restored to the upstream base. The two non-DB LangSDK file-descriptor failures also reproduce with both PR files restored to that base (extra descriptors 3–6, including `/usr/bin/dash`). Documentation build (`breeze build-docs apache-airflow`) passed. These seven baseline failures keep the broader checks from being fully green; no unrelated changes or test skips are included.
These tests exercise the transaction boundary directly; they do not establish multi-scheduler process acceptance.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Codex (GPT-6)
Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Retry scheduled DagRun creation after database errors [airflow]
Posted by "boqiang (via GitHub)" <gi...@apache.org>.
boqiang commented on PR #74761:
URL: https://github.com/apache/airflow/pull/74761#issuecomment-6208964073
Thanks for catching the `StaleDataError` gap. Confirmed with a real failed ORM flush, and addressed in https://github.com/apache/airflow/commit/a53b3bc1d0682f3b0c03c52bad26df75cc982ae4.
Both handlers now propagate `(DBAPIError, StaleDataError)`, matching the existing retry boundary. The expanded regression uses a pending Pool UPDATE whose row has been deleted through the raw connection, so SQLAlchemy itself raises `StaleDataError` and invalidates the Session. It covers both interception points, transient recovery and retry exhaustion, the original exception identity, bounded retries, and restoration of the Pool data after rollback.
With only the production handlers restored to their pre-review version, all four new cases fail with `PendingRollbackError` after the original `StaleDataError`. With the fix, the expanded cases pass on SQLite and PostgreSQL (8 passed, 2 MySQL-only cases skipped on each). The complete MySQL 8.4 scheduler test file passes: 570 passed, 2 skipped, including both native InnoDB deadlock/autoflush cases. Ruff, airflow-core mypy, and fast/manual prek checks also pass.
`CommitProhibitorGuard` and the retry policy are unchanged. The PR description now includes these results and keeps the persistent-failure isolation and multi-scheduler validation limits explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Retry scheduled DagRun creation after database errors [airflow]
Posted by "boqiang (via GitHub)" <gi...@apache.org>.
boqiang commented on PR #74761:
URL: https://github.com/apache/airflow/pull/74761#issuecomment-6200076572
Could a maintainer please review and approve the pending CI workflows? All three workflows are awaiting approval and have not started. This PR only changes scheduler error propagation and regression tests; it does not modify workflow files. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Retry scheduled DagRun creation after database errors [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #74761:
URL: https://github.com/apache/airflow/pull/74761#issuecomment-6176944997
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org