You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ds...@apache.org on 2025/12/11 16:56:30 UTC
(airflow) branch main updated: Add todo re exception handling in dagrun create loop (#59121)
This is an automated email from the ASF dual-hosted git repository.
dstandish pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new c9e190fb254 Add todo re exception handling in dagrun create loop (#59121)
c9e190fb254 is described below
commit c9e190fb25485c50f7d2b9042ede735413cd1935
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Thu Dec 11 08:56:19 2025 -0800
Add todo re exception handling in dagrun create loop (#59121)
In SchedulerJobRunner, in _create_dag_runs, we do except Exception: ... continue but this does not work when the exception is the result of a db error; in that case you need to roll the transaction back.
To fix this we need either smaller transactions (e.g. one per dag run create attempt) or savepoints. either way it's a bit of a refactor.
Issue: https://github.com/apache/airflow/issues/59120
---
airflow-core/src/airflow/jobs/scheduler_job_runner.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/airflow-core/src/airflow/jobs/scheduler_job_runner.py b/airflow-core/src/airflow/jobs/scheduler_job_runner.py
index 5d8df8e970e..ceff2ff5ca6 100644
--- a/airflow-core/src/airflow/jobs/scheduler_job_runner.py
+++ b/airflow-core/src/airflow/jobs/scheduler_job_runner.py
@@ -1831,6 +1831,10 @@ class SchedulerJobRunner(BaseJobRunner, LoggingMixin):
# and continue to the next serdag.
except Exception:
self.log.exception("Failed creating DagRun for %s", serdag.dag_id)
+ # todo: continuing here does not work because session needs rollback
+ # but you need either to make smaller transactions and commit after every dag run
+ # or to use savepoints.
+ # https://github.com/apache/airflow/issues/59120
continue
if self._should_update_dag_next_dagruns(
serdag,