You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "arkadiuszbach (via GitHub)" <gi...@apache.org> on 2026/10/12 20:12:22 UTC
[I] Triggerer restarts after a transient metadata database disconnect [airflow]
arkadiuszbach opened a new issue, #74614:
URL: https://github.com/apache/airflow/issues/74614
On Airflow 3.2.2 (Python 3.12, PostgreSQL), a brief interruption to the metadata database connection caused the triggerer to exit and its pod to restart during `Trigger.clean_unused()`.
This issue is also present on `main` at commit [`194e0ea4`](https://github.com/apache/airflow/blob/194e0ea4c9890a7eed1b6df19eb72159f3e44841/airflow-core/src/airflow/models/trigger.py#L233): `Trigger.clean_unused()` retries its UPDATE using `run_with_db_retries()`, which does not roll back the session between attempts. After a connection loss invalidates the transaction, the next attempt reuses that invalid transaction and raises `PendingRollbackError`.
```text
2026-09-14T06:21:13.416868Z [error ] Exception when executing TriggerRunnerSupervisor.run [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:221
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 218, in _execute
self.trigger_runner.run()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 614, in run
self.clean_unused()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 661, in clean_unused
Trigger.clean_unused()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 100, in wrapper
return func(*args, session=session, **kwargs) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/trigger.py", line 230, in clean_unused
for attempt in run_with_db_retries():
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 438, in __iter__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 393, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/trigger.py", line 232, in clean_unused
session.execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1660, in orm_execute_statement
return super().orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
result = conn.execute(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
return meth(
^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
conn = self._revalidate_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
self._invalid_transaction()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
2026-09-14T06:21:13.421641Z [info ] Waiting for triggers to clean up [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:224
2026-09-14T06:21:13.527625Z [info ] Process exited [supervisor] exit_code=0 loc=supervisor.py:860 pid=29 signal_sent=SIGINT
2026-09-14T06:21:13.527858Z [info ] Exited trigger loop [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:229
```
PR [#70961](https://github.com/apache/airflow/pull/70961) may address this incident because its cleanup retries use a fresh database session for each attempt.
---
Drafted-by: Codex (GPT-6.1 Sol); reviewed by @arkadiuszbach before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggerer restarts after a transient metadata database disconnect [airflow]
Posted by "RishabhBajaj97 (via GitHub)" <gi...@apache.org>.
RishabhBajaj97 commented on issue #74614:
URL: https://github.com/apache/airflow/issues/74614#issuecomment-6144294746
I would like to take a stab at this.
---
Drafted-by: Claude Code (Opus 4.7) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggerer restarts after a transient metadata database disconnect [airflow]
Posted by "RishabhBajaj97 (via GitHub)" <gi...@apache.org>.
RishabhBajaj97 commented on issue #74614:
URL: https://github.com/apache/airflow/issues/74614#issuecomment-6146778889
Minimal fix raised: #74650. It adds `session.rollback()` on `DBAPIError`/`StaleDataError` inside the existing `run_with_db_retries()` block in `Trigger.clean_unused`, matching the pattern already used by `retry_db_transaction` in `airflow/utils/retries.py`. 2-file change, +57/-8, with a regression test that fails without the fix and passes with it.
For maintainer awareness: #70961 is an in-review PR that batches `clean_unused`. Its restructure to `with create_session(scoped=False)` per retry attempt would incidentally close this issue as a side effect, because each attempt gets a fresh session. So there are two viable paths:
1. Merge #74650 now as a targeted fix, independent of the batching PR's review timeline.
2. Treat this issue as subsumed by #70961 and close #74650 as redundant once #70961 lands.
Happy either way — flagging the overlap so reviewers can pick without duplicate effort. Cross-reference comment posted on #70961 as well.
---
Drafted-by: Claude Code (Opus 4.7) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggerer restarts after a transient metadata database disconnect [airflow]
Posted by "RishabhBajaj97 (via GitHub)" <gi...@apache.org>.
RishabhBajaj97 commented on issue #74614:
URL: https://github.com/apache/airflow/issues/74614#issuecomment-6162493675
@arkadiuszbach can I get a review on https://github.com/apache/airflow/pull/74650, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org