You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "arkadiuszbach (via GitHub)" <gi...@apache.org> on 2026/10/12 20:12:22 UTC

[I] Triggerer restarts after a transient metadata database disconnect [airflow]

arkadiuszbach opened a new issue, #74614:
URL: https://github.com/apache/airflow/issues/74614

   On Airflow 3.2.2 (Python 3.12, PostgreSQL), a brief interruption to the metadata database connection caused the triggerer to exit and its pod to restart during `Trigger.clean_unused()`.
   
   This issue is also present on `main` at commit [`194e0ea4`](https://github.com/apache/airflow/blob/194e0ea4c9890a7eed1b6df19eb72159f3e44841/airflow-core/src/airflow/models/trigger.py#L233): `Trigger.clean_unused()` retries its UPDATE using `run_with_db_retries()`, which does not roll back the session between attempts. After a connection loss invalidates the transaction, the next attempt reuses that invalid transaction and raises `PendingRollbackError`.
   
   ```text
   2026-09-14T06:21:13.416868Z [error    ] Exception when executing TriggerRunnerSupervisor.run [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:221
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 218, in _execute
       self.trigger_runner.run()
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 614, in run
       self.clean_unused()
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 661, in clean_unused
       Trigger.clean_unused()
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 100, in wrapper
       return func(*args, session=session, **kwargs)  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/trigger.py", line 230, in clean_unused
       for attempt in run_with_db_retries():
                      ^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 438, in __iter__
       do = self.iter(retry_state=retry_state)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
       result = action(retry_state)
                ^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 393, in <lambda>
       self._add_action_func(lambda rs: rs.outcome.result())
                                        ^^^^^^^^^^^^^^^^^^^
     File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
       return self.__get_result()
              ^^^^^^^^^^^^^^^^^^^
     File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
       raise self._exception
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/trigger.py", line 232, in clean_unused
       session.execute(
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
       return self._execute_internal(
              ^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
       result: Result[Any] = compile_state_cls.orm_execute_statement(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1660, in orm_execute_statement
       return super().orm_execute_statement(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
       result = conn.execute(
                ^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
       return meth(
              ^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
       return connection._execute_clauseelement(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
       ret = self._execute_context(
             ^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
       conn = self._revalidate_connection()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
       self._invalid_transaction()
     File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
       raise exc.PendingRollbackError(
   sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.  Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
   2026-09-14T06:21:13.421641Z [info     ] Waiting for triggers to clean up [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:224
   2026-09-14T06:21:13.527625Z [info     ] Process exited                 [supervisor] exit_code=0 loc=supervisor.py:860 pid=29 signal_sent=SIGINT
   2026-09-14T06:21:13.527858Z [info     ] Exited trigger loop            [airflow.jobs.triggerer_job_runner.TriggererJobRunner] loc=triggerer_job_runner.py:229
   ```
   
   PR [#70961](https://github.com/apache/airflow/pull/70961) may address this incident because its cleanup retries use a fresh database session for each attempt.
   
   ---
   
   Drafted-by: Codex (GPT-6.1 Sol); reviewed by @arkadiuszbach before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Triggerer restarts after a transient metadata database disconnect [airflow]

Posted by "RishabhBajaj97 (via GitHub)" <gi...@apache.org>.
RishabhBajaj97 commented on issue #74614:
URL: https://github.com/apache/airflow/issues/74614#issuecomment-6144294746

   I would like to take a stab at this.
   
   ---
   Drafted-by: Claude Code (Opus 4.7) (no human review before posting)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Triggerer restarts after a transient metadata database disconnect [airflow]

Posted by "RishabhBajaj97 (via GitHub)" <gi...@apache.org>.
RishabhBajaj97 commented on issue #74614:
URL: https://github.com/apache/airflow/issues/74614#issuecomment-6146778889

   Minimal fix raised: #74650. It adds `session.rollback()` on `DBAPIError`/`StaleDataError` inside the existing `run_with_db_retries()` block in `Trigger.clean_unused`, matching the pattern already used by `retry_db_transaction` in `airflow/utils/retries.py`. 2-file change, +57/-8, with a regression test that fails without the fix and passes with it.
   
   For maintainer awareness: #70961 is an in-review PR that batches `clean_unused`. Its restructure to `with create_session(scoped=False)` per retry attempt would incidentally close this issue as a side effect, because each attempt gets a fresh session. So there are two viable paths:
   
   1. Merge #74650 now as a targeted fix, independent of the batching PR's review timeline.
   2. Treat this issue as subsumed by #70961 and close #74650 as redundant once #70961 lands.
   
   Happy either way — flagging the overlap so reviewers can pick without duplicate effort. Cross-reference comment posted on #70961 as well.
   
   ---
   Drafted-by: Claude Code (Opus 4.7) (no human review before posting)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Triggerer restarts after a transient metadata database disconnect [airflow]

Posted by "RishabhBajaj97 (via GitHub)" <gi...@apache.org>.
RishabhBajaj97 commented on issue #74614:
URL: https://github.com/apache/airflow/issues/74614#issuecomment-6162493675

   @arkadiuszbach can I get a review on https://github.com/apache/airflow/pull/74650, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org