You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "uplsh580 (via GitHub)" <gi...@apache.org> on 2026/08/11 02:18:02 UTC
[I] First authenticated request after MySQL idle disconnect returns 500: RevokedToken.is_revoked runs on a stale session before FAB's recovery (#62919) can kick in [airflow]
uplsh580 opened a new issue, #71395:
URL: https://github.com/apache/airflow/issues/71395
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.2.2
### What happened and how to reproduce it?
With MySQL as the metadata DB and the FAB auth manager, after the api-server sits idle
longer than MySQL's `wait_timeout`, the **first** authenticated request (any UI or REST
endpoint) fails with HTTP 500. The exception is an `OperationalError` (MySQL error 4031,
"The client was disconnected by the server because of inactivity") raised from
`RevokedToken.is_revoked` inside `BaseAuthManager.get_user_from_token`:
https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/api_fastapi/auth/managers/base_auth_manager.py#L152
The immediately following request succeeds, because the failed query poisons and
effectively resets the pooled connection. In production this shows up as intermittent
500s on the first request after overnight idle periods.
<details>
<summary>Traceback (trimmed)</summary>
```
File ".../airflow/api_fastapi/auth/managers/base_auth_manager.py", line 152, in get_user_from_token
if (jti := payload.get("jti")) and RevokedToken.is_revoked(jti):
File ".../airflow/utils/session.py", line 100, in wrapper
return func(*args, session=session, **kwargs)
File ".../airflow/models/revoked_token.py", line 61, in is_revoked
return bool(session.scalar(select(exists().where(cls.jti == jti))))
...
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (4031,
'The client was disconnected by the server because of inactivity.
See wait_timeout and interactive_timeout for configuring this behavior.')
```
</details>
### How to reproduce
1. Run Airflow 3.2.x with MySQL as the metadata DB and the FAB auth manager.
2. Lower the idle timeout to make the repro fast: `SET GLOBAL wait_timeout = 60;`
3. Make any authenticated API request (this warms up the pooled connection).
4. Wait longer than `wait_timeout` (e.g. 90 s) without touching the api-server.
5. Repeat the same request → HTTP 500 with `OperationalError` 4031 raised from
`RevokedToken.is_revoked`. Repeat once more → succeeds.
### What you think should happen instead?
#### Root cause
1. The FAB auth manager shares core's scoped `settings.Session`.
`FabAuthManager.deserialize_user` leaves a transaction open after the request is
served, so the connection stays checked out across requests. Once it idles past
MySQL's `wait_timeout`, the server drops it (error 4031). Because the connection is
never returned to the pool, there is no checkout event — `pool_pre_ping` /
`pool_recycle` cannot help.
2. This exact failure mode was reported in #62903 and fixed by #62919, which added a
discard-and-retry recovery to `FabAuthManager.deserialize_user`.
3. However, 3.2.0 introduced the JWT revocation check (#61339 / AIP-84):
`RevokedToken.is_revoked` in `get_user_from_token` now runs **before**
`deserialize_user`, so it is the first thing to touch the poisoned scoped session —
and it has no recovery logic. The 500 that #62919 fixed is back, just raised one
step earlier in the auth path.
### What you think should happen instead?
The first request after an idle disconnect should recover transparently, as it does in
`deserialize_user` since #62919. The revoked-token check should apply the same
recovery: on `SQLAlchemyError`, discard the scoped session (`settings.Session.remove()`)
and retry once on a fresh connection. Something like:
```python
if jti := payload.get("jti"):
try:
revoked = RevokedToken.is_revoked(jti)
except SQLAlchemyError:
log.warning(
"Revoked-token check failed on a stale DB session; "
"discarding the session and retrying once.",
exc_info=True,
)
with suppress(Exception):
settings.Session.remove()
revoked = RevokedToken.is_revoked(jti)
if revoked:
raise InvalidTokenError("Token has been revoked")
```
Alternatively, the recovery could be hoisted so it covers the whole
`get_user_from_token` path instead of being duplicated per call site.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow-providers-fab==3.6.4 (already includes the #62919 fix — it is bypassed, not missing)
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
1.25
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
- Occurs deterministically whenever the first request after an idle period touches a
connection older than `wait_timeout`; looks intermittent in production.
- Related prior work on the same class of failure:
- #62903 / #62919 — same disconnect, recovery added to `deserialize_user` (this
check runs earlier and bypasses it)
- #62335 / #62336 — `Session.remove()` hardening in FAB's cleanup middleware
- #66493 / #66494 — same call site, different concern (per-request DB roundtrip /
pool exhaustion)
- We have been running the retry patch above in production for several weeks
(applied at image build time) and it eliminated these 500s entirely.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org