You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "antonlin1 (via GitHub)" <gi...@apache.org> on 2026/05/06 21:07:44 UTC
[I] [3.2] API server pool exhaustion from per-request RevokedToken.is_revoked DB query [airflow]
antonlin1 opened a new issue, #66493:
URL: https://github.com/apache/airflow/issues/66493
### Apache Airflow version
3.2.0 (introduced in commit b3306f15cd, "AIP-84: Add JWT token revokation for logout invalidation", PR #61339 / #47952)
### What happened?
Every authenticated API request now performs a synchronous DB query inside the FastAPI auth dependency:
```python
# airflow-core/src/airflow/api_fastapi/auth/managers/base_auth_manager.py:153
if (jti := payload.get("jti")) and RevokedToken.is_revoked(jti):
raise InvalidTokenError("Token has been revoked")
```
`RevokedToken.is_revoked` ([revoked_token.py:58-61](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/revoked_token.py#L58-L61)) runs `session.scalar(...)` via `@provide_session`, holding a SQLAlchemy connection per in-flight request. With the default pool of `5+10=15` shared across api-server, scheduler, dag-processor, and triggerer, modest concurrent load (UI multi-endpoint polling, fan-out DAGs) exhausts the pool and request handlers time out in `QueuePool._do_get` after 30s.
The UI freezes once a few task instances start running because every poll request blocks on connection checkout. Stacktrace from a stock 3.2.0 standalone:
```
File "airflow/api_fastapi/auth/managers/base_auth_manager.py", line 153, in get_user_from_token
if (jti := payload.get("jti")) and RevokedToken.is_revoked(jti):
File "airflow/utils/session.py", line 100, in wrapper
return func(*args, session=session, **kwargs)
File "airflow/models/revoked_token.py", line 61, in is_revoked
return bool(session.scalar(select(exists().where(cls.jti == jti))))
...
File "sqlalchemy/pool/impl.py", line 166, in _do_get
raise exc.TimeoutError(
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00
```
`is_revoked` was the bottom of the failing stack on every endpoint we hit: `/ui/config`, `/ui/backfills`, `/api/v2/dags/.../details`, `/api/v2/dags/.../dagRuns/...`. Multi-second `duration_us` values (60s, 90s, 120s, 150s) come from FastAPI resolving the auth dependency multiple times in the same handler — each checkout times out at 30s independently.
### What you think should happen instead?
Cache hit rate on `is_revoked` is ≈100% in practice — revocation only happens on explicit logout. The check should not require a DB roundtrip on every request. An in-process TTL cache (with bounded staleness across uvicorn workers) collapses the per-request DB roundtrip into a near-free in-memory lookup.
### How to reproduce
Stock 3.2.0 with the default config:
```bash
pip install apache-airflow==3.2.0
AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE=3 \
AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW=2 \
airflow standalone &
# Get a JWT (admin password is in ~/airflow/simple_auth_manager_passwords.json.generated):
JWT=$(curl -sf -X POST http://localhost:8080/auth/token \
-H 'Content-Type: application/json' \
-d '{"username":"admin","password":"<PWD>"}' \
| python3 -c 'import json,sys;print(json.load(sys.stdin)["access_token"])')
# 60 concurrent requests against an authenticated DB-backed endpoint:
seq 60 | xargs -P 30 -I {} curl -sS -o /dev/null \
-w "%{http_code} %{time_total}s\n" \
-H "Authorization: Bearer $JWT" \
http://localhost:8080/api/v2/dags
```
Observed: ~12% of requests return 500 with `QueuePool TimeoutError`, p50 latency ~15s, p99 ~30s.
### Operating System
macOS 24.4.0 (also reproduces on Linux per the standard SQLAlchemy/uvicorn pool dynamics).
### Versions of Apache Airflow Providers
N/A — affects core auth path.
### Deployment
Standalone (issue is independent of deployment topology — affects any uvicorn-driven api-server with a shared SQLAlchemy pool).
### Deployment details
- Default `[database]` pool config (5+10) reproduces with sufficient concurrent request volume; deliberately small pool (3+2) makes it deterministic.
- Backed by SQLite (default standalone) but the failure is at the pool-checkout level, not SQLite-specific.
### Anything else?
- Bisected to commit b3306f15cd (PR #47952 / #61339, "AIP-84: Add JWT token revokation for logout invalidation"). Pre-3.2 the auth path did no DB work.
- Pull request with proposed fix coming next (in-process `cachetools.TTLCache` mirroring the existing `DBDagBag` cache pattern in `airflow/models/dagbag.py`). Cache hit rate is ≈100% in practice (revocation is rare). Local before/after with the fix at the same `pool_size=3, max_overflow=2`: 60 requests / 30 concurrent dropped from 31s wall + 12% timeouts + 30s p99 → 1.04s wall + 0% timeouts + 0.6s p99.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [3.2] API server pool exhaustion from per-request RevokedToken.is_revoked DB query [airflow]
Posted by "antonlin1 (via GitHub)" <gi...@apache.org>.
antonlin1 closed issue #66493: [3.2] API server pool exhaustion from per-request RevokedToken.is_revoked DB query
URL: https://github.com/apache/airflow/issues/66493
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org