You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "anmolxlight (via GitHub)" <gi...@apache.org> on 2026/07/22 10:24:06 UTC
[PR] Fix Execution API JWT reissue TOCTOU race [airflow]
anmolxlight opened a new pull request, #70227:
URL: https://github.com/apache/airflow/pull/70227
closes: #70222
## Problem
Intermittent `ExpiredSignatureError` / `Failed to validate JWT` on `/execution/...` for tasks that are not necessarily older than `[execution_api] jwt_expiration_time`.
Root cause is a TOCTOU race in `JWTReissueMiddleware`:
1. `JWTBearer` validates the bearer token (still valid) and caches claims on the ASGI request scope.
2. The request handler runs (e.g. heartbeat).
3. Middleware re-calls `avalidated_claims` on the same bearer string.
4. If the token crossed `exp` during the request, that second validation raises `ExpiredSignatureError`.
5. The exception is swallowed, so no `Refreshed-API-Token` header is returned.
6. The client's next call arrives with a fully expired token → 403, and after enough failed heartbeats the task is killed.
This matches the intermittent multi-task failures reported in #70222 and the long-running LocalExecutor case in #67939.
## Fix
Reuse the claims already validated and cached by `JWTBearer` (`request.scope["ti_token"]`) instead of re-parsing/re-validating the Authorization header. If remaining lifetime is below the existing refresh threshold, issue a fresh token from those claims.
No expiry leeway is added. Workload tokens remain non-refreshable.
## Tests
- Updated threshold reissue tests to populate the ASGI scope the way production does.
- Added a mid-request expiry regression test (custom route advances the clock past `exp` during the handler) asserting `Refreshed-API-Token` is still issued and `avalidated_claims` is not called by middleware.
## Local verification
```bash
uv run --project airflow-core pytest airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py -xvs
ruff check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py
ruff format --check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py
```
All 5 tests passed.
## Notes
Related open PR #68499 targets the same race for #67939. This PR is based on current `main`, includes ashb's requested mid-request time-travel regression shape, and closes #70222.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Execution API JWT reissue TOCTOU race [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #70227: Fix Execution API JWT reissue TOCTOU race
URL: https://github.com/apache/airflow/pull/70227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Execution API JWT reissue TOCTOU race [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #70227:
URL: https://github.com/apache/airflow/pull/70227#issuecomment-5839318407
Hello @anmolxlight - thank you for your contributions to Apache Airflow!
The Airflow community has introduced a limit of **5 open pull requests at a time** for contributors without write access to the repository. You currently have 10 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
* #70227 - Fix Execution API JWT reissue TOCTOU race
* #71959 - Document REMOTE_TASK_LOG/DEFAULT_REMOTE_CONN_ID for custom logging configs
* #72767 - Ignore stale defer-exit success once the resumed attempt is running
* #73451 - Fix DetachedInstanceError crash in mapped task expansion
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
* #68239 - Fix spurious 'remote_log_handler_unavailable' warning in triggerer
* #68557 - Use Configuration.socket_options instead of monkey-patching urllib3 in cncf-kubernetes
* #70029 - Fix OAuth redirect URI scheme behind reverse proxy (#70023)
* #71074 - Fix duplicate pending Dag runs for the same asset partition key
* #71939 - Fix Spark standalone driver status polling to use the REST API port
* #72495 - Add machine-readable RBAC reference to BaseAuthManager
This is **not** a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.
What we ask you to do is to make your **first prioritization decision**: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or `gh pr reopen <PR_NUMBER> --repo apache/airflow`. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.
While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.
Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org