You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "chsa21 (via GitHub)" <gi...@apache.org> on 2026/07/22 09:13:43 UTC

[I] Execution API JWT expires unexpectedly in Airflow 3.2.2 (ExpiredSignatureError on /execution/...) [airflow]

chsa21 opened a new issue, #70222:
URL: https://github.com/apache/airflow/issues/70222

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.2.2
   
   ### What happened and how to reproduce it?
   
   We are observing intermittent Execution API authentication failures in Apache Airflow 3.2.2.
   
   Multiple tasks fail at approximately the same time because the API Server rejects their JWT with:
   
   Failed to validate JWT
   detail=Signature has expired
   exception=ExpiredSignatureError
   path=/execution/...
   
   The affected tasks are not necessarily running for one hour, although the default configuration is used:
   
   [execution_api]
   jwt_expiration_time = 3600
   
   [api_auth]
   jwt_expiration_time = 86400
   jwt_cli_expiration_time = 3600
   
   All Airflow components (Scheduler, API Server and Worker) run on the same physical host.
   
   We could not identify any infrastructure issues such as VM snapshots, clock jumps, or system time changes.
   
   Unfortunately we currently do not have reliable reproduction steps. The problem occurs intermittently in production.
   
   We found previous issues describing similar Execution API JWT expiration problems. According to the issue history this should have been fixed before Airflow 3.2.2, however we are still observing this behavior.
   
   ### What you think should happen instead?
   
   Execution API JWTs should remain valid for their configured lifetime or be refreshed automatically.
   
   Running tasks should not fail with ExpiredSignatureError when using the default Airflow configuration.
   
   ### Operating System
   
   RedHat 9.8
   
   ### Deployment
   
   _No response_
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   Not Applicable
   
   ### Helm Chart configuration
   
   Not Applicable
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   - Apache Airflow 3.2.2
   - Execution API JWT expiration time: 3600 (default)
   - API auth JWT expiration time: 86400 (default)
   - API Server log:
   
     Failed to validate JWT
     detail=Signature has expired
     exception=ExpiredSignatureError
     path=/execution/...
   
   - Multiple tasks fail simultaneously.
   - Other Airflow 3.2.2 environments with the same configuration do not show this behavior.
   - The affected tasks are not necessarily running for one hour before failing.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Execution API JWT expires unexpectedly in Airflow 3.2.2 (ExpiredSignatureError on /execution/...) [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #70222:
URL: https://github.com/apache/airflow/issues/70222#issuecomment-5077237265

   **Triage proposal**
   
   The signature here — `ExpiredSignatureError` on `/execution/...` for tasks that have not been running anywhere near the token lifetime, several failing in the same window — matches a mid-request expiry race in `JWTReissueMiddleware`, not a missing backport. Two corrections to the report's premises first:
   
   - **The two-token fix is already in 3.2.2.** [apache/airflow#60108](https://github.com/apache/airflow/pull/60108) was backported as [apache/airflow#66989](https://github.com/apache/airflow/pull/66989) (`f00b3c948e`, an ancestor of the `3.2.2` tag). That fix covers tokens expiring while a task waits in an *executor queue*, which is a different failure mode — so its presence neither contradicts nor explains this report.
   - **`[execution_api] jwt_expiration_time` defaults to `600`, not `3600`** ([`config.yml` @ 3.2.2](https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/config_templates/config.yml)). The `3600` quoted in the report is `[api_auth] jwt_cli_expiration_time`. Unless it has been overridden, the Execution API token lifetime in play is 10 minutes — which is consistent with "the affected tasks are not necessarily running for one hour".
   
   On the mechanism: `JWTBearer` validates the bearer token and caches the claims on the ASGI request scope ([`security.py` @ 3.2.2](https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/api_fastapi/execution_api/security.py#L88-L139)); after the handler returns, [`JWTReissueMiddleware`](https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/api_fastapi/execution_api/app.py#L130-L164) re-validates the *same* bearer string. If the token crosses `exp` during the request, that second validation raises, the exception is swallowed, and no `Refreshed-API-Token` is returned — so the client keeps a dead token, the next call 403s, and after enough failed heartbeats the task is killed. That is intermittent by nature and can take out several tasks at once, which fits the report.
   
   **Proposed disposition: DUPLICATE of [apache/airflow#67939](https://github.com/apache/airflow/issues/67939).**
   
   Same root cause, same code path. [apache/airflow#67939](https://github.com/apache/airflow/issues/67939) is older (filed 2026-06-03), on the same 3.2.2, and carries the reliable reproduction this report is missing (long-running `BashOperator` under `LocalExecutor`).
   
   Worth noting: this race now has **two open PRs making the same change** — [apache/airflow#68499](https://github.com/apache/airflow/pull/68499) against #67939, and [apache/airflow#70227](https://github.com/apache/airflow/pull/70227) against this issue. Both reuse the claims already cached in the request scope instead of re-validating. The reissue middleware is byte-identical between the `3.2.2` tag and current `main`, so this is not fixed on `main` either.
   
   Action items:
   
   1. Confirm the merge direction — fold this into #67939 as canonical, or keep both open.
   2. Reconcile the two open PRs. They fix the same race the same way; carrying both is reviewer noise whichever issue survives.
   3. From the reporter, to confirm the mechanism: which executor is in use, and the decoded `iat`/`exp` of one failing token next to the API-server timestamp of the rejection.
   
   Flagging this as a **low-confidence merge direction** — this report's "several tasks fail simultaneously" is a broader signature than #67939's single long-running task, so an argument for keeping them separate is reasonable. Please challenge. My read on severity is medium-to-high (it kills running tasks under default config), but scoring is the team's call.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @shahar1 before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Execution API JWT expires unexpectedly in Airflow 3.2.2 (ExpiredSignatureError on /execution/...) [airflow]

Posted by "chsa21 (via GitHub)" <gi...@apache.org>.
chsa21 commented on issue #70222:
URL: https://github.com/apache/airflow/issues/70222#issuecomment-5088017137

   Hi, thanks for the analysis.
   
   The issue occurred again on Saturday.
   
   Regarding your questions:
   - Executor: LocalExecutor
   - Unfortunately, we don't currently log the decoded JWT claims (iat/exp), so we can't provide those values.
   - I've attached an anonymized excerpt from the API server log. It shows the sequence of:
     1. JWT reissue middleware failed to refresh token with ExpiredSignatureError,
     2. the subsequent 403 on the /execution/... endpoint, and
     3. Failed to validate JWT for the following request.
   
   This seems consistent with the race condition you described, where the token expires during request processing and no refreshed token is returned.
   
   Please let me know if there is any other information that would help confirm the root cause.
   
   Log:
   2026-07-24T23:37:03.964536Z [warning  ] JWT reissue middleware failed to refresh token [airflow.api_fastapi.execution_api.app] error='Signature has expired' loc=app.py:158
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: Traceback (most recent call last):
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/execution_api/app.py", line 141, in dispatch
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     claims = await validator.avalidated_claims(token, {})
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     claims = jwt.decode(
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         unvalidated,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ...<5 lines>...
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         leeway=self.leeway,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     )
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 368, in decode
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     decoded = self.decode_complete(
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         jwt,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ...<8 lines>...
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         leeway=leeway,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     )
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 275, in decode_complete
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     self._validate_claims(
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ~~~~~~~~~~~~~~~~~~~~~^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         payload,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         ^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ...<4 lines>...
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         subject=subject,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         ^^^^^^^^^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     )
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 408, in _validate_claims
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     self._validate_exp(payload, now, leeway)
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 511, in _validate_exp
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     raise ExpiredSignatureError("Signature has expired")
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: jwt.exceptions.ExpiredSignatureError: Signature has expired
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: 2026-07-24T23:37:03.965604Z [info     ] request finished               [http.access] client_addr=? duration_us=3424 loc=http_access_log.py:98 method=PATCH path=/execution/task-instances/019f95df-f1c7-757e-985e-cf333fb5cb35/run query= status_code=403
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: 2026-07-24T23:37:03.978324Z [warning  ] Failed to validate JWT         [airflow.api_fastapi.execution_api.security] correlation_id=019f967d-54e8-7c2b-9452-4e5637085cdb loc=security.py:127 token=eyJ***
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: Traceback (most recent call last):
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/execution_api/security.py", line 125, in __call__
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     claims = await validator.avalidated_claims(creds.credentials, dict(self.required_claims))
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/auth/tokens.py", line 331, in avalidated_claims
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     claims = jwt.decode(
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         unvalidated,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ...<5 lines>...
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         leeway=self.leeway,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     )
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 368, in decode
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     decoded = self.decode_complete(
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         jwt,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ...<8 lines>...
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         leeway=leeway,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     )
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 275, in decode_complete
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     self._validate_claims(
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ~~~~~~~~~~~~~~~~~~~~~^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         payload,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         ^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ...<4 lines>...
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         subject=subject,
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:         ^^^^^^^^^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     )
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 408, in _validate_claims
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     self._validate_exp(payload, now, leeway)
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:   File "path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py", line 511, in _validate_exp
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:     raise ExpiredSignatureError("Signature has expired")
   Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: jwt.exceptions.ExpiredSignatureError: Signature has expired


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Execution API JWT expires unexpectedly in Airflow 3.2.2 (ExpiredSignatureError on /execution/...) [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #70222: Execution API JWT expires unexpectedly in Airflow 3.2.2 (ExpiredSignatureError on /execution/...)
URL: https://github.com/apache/airflow/issues/70222


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org