You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "andreahlert (via GitHub)" <gi...@apache.org> on 2026/02/08 10:40:08 UTC
[PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
andreahlert opened a new pull request, #61631:
URL: https://github.com/apache/airflow/pull/61631
Closes: #53140
When a running Task Instance is cleared, `prepare_db_for_next_try()` archives the current try to the Task Instance History (TIH) table and assigns a new UUID. This means heartbeats from the old process will fail with a 404 "Not Found" because the UUID no longer exists in the TI table. However, a generic 404 is misleading since the TI did exist - it was just cleared.
This PR adds a TIH lookup in the `NoResultFound` handler of the heartbeat endpoint:
- If the UUID is **not** in TI but **is** in TIH: return **410 Gone**, indicating the TI was cleared/moved
- If the UUID is **not** in TI and **not** in TIH: return **404 Not Found** (unchanged behavior)
This gives the task SDK supervisor a more specific signal and matches HTTP semantics (410 means "the target resource is no longer available at the origin server and this condition is likely to be permanent").
Changes:
- `execution_api/routes/task_instances.py`: Added TIH existence check in the heartbeat `NoResultFound` handler; added 410 to OpenAPI response schema
- `execution_time/supervisor.py`: Added `HTTPStatus.GONE` to the set of status codes that the supervisor treats as "stop heartbeating" (alongside NOT_FOUND and CONFLICT)
- `test_task_instances.py`: Added test using `prepare_db_for_next_try()` to properly simulate a cleared TI and verify the 410 response
Previous attempt: #56443 (closed as stale). This PR addresses the review feedback from that PR by actually checking the TIH table rather than blindly changing 404 to 410.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-3943528010
Tests need fixing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "andreahlert (via GitHub)" <gi...@apache.org>.
andreahlert commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-4158086183
Friendly ping to @ashb @kaxil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-3912458435
It's an intermittent failure, restarting it and taking a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "andreahlert (via GitHub)" <gi...@apache.org>.
andreahlert commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-4045686331
Rebased on main. CI failures are all `fatal: could not read Username for 'https://github.com': terminal prompts disabled` — GitHub Actions infra issue, not related to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "andreahlert (via GitHub)" <gi...@apache.org>.
andreahlert commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-3913138917
> Small comments, otherwise LGTM thanks!
Done! Thanks for reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "andreahlert (via GitHub)" <gi...@apache.org>.
andreahlert commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-4066024016
Rebased and ready for review. @amoghrajesh @ashb @kaxil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-4035212146
@andreahlert This PR has been converted to **draft** because it does not yet meet our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
**Issues found:**
- :x: **Other failing CI checks**: Failing: Basic tests / React UI tests. Run `prek run --from-ref main` locally to reproduce. See [static checks docs](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
> **Note:** Your branch is **488 commits behind `main`**. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.
**What to do next:**
- The comment informs you what you need to do.
- Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
- Maintainers will then proceed with a normal review.
Converting a PR to draft is **not** a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the [Airflow Slack](https://s.apache.org/airflow-slack).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-4045238586
Tests are failing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-3918915742
@andreahlert can you check the failing tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #61631:
URL: https://github.com/apache/airflow/pull/61631#discussion_r2815146503
##########
airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -587,6 +591,24 @@ def ti_heartbeat(
"Retrieved current task state", state=previous_state, current_hostname=hostname, current_pid=pid
)
except NoResultFound:
+ # Check if the TI exists in the Task Instance History table.
+ # If it does, it was likely cleared while running, so return 410 Gone
+ # instead of 404 Not Found to give the client a more specific signal.
+ tih_exists = session.scalar(
+ select(func.count(TIH.task_instance_id)).where(TIH.task_instance_id == ti_id_str)
+ )
+ if tih_exists:
+ log.error(
+ "Task Instance not found in TI table but exists in Task Instance History",
Review Comment:
```suggestion
"TaskInstance was previously cleared and archived in history, heartbeat skipped",
```
##########
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py:
##########
@@ -1591,6 +1591,41 @@ def test_ti_heartbeat_non_existent_task(self, client, session, create_task_insta
"message": "Task Instance not found",
}
+ def test_ti_heartbeat_cleared_task_returns_410(self, client, session, create_task_instance):
+ """Test that a 410 error is returned when a TI was cleared and moved to TIH."""
+ ti = create_task_instance(
+ task_id="test_ti_heartbeat_cleared",
+ state=State.RUNNING,
+ hostname="random-hostname",
+ pid=1547,
+ session=session,
+ )
+ session.commit()
+ old_ti_id = ti.id
+
+ # Simulate task being cleared: this archives the current try to TIH
+ # and assigns a new UUID to the TI, mirroring prepare_db_for_next_try().
+ ti.prepare_db_for_next_try(session)
+ session.commit()
+
+ # Pre-condition: old UUID is gone from TI table but exists in TIH
Review Comment:
```suggestion
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -587,6 +591,24 @@ def ti_heartbeat(
"Retrieved current task state", state=previous_state, current_hostname=hostname, current_pid=pid
)
except NoResultFound:
+ # Check if the TI exists in the Task Instance History table.
+ # If it does, it was likely cleared while running, so return 410 Gone
+ # instead of 404 Not Found to give the client a more specific signal.
+ tih_exists = session.scalar(
+ select(func.count(TIH.task_instance_id)).where(TIH.task_instance_id == ti_id_str)
+ )
+ if tih_exists:
+ log.error(
+ "Task Instance not found in TI table but exists in Task Instance History",
+ ti_id=ti_id_str,
+ )
+ raise HTTPException(
+ status_code=status.HTTP_410_GONE,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found, it may have been moved to the Task Instance History table",
Review Comment:
```suggestion
"message": "Task Instance not found",
```
We know it didn't move ti TIH from the check above 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61631:
URL: https://github.com/apache/airflow/pull/61631#issuecomment-4184799283
### Backport successfully created: v3-2-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/pull/64693"><img src="https://img.shields.io/badge/PR-64693-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return 410 Gone for heartbeat when cleared TI exists in TIH [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #61631:
URL: https://github.com/apache/airflow/pull/61631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org