You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2026/06/12 09:34:23 UTC
[PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
amoghrajesh opened a new pull request, #68438:
URL: https://github.com/apache/airflow/pull/68438
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [x] Yes: claude sonnet 4.6
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
### What
Renames `task_store` / `asset_store` to `task_state_store` / `asset_state_store` across the codebase. This is a **purely mechanical rename**, no logic, behaviour, or API contract changes.
### Why
@ashb flagged the name `AssetStore` as a significant naming issue: it implies storing actual asset data (potentially GBs of files/objects), whereas these stores hold lightweight state metadata such as job IDs, watermarks, cursors, coordination tokens. The suffix `StateStore` makes the intent unambiguous.
### Proposed change
Rename everything to `TaskStateStore` / `AssetStateStore` to make the intent clear. Scope of the rename:
- **Models**: `TaskStoreModel` -> `TaskStateStoreModel`, `AssetStoreModel` → `AssetStateStoreModel`
- **DB cleanup service**: `MetastoreStoreBackend` -> `MetastoreStateStoreBackend`
- **Execution API**: datamodels, route handlers, Cadwyn version schemas
- **Core API**: datamodels, route handlers, OpenAPI spec — URL path `/store` -> `/state-store`
- **CLI**: `cleanup-task-store` -> `cleanup-task-state-store`
- **Task SDK**: `TaskStateStoreOperations`, `AssetStateStoreOperations`, context keys `task_state_store` / `asset_state_store`, comms models
- **UI**: page components, generated OpenAPI client, i18n strings
- **Docs**: all RST files and toctree entries
- **Tests**: all affected test files and fixtures
> Note: Execution API URL paths `/store/ti` and `/store/asset` are **intentionally unchanged** — workers already use them and changing them would require a versioned migration for no benefit.
### Testing
Most of the testing here is for sanity purpose
1. DB sanity
<img width="1728" height="1031" alt="image" src="https://github.com/user-attachments/assets/b3bf5953-98f9-4907-a9f0-fa431d1c771e" />
<img width="1728" height="1031" alt="image" src="https://github.com/user-attachments/assets/3823f4d2-56e8-4c7c-b8ec-12df55067637" />
2. Example dags run fine
<img width="1728" height="917" alt="image" src="https://github.com/user-attachments/assets/688c17b4-e2da-4900-8404-02e1ac7ab624" />
<img width="1728" height="902" alt="image" src="https://github.com/user-attachments/assets/65f25ea7-cb08-4b8f-8e22-9cb7d1302fc6" />
<img width="1728" height="902" alt="image" src="https://github.com/user-attachments/assets/99071034-311f-499e-b0a4-25cc5df9dec9" />
3. UI renders well
<img width="1728" height="902" alt="image" src="https://github.com/user-attachments/assets/c4cc003b-c8ab-4a93-bacc-8ad83072e64e" />
<img width="914" height="902" alt="image" src="https://github.com/user-attachments/assets/56865ae8-9a59-4022-a21e-37e985e23a8e" />
(Since these use public APIs, we should be good)
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #68438:
URL: https://github.com/apache/airflow/pull/68438#issuecomment-4700824859
Thanks for reviews all, merging this one now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407841663
##########
airflow-core/src/airflow/state/metastore.py:
##########
@@ -135,9 +141,9 @@ def set(
assert session is not None
match scope:
case TaskScope():
- self._set_task_store(scope, key, value, expires_at=expires_at, session=session)
+ self._set_task_state_store(scope, key, value, expires_at=expires_at, session=session)
Review Comment:
I like the latter better, making changes for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407840294
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3545,7 +3545,7 @@ def _update_asset_orphanage(self, *, session: Session = NEW_SESSION) -> None:
self._orphan_unreferenced_assets(orphan_query, session=session)
self._activate_referenced_assets(activate_query, session=session)
- self._cleanup_orphaned_asset_store(session=session)
+ self._cleanup_orphaned_asset_state_store(session=session)
Review Comment:
No need for it. That's because:
- task state store rows are tied to dag_run_id and clean up immediately when a dagrun is gone
- there's a cleanup utility provided for this `airflow state-store cleanup-task-state-store` whcih handles expiry-based cleanup for long-lived rows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68438:
URL: https://github.com/apache/airflow/pull/68438#issuecomment-4700827395
### Backport failed to create: airflow-ctl/v0-1-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/27489674602'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>airflow-ctl/v0-1-test</td>
<td><a href="https://github.com/apache/airflow/commit/74a52cf5d3d99b22deb46a1d0bcf98ad9ac68f31"><img src='https://img.shields.io/badge/Commit-74a52cf-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 74a52cf airflow-ctl/v0-1-test
```
This should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #68438:
URL: https://github.com/apache/airflow/pull/68438#issuecomment-4703123704
@amoghrajesh Noticed that my German translation catch-up now had new gaps as you renamed the new language keys. I reworked my PR (https://github.com/apache/airflow/pull/68356) but I assume a lot of other parallel running translation PRs were outdated on the new keys due to the late change.
Does it make sense to bulk-rename already existing translations on main or do the translations need to change in your view adjusting to new key?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407928608
##########
airflow-core/src/airflow/state/metastore.py:
##########
@@ -135,9 +141,9 @@ def set(
assert session is not None
match scope:
case TaskScope():
- self._set_task_store(scope, key, value, expires_at=expires_at, session=session)
+ self._set_task_state_store(scope, key, value, expires_at=expires_at, session=session)
Review Comment:
Done in [review from ash](https://github.com/apache/airflow/pull/68438/commits/581f5b968f6e9ada8f12efafeec13f8e982488f9)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407836326
##########
airflow-core/src/airflow/cli/commands/state_store_command.py:
##########
@@ -19,18 +19,18 @@
import logging
from airflow.state import get_state_backend
-from airflow.state.metastore import MetastoreStoreBackend
+from airflow.state.metastore import MetastoreStateStoreBackend
Review Comment:
Alright yeah sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407928867
##########
airflow-core/src/airflow/cli/commands/state_store_command.py:
##########
@@ -19,18 +19,18 @@
import logging
from airflow.state import get_state_backend
-from airflow.state.metastore import MetastoreStoreBackend
+from airflow.state.metastore import MetastoreStateStoreBackend
Review Comment:
Handled in [review from ash](https://github.com/apache/airflow/pull/68438/commits/581f5b968f6e9ada8f12efafeec13f8e982488f9)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407838771
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -86,7 +86,7 @@
TaskInletAssetReference,
TaskOutletAssetReference,
)
-from airflow.models.asset_store import AssetStoreModel
+from airflow.models.asset_state_store import AssetStateStoreModel
Review Comment:
This is the orphaned-row sweep that runs alongside `_orphan_unreferenced_assets` — it issues a single bulk `DELETE` keyed on `asset_id NOT IN (active assets)`. The store backend API is scoped to individual key get/set/delete operations and doesn't expose a bulk "delete all rows for these asset IDs" operation, so we'd need to either add a method to the backend for this specific use case or keep the direct model access here. Given it's a scheduler-internal maintenance operation (not user-facing task execution), the direct model access feels reasonable. Happy to add a `delete_orphaned_asset_state` method to the backend if you'd prefer.
---
Drafted-by: Claude Sonnet 4.6 (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #68438:
URL: https://github.com/apache/airflow/pull/68438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407838771
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -86,7 +86,7 @@
TaskInletAssetReference,
TaskOutletAssetReference,
)
-from airflow.models.asset_store import AssetStoreModel
+from airflow.models.asset_state_store import AssetStateStoreModel
Review Comment:
This is the orphaned-row sweep that runs alongside `_orphan_unreferenced_assets` — it issues a single bulk `DELETE` keyed on `asset_id NOT IN (active assets)`. The store backend API is scoped to individual key get/set/delete operations and doesn't expose a bulk "delete all rows for these asset IDs" operation, so we'd need to either add a method to the backend for this specific use case or keep the direct model access here. Given it's a scheduler-internal maintenance operation (not user-facing task execution), the direct model access feels reasonable. Happy to add a `delete_orphaned_asset_state` method to the backend if you'd prefer.
---
Drafted-by: Claude Sonnet 4.6 (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3407838513
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -86,7 +86,7 @@
TaskInletAssetReference,
TaskOutletAssetReference,
)
-from airflow.models.asset_store import AssetStoreModel
+from airflow.models.asset_state_store import AssetStateStoreModel
Review Comment:
This is the orphaned-row sweep that runs alongside `_orphan_unreferenced_assets`, it issues a single bulk DELETE keyed on asset_id NOT IN (active assets). The store backend API is scoped to individual key get/set/delete operations and does not expose a bulk "delete all rows for these asset IDs" operation, so we'd need to either add a method to the backend for this specific use case or keep the direct model access here. Given it's a scheduler's maintenance operation, the direct model access feels reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #68438:
URL: https://github.com/apache/airflow/pull/68438#issuecomment-4698468940
The failures are unrelated, rebasing it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename task_store/asset_store to task_state_store/asset_state_store [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #68438:
URL: https://github.com/apache/airflow/pull/68438#discussion_r3403050321
##########
airflow-core/src/airflow/cli/commands/state_store_command.py:
##########
@@ -19,18 +19,18 @@
import logging
from airflow.state import get_state_backend
-from airflow.state.metastore import MetastoreStoreBackend
+from airflow.state.metastore import MetastoreStateStoreBackend
Review Comment:
So this one, I think could remain as `MetastoreBackend`, since we are already in the `airflow.state` package.
##########
airflow-core/src/airflow/state/metastore.py:
##########
@@ -135,9 +141,9 @@ def set(
assert session is not None
match scope:
case TaskScope():
- self._set_task_store(scope, key, value, expires_at=expires_at, session=session)
+ self._set_task_state_store(scope, key, value, expires_at=expires_at, session=session)
Review Comment:
or
```suggestion
self._store_task_state(scope, key, value, expires_at=expires_at, session=session)
```
##########
airflow-core/src/airflow/state/metastore.py:
##########
@@ -135,9 +141,9 @@ def set(
assert session is not None
match scope:
case TaskScope():
- self._set_task_store(scope, key, value, expires_at=expires_at, session=session)
+ self._set_task_state_store(scope, key, value, expires_at=expires_at, session=session)
Review Comment:
```suggestion
self._set_task_state(scope, key, value, expires_at=expires_at, session=session)
```
We are definitionally in the state store already, and we aren't setting the state store, but setting/storing state.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -86,7 +86,7 @@
TaskInletAssetReference,
TaskOutletAssetReference,
)
-from airflow.models.asset_store import AssetStoreModel
+from airflow.models.asset_state_store import AssetStateStoreModel
Review Comment:
Nit: why is the scheduler importing this directly rather than going via the store backend?
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3545,7 +3545,7 @@ def _update_asset_orphanage(self, *, session: Session = NEW_SESSION) -> None:
self._orphan_unreferenced_assets(orphan_query, session=session)
self._activate_referenced_assets(activate_query, session=session)
- self._cleanup_orphaned_asset_store(session=session)
+ self._cleanup_orphaned_asset_state_store(session=session)
Review Comment:
nit/observation: there is no cleanup of orphaned task state store entries -- should there be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org