You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jroachgolf84 (via GitHub)" <gi...@apache.org> on 2026/05/21 19:24:09 UTC
[PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
jroachgolf84 opened a new pull request, #67299:
URL: https://github.com/apache/airflow/pull/67299
## Description
closes: #65782
##### Was generative AI tooling used to co-author this PR?
No, generative AI was not used to co-author this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313447530
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
Review Comment:
I can work on a PR for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367139349
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require some amount of data to be persisted outside of a task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Task and Asset store provide two key/value stores to persist data like a job ID, watermark, or row count. These two stores are differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept JSON-able values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
Review Comment:
Good point, I had it in my backlog, tried to unify that here: https://github.com/apache/airflow/pull/68133
Now I am enforcing a configurable limit on core API as opposed to hardcoded and in execution API i choose to warn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367308389
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days after which task store rows expire. When a key is written with no explicit retention, expires_at is computed on the worker as now + default_retention_days. Changing this setting does not affect already-written rows.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset store rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task store keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task store entries after success for observability (e.g. the submitted job ID or the last row count is still readable from the UI or REST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task store only**. It has no effect on asset store. Asset store is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_store`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+.. _task-and-asset-store:worker-backends:
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task store and asset store values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStoreAccessor.set()`` calls ``serialize_task_store_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_store_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-store``. This process removes store rows according to the following rules:
+
+**Time-based expiry (task store only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task store only)**
+ Keys written with no explicit retention get an ``expires_at`` of now + default_retention_days computed at write time. Garbage collection deletes rows where ``expires_at < now()``."
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset store)**
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3357087414
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
Review Comment:
Agreed, resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313141780
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
+
+.. important::
+
+ **References must be deterministic.** Given the same inputs (``ti_id`` + ``key`` for task state; ``asset_ref`` + ``key`` for asset state), the serialization method must always return the same reference string. Do not embed timestamps, random UUIDs, or any other non-deterministic component in the reference path.
+
+ When a key is deleted or cleared, Airflow clears the database reference *first*, then calls the backend's ``delete()`` or ``clear()`` method. If backend cleanup fails after the DB row is gone, the external object is orphaned. Because the reference is deterministic, a subsequent ``set()`` for the same key will overwrite the orphaned object, making the failure recoverable. A non-deterministic reference would leave the external object permanently orphaned with no way to locate it.
+
+Example skeleton:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class S3StateBackend(BaseStateBackend):
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3341115883
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,255 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </core-concepts/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-states``. This process removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+.. important::
+
+ Garbage collection only works for the ``MetastoreStateBackend``. Custom backends are explicitly skipped.
+
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 55 45
+
+ * - Signature
+ - Description
+ * - ``def get(self, scope: StateScope, key: str, *, session: Session | None = None) -> str | None``
+ - Return the stored value, or ``None``.
+ * - ``def set(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: Session | None = None) -> None``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``def delete(self, scope: StateScope, key: str, *, session: Session | None = None) -> None``
+ - Delete a single key; no-op if absent.
+ * - ``def clear(self, scope: StateScope, *, all_map_indices: bool = False, session: Session | None = None) -> None``
+ - Delete all keys under the scope.
+ * - ``async def aget(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> str | None``
+ - Async variant of ``get``.
+ * - ``async def aset(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: AsyncSession | None = None) -> None``
+ - Async variant of ``set``.
+ * - ``async def adelete(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> None``
+ - Async variant of ``delete``.
+ * - ``async def aclear(self, scope: StateScope, *, all_map_indices: bool = False, session: AsyncSession | None = None) -> None``
+ - Async variant of ``clear``.
Review Comment:
I think that's fair, I'll use the same approach as Secrets Backend does.
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,255 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </core-concepts/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-states``. This process removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+.. important::
+
+ Garbage collection only works for the ``MetastoreStateBackend``. Custom backends are explicitly skipped.
+
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 55 45
+
+ * - Signature
+ - Description
+ * - ``def get(self, scope: StateScope, key: str, *, session: Session | None = None) -> str | None``
+ - Return the stored value, or ``None``.
+ * - ``def set(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: Session | None = None) -> None``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``def delete(self, scope: StateScope, key: str, *, session: Session | None = None) -> None``
+ - Delete a single key; no-op if absent.
+ * - ``def clear(self, scope: StateScope, *, all_map_indices: bool = False, session: Session | None = None) -> None``
+ - Delete all keys under the scope.
+ * - ``async def aget(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> str | None``
+ - Async variant of ``get``.
+ * - ``async def aset(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: AsyncSession | None = None) -> None``
+ - Async variant of ``set``.
+ * - ``async def adelete(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> None``
+ - Async variant of ``delete``.
+ * - ``async def aclear(self, scope: StateScope, *, all_map_indices: bool = False, session: AsyncSession | None = None) -> None``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if isinstance(scope, TaskScope):
+ return self._task_store.get(scope, key)
+ elif isinstance(scope, AssetScope):
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly using worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``def serialize_task_state_to_ref(self, *, value: str, key: str, ti_id: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``def deserialize_task_state_from_ref(self, stored: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``def serialize_asset_state_to_ref(self, *, value: str, key: str, asset_ref: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``def deserialize_asset_state_from_ref(self, stored: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
Review Comment:
Ack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3306285187
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
Review Comment:
Need to update to remove `AssetState`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3360455668
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,207 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key, default)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark", default="initial_watermark")
+
+ # Using self.asset_store
+ watermark = self.asset_store[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike task store, asset store has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated. Like with task store, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using self.asset_store
+ self.asset_store[my_asset].set("watermark", "2024-06-01T00:00:00Z")
Review Comment:
```suggestion
# Using context
context["asset_store"][my_asset].set("watermark", default="2024-06-01T00:00:00Z")
# Using self.asset_store
self.asset_store[my_asset].set("watermark", default="2024-06-01T00:00:00Z")
```
Hard to grok for readers without default
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,207 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
Review Comment:
```suggestion
```
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,207 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
Review Comment:
```suggestion
```
Fixed by https://github.com/apache/airflow/pull/67619 right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4613478494
@amoghrajesh - do you mind taking one last look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3360468590
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,276 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key", default="my_default_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
Review Comment:
```suggestion
Returns the stored JSON value, or the ``default`` value if the key does not exist.
```
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task store** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
Review Comment:
```suggestion
Number of days after which task store rows expire. When a key is written with no explicit retention, expires_at is computed on the worker as now + default_retention_days. Changing this setting does not affect already-written rows.
```
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task store** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset store rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task store keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task store entries after success for observability (e.g. the submitted job ID or the last row count is still readable from the UI or REST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task store only**. It has no effect on asset store. Asset store is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_store`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+.. _task-and-asset-store:worker-backends:
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task store and asset store values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStoreAccessor.set()`` calls ``serialize_task_store_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_store_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-store``. This process removes store rows according to the following rules:
+
+**Time-based expiry (task store only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task store only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
Review Comment:
```suggestion
Keys written with no explicit retention get an ``expires_at`` of now + default_retention_days computed at write time. Garbage collection deletes rows where ``expires_at < now()``."
```
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,207 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key, default)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
Review Comment:
```suggestion
Returns the stored JSON value, or the ``default`` value if the key does not exist.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313070928
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
Review Comment:
Fixed here, as well as in all the other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3326471563
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3326472577
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3321801899
##########
airflow-core/docs/administration-and-deployment/state-api.rst:
##########
@@ -0,0 +1,142 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-api:
+
+State REST API
Review Comment:
I am not sure of the value of a separate API page here, since the auto-generated page will already have it at https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313102055
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
Review Comment:
Can you expand on this a bit more? What do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313194711
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Accessing asset state using ``AssetState``
+------------------------------------------
+
+Asset state can also be retrieved for an asset using the ``airflow.sdk.AssetState`` class. This approach does NOT require that an asset is passed to a task using ``inlets``.
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(name="my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, the ``name`` of the asset is used to retrieve it's state. However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(uri="s3://bucket/my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_state"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_state = context["asset_state"]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_state"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_state"][my_asset]``), the shorthand (``context["asset_state"]``) when the task has exactly one inlet, and when using the ``AssetState`` Task SDK class.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3326457840
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
Review Comment:
Sounds good, I'll add that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3325600508
##########
airflow-core/docs/administration-and-deployment/state-api.rst:
##########
@@ -0,0 +1,142 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-api:
+
+State REST API
Review Comment:
Okay, I'll remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342745585
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
+
+Task and Asset State
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task state**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342801173
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
+* Keys written with ``retention=NEVER_EXPIRE`` have ``expires_at = NULL`` and a flag that marks them as permanent. They are **never** deleted by this command regardless of configuration.
+
+If ``[state_store] default_retention_days = 0``, keys written without an explicit retention have ``expires_at = NULL`` (no expiry) and are also skipped. Only keys with a non-null, past ``expires_at`` are removed.
+
+.. note::
+
+ Custom backends (``[state_store] backend`` set to anything other than the default) are **explicitly skipped**. The cleanup command prints a message and exits cleanly without deleting anything. If your custom backend needs its own retention logic, implement it in ``BaseStateBackend.cleanup()`` and call it from your own maintenance process.
+
+
+Running cleanup
+---------------
+
+The command is::
+
+ airflow state-store cleanup-task-states
+
+It reads ``[state_store] default_retention_days`` and ``[state_store] state_cleanup_batch_size`` from the ``airflow.cfg`` file, then deletes all eligible rows.
+
+**Dry run**
+
+Use ``--dry-run`` to preview what would be deleted without removing anything::
+
+ airflow state-store cleanup-task-states --dry-run
+
+The output lists every row that would be deleted, grouped by dag, run, task, map index, and key.
+
+**Batching**
+
+By default (``state_cleanup_batch_size = 0``) all eligible rows are deleted in a single statement. On deployments with large ``task_state`` tables, set a batch size to reduce lock duration per transaction::
+
+ # airflow.cfg
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+The command then deletes rows in batches of 10,000, committing after each batch, until no eligible rows remain.
+
+
+Scheduling regular cleanup
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3357082429
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and JSON values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
+
+When to use Task and Asset Store
+--------------------------------
+
+Use this table to choose the right mechanism for your use case.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 22 78
+
+ * - Mechanism
+ - When to use it
+ * - **XCom**
+ - Pass data *between tasks* within a single Dag run (e.g. the output of one task consumed by a downstream task) or across different multiple Dag runs (referencing the data persisted from another run). XComs are cleared on retry, and should NOT be used to persist data across task retries or across runs.
+ * - **Variables**
+ - Dag-wide or installation-wide configuration that changes infrequently and is set by operators rather than by tasks themselves.
Review Comment:
Updated in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3366363876
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,278 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
Review Comment:
The examples reach the store via `context["task_store"]` inside `def my_task(**context):`, but the text never says the task has to accept `**context` (or call `airflow.sdk.get_current_context()`). A reader using the common TaskFlow form `def my_task():` will hit a `NameError`. One sentence by this first example, and showing the `get_current_context()` form once, would close the gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367312884
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4605709616
Feedback has been implemented!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342797049
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,229 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+When using asset state within a task, ``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+If using asset state in a ``BaseEventTrigger``, the ``self.asset_state`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_state"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+To see asset state in-action in a real DAG, checkout the DAG in `example_asset_state.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_state.py>`_.
+
+Accessing asset state in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset state can be retrieved using the ``self.asset_state`` attribute.
+
+.. code-block:: python
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_state = self.asset_state[my_data]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_state = self.asset_state[my_data]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_state"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_state = context["asset_state"]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_state"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_state"][my_asset]``), the shorthand (``context["asset_state"]``) when the task has exactly one inlet, and when using the ``self.asset_state`` attribute.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_state"][my_asset].get("watermark")
+
+ # Using self.asset_state
+ watermark = self.asset_state[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike Task state, asset state has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using self.asset_state
+ self.asset_state[my_asset].set("watermark", "2024-06-01T00:00:00Z")
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342764818
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4622121334
Yep will be looking at it soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3357040163
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #67299:
URL: https://github.com/apache/airflow/pull/67299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313121071
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
+
+.. important::
+
+ **References must be deterministic.** Given the same inputs (``ti_id`` + ``key`` for task state; ``asset_ref`` + ``key`` for asset state), the serialization method must always return the same reference string. Do not embed timestamps, random UUIDs, or any other non-deterministic component in the reference path.
+
+ When a key is deleted or cleared, Airflow clears the database reference *first*, then calls the backend's ``delete()`` or ``clear()`` method. If backend cleanup fails after the DB row is gone, the external object is orphaned. Because the reference is deterministic, a subsequent ``set()`` for the same key will overwrite the orphaned object, making the failure recoverable. A non-deterministic reference would leave the external object permanently orphaned with no way to locate it.
+
+Example skeleton:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class S3StateBackend(BaseStateBackend):
+
+ def _task_ref(self, ti_id: str, key: str) -> str:
+ return f"airflow/task-state/{ti_id}/{key}"
+
+ def _asset_ref(self, asset_ref: str, key: str) -> str:
+ import hashlib
+
+ safe = hashlib.sha256(asset_ref.encode()).hexdigest()[:16]
+ return f"airflow/asset-state/{safe}/{key}"
+
+ def serialize_task_state_to_ref(self, *, value, key, ti_id):
+ s3_key = self._task_ref(ti_id, key)
+ s3_client.put_object(Bucket=BUCKET, Key=s3_key, Body=value.encode())
+ return s3_key
+
+ def deserialize_task_state_from_ref(self, stored):
+ s3_object = s3_client.get_object(Bucket=BUCKET, Key=stored)
+ return s3_object["Body"].read().decode()
Review Comment:
Updated! There were a few others in there I needed to update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313063715
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
Review Comment:
Thanks, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367315107
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require some amount of data to be persisted outside of a task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Task and Asset store provide two key/value stores to persist data like a job ID, watermark, or row count. These two stores are differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept JSON-able values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
Review Comment:
Documentation updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367047966
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored JSON value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark", default="initial_watermark")
+
+ # Using self.asset_store
+ watermark = self.asset_store[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike task store, asset store has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated. Like with task store, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].set("watermark", default="2024-06-01T00:00:00Z")
Review Comment:
Ah, might've been my suggestion by mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4638644789
@kaxil - feedback has been implemented!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3366363923
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,278 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key", default="my_default_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+---------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored JSON value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get(
+ "job_id", default="123456789"
+ ) # returns the value associated with `job_id` or the default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped during garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk import NEVER_EXPIRE
+
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_store.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* task store keys for this task instance.
+
+For :doc:`mapped tasks </authoring-and-scheduling/dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe the store across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_store.clear()
+
+ # clear all map indices (fleet-wide)
+ task_store.clear(all_map_indices=True)
+
+
+Some Example Use Cases
+----------------------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk import NEVER_EXPIRE
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_store = context["task_store"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_store.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job. Another example of this sort of logic can be found in `example_task_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_store.py>`_.
+
+For ``BaseOperator`` subclasses, the :class:`~airflow.sdk.bases.resumablemixin.ResumableJobMixin` encapsulates this pattern. It persists the external job ID to task store after submission and, on retry, reconnects to an active job or resubmits if the prior job reached a terminal failure state.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ # Retrieve the task_store
+ task_store = context["task_store"]
+ raw = task_store.get("last_page")
+
+ start_page = raw + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_store.set("last_page", page) # Update the task_store for reuse later
+
+
+On a retry, the task reads ``last_page`` and skips pages that were already processed.
+
+Progress metadata
+~~~~~~~~~~~~~~~~~
+
+Task store can expose in-progress metrics for observability — row counts, status strings, or lightweight JSON payloads — without requiring XCom or an external system.
+
+.. code-block:: python
+
+
+ from airflow.sdk import DAG, task
+
+ with DAG("row_ingest", schedule="@hourly"):
+
+ @task
+ def ingest_rows(**context):
+ task_store = context["task_store"]
+ total = 0
+
+ for batch in get_batches():
+ load(batch)
+ total += len(batch)
+ task_store.set(
+ "progress",
+ {"rows_loaded": total, "status": "running"},
+ )
+
+ task_store.set(
+ "progress",
+ {"rows_loaded": total, "status": "done"},
+ )
+
+The ``progress`` key is visible through the REST API and the Airflow UI while the task is running.
Review Comment:
This presents REST/UI visibility as a feature with no caveat. Unlike Variables and Connections, store values are persisted as a plain `Text` column with no Fernet encryption, and they are readable through the REST API and UI by anyone with access to the task or asset. Worth one explicit line here: do not put secrets or credentials in task or asset store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367047616
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
Review Comment:
+1 lets remove for now, and add it in the later PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367309812
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored JSON value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark", default="initial_watermark")
+
+ # Using self.asset_store
+ watermark = self.asset_store[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike task store, asset store has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated. Like with task store, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].set("watermark", default="2024-06-01T00:00:00Z")
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313632716
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
Review Comment:
PR for this: https://github.com/apache/airflow/pull/67619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4553264725
And a few things that I think are missing.
- A rest API page
- Maybe as a follow up task we should be mentioning the new ResumableJobMixin: https://github.com/apache/airflow/pull/67118, maybe task sdk docs
- `example_task_state.py` and `example_asset_state.py` can probably be mentioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313094722
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
Review Comment:
Thanks, that's a good catch. I've updated the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4547785464
I'm going to need to re-work the `assets` docs once I get feedback on this PR: #67248. Other than that, I'm curious to see what you think of the docs @amoghrajesh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342756046
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
+
+Task and Asset State
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task state**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run
+ * - **Asset state**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and JSON values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <state-store:worker-backends>`.
+
+When to use Task and Asset State
+--------------------------------
+
+Use this table to choose the right mechanism for your use case.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 22 78
+
+ * - Mechanism
+ - When to use it
+ * - **XCom**
+ - Pass data *between tasks* within a single Dag run (e.g. the output of one task consumed by a downstream task). XComs are cleared on retry, and should NOT be used to persist state across task retries or across runs.
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3343372624
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,255 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </core-concepts/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-states``. This process removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+.. important::
+
+ Garbage collection only works for the ``MetastoreStateBackend``. Custom backends are explicitly skipped.
+
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 55 45
+
+ * - Signature
+ - Description
+ * - ``def get(self, scope: StateScope, key: str, *, session: Session | None = None) -> str | None``
+ - Return the stored value, or ``None``.
+ * - ``def set(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: Session | None = None) -> None``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``def delete(self, scope: StateScope, key: str, *, session: Session | None = None) -> None``
+ - Delete a single key; no-op if absent.
+ * - ``def clear(self, scope: StateScope, *, all_map_indices: bool = False, session: Session | None = None) -> None``
+ - Delete all keys under the scope.
+ * - ``async def aget(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> str | None``
+ - Async variant of ``get``.
+ * - ``async def aset(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: AsyncSession | None = None) -> None``
+ - Async variant of ``set``.
+ * - ``async def adelete(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> None``
+ - Async variant of ``delete``.
+ * - ``async def aclear(self, scope: StateScope, *, all_map_indices: bool = False, session: AsyncSession | None = None) -> None``
+ - Async variant of ``clear``.
Review Comment:
Addressed in next commit.
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,255 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </core-concepts/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-states``. This process removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+.. important::
+
+ Garbage collection only works for the ``MetastoreStateBackend``. Custom backends are explicitly skipped.
+
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 55 45
+
+ * - Signature
+ - Description
+ * - ``def get(self, scope: StateScope, key: str, *, session: Session | None = None) -> str | None``
+ - Return the stored value, or ``None``.
+ * - ``def set(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: Session | None = None) -> None``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``def delete(self, scope: StateScope, key: str, *, session: Session | None = None) -> None``
+ - Delete a single key; no-op if absent.
+ * - ``def clear(self, scope: StateScope, *, all_map_indices: bool = False, session: Session | None = None) -> None``
+ - Delete all keys under the scope.
+ * - ``async def aget(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> str | None``
+ - Async variant of ``get``.
+ * - ``async def aset(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: AsyncSession | None = None) -> None``
+ - Async variant of ``set``.
+ * - ``async def adelete(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> None``
+ - Async variant of ``delete``.
+ * - ``async def aclear(self, scope: StateScope, *, all_map_indices: bool = False, session: AsyncSession | None = None) -> None``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if isinstance(scope, TaskScope):
+ return self._task_store.get(scope, key)
+ elif isinstance(scope, AssetScope):
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly using worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``def serialize_task_state_to_ref(self, *, value: str, key: str, ti_id: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``def deserialize_task_state_from_ref(self, stored: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``def serialize_asset_state_to_ref(self, *, value: str, key: str, asset_ref: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``def deserialize_asset_state_from_ref(self, stored: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3339416293
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
Review Comment:
```suggestion
A task state row is eligible for deletion when its expires, i.e: ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker when a task state row is written:
```
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
Review Comment:
```suggestion
The cleanup command operates only on **task state** rows in the default state backend. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
```
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
+* Keys written with ``retention=NEVER_EXPIRE`` have ``expires_at = NULL`` and a flag that marks them as permanent. They are **never** deleted by this command regardless of configuration.
+
+If ``[state_store] default_retention_days = 0``, keys written without an explicit retention have ``expires_at = NULL`` (no expiry) and are also skipped. Only keys with a non-null, past ``expires_at`` are removed.
+
+.. note::
+
+ Custom backends (``[state_store] backend`` set to anything other than the default) are **explicitly skipped**. The cleanup command prints a message and exits cleanly without deleting anything. If your custom backend needs its own retention logic, implement it in ``BaseStateBackend.cleanup()`` and call it from your own maintenance process.
+
+
+Running cleanup
+---------------
+
+The command is::
+
+ airflow state-store cleanup-task-states
+
+It reads ``[state_store] default_retention_days`` and ``[state_store] state_cleanup_batch_size`` from the ``airflow.cfg`` file, then deletes all eligible rows.
+
+**Dry run**
+
+Use ``--dry-run`` to preview what would be deleted without removing anything::
+
+ airflow state-store cleanup-task-states --dry-run
+
+The output lists every row that would be deleted, grouped by dag, run, task, map index, and key.
+
+**Batching**
+
+By default (``state_cleanup_batch_size = 0``) all eligible rows are deleted in a single statement. On deployments with large ``task_state`` tables, set a batch size to reduce lock duration per transaction::
+
+ # airflow.cfg
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+The command then deletes rows in batches of 10,000, committing after each batch, until no eligible rows remain.
+
+
+Scheduling regular cleanup
Review Comment:
Umm, I do not think we should mention "schedule"
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
Review Comment:
```suggestion
* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``.
```
Negative isn't possible
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
+* Keys written with ``retention=NEVER_EXPIRE`` have ``expires_at = NULL`` and a flag that marks them as permanent. They are **never** deleted by this command regardless of configuration.
+
+If ``[state_store] default_retention_days = 0``, keys written without an explicit retention have ``expires_at = NULL`` (no expiry) and are also skipped. Only keys with a non-null, past ``expires_at`` are removed.
+
+.. note::
+
+ Custom backends (``[state_store] backend`` set to anything other than the default) are **explicitly skipped**. The cleanup command prints a message and exits cleanly without deleting anything. If your custom backend needs its own retention logic, implement it in ``BaseStateBackend.cleanup()`` and call it from your own maintenance process.
+
+
+Running cleanup
+---------------
+
+The command is::
+
+ airflow state-store cleanup-task-states
+
+It reads ``[state_store] default_retention_days`` and ``[state_store] state_cleanup_batch_size`` from the ``airflow.cfg`` file, then deletes all eligible rows.
+
+**Dry run**
+
+Use ``--dry-run`` to preview what would be deleted without removing anything::
+
+ airflow state-store cleanup-task-states --dry-run
+
+The output lists every row that would be deleted, grouped by dag, run, task, map index, and key.
+
+**Batching**
+
+By default (``state_cleanup_batch_size = 0``) all eligible rows are deleted in a single statement. On deployments with large ``task_state`` tables, set a batch size to reduce lock duration per transaction::
+
+ # airflow.cfg
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+The command then deletes rows in batches of 10,000, committing after each batch, until no eligible rows remain.
+
+
+Scheduling regular cleanup
+--------------------------
+
+Because Airflow does not run cleanup automatically, you need to schedule it. The recommended approaches are:
+
+**System cron**
+
+Add a cron entry on the host that runs the Airflow scheduler or API server:
+
+.. code-block:: bash
+
+ # Run cleanup daily at 02:00
+ 0 2 * * * airflow state-store cleanup-task-states
+
+**Airflow Dag**
+
+You can also drive cleanup from a Dag so it is visible in the UI and benefits from Airflow's retry and alerting:
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ with DAG(
+ "state_store_cleanup",
+ schedule=timedelta(days=1),
+ catchup=False,
+ ):
+ BashOperator(
+ task_id="cleanup_task_states",
+ bash_command="airflow state-store cleanup-task-states",
+ )
Review Comment:
Unsure if we need this section, we should just focus on the CLI command and let the users handle it as they like
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,255 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </core-concepts/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-states``. This process removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
Review Comment:
```suggestion
Past rows are deleted, ie: rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
```
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
Review Comment:
Lets mention delete and clear here too
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
Review Comment:
Maybe add a Note that `None` cannot be set
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
Review Comment:
Mention example with `default` too
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_state = context["task_state"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_state.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_state.set("job_id", str(job_id), retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job. Another example of this sort of logic can be found in `example_task_state.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_state.py>`_.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ # Retrieve the task_state
+ task_state = context["task_state"]
+ raw = task_state.get("last_page")
+
+ start_page = raw + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_state.set("last_page", page) # Update the task_state for reuse later
+
+
+On a retry, the task reads ``last_page`` and skips pages that were already processed.
+
+Progress metadata
+~~~~~~~~~~~~~~~~~
+
+Task state can expose in-progress metrics for observability — row counts, status strings, or lightweight JSON payloads — without requiring XCom or an external system.
+
+.. code-block:: python
+
+ import json
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("row_ingest", schedule="@hourly"):
+
+ @task
+ def ingest_rows(**context):
+ task_state = context["task_state"]
+ total = 0
+
+ for batch in get_batches():
+ load(batch)
+ total += len(batch)
+ task_state.set(
+ "progress",
+ {
+ "rows_loaded": total,
+ "status": "running"
+ },
+ )
+
+ task_state.set(
+ "progress",
+ json.dumps({
+ "rows_loaded": total,
+ "status": "done"
+ }),
Review Comment:
```suggestion
},
```
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
Review Comment:
Now its `get(key, default)`
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_state = context["task_state"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_state.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_state.set("job_id", str(job_id), retention=NEVER_EXPIRE)
Review Comment:
```suggestion
task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
```
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
Review Comment:
It's possible:
```
from airflow.sdk import NEVER_EXPIRE
NEVER_EXPIRE
Out[3]: datetime.timedelta(days=999999999, seconds=86399, microseconds=999999)
```
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
+* Keys written with ``retention=NEVER_EXPIRE`` have ``expires_at = NULL`` and a flag that marks them as permanent. They are **never** deleted by this command regardless of configuration.
+
+If ``[state_store] default_retention_days = 0``, keys written without an explicit retention have ``expires_at = NULL`` (no expiry) and are also skipped. Only keys with a non-null, past ``expires_at`` are removed.
+
+.. note::
+
+ Custom backends (``[state_store] backend`` set to anything other than the default) are **explicitly skipped**. The cleanup command prints a message and exits cleanly without deleting anything. If your custom backend needs its own retention logic, implement it in ``BaseStateBackend.cleanup()`` and call it from your own maintenance process.
Review Comment:
```suggestion
Custom backends (``[state_store] backend`` set to anything other than the default) are **explicitly not cleaned up**. The cleanup command prints a message and exits cleanly without deleting anything. If your custom backend needs its own retention logic, implement it in ``BaseStateBackend.cleanup()`` and call it from your own maintenance process.
```
Better?
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
Review Comment:
```suggestion
Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
```
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,255 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </core-concepts/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-states``. This process removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+.. important::
+
+ Garbage collection only works for the ``MetastoreStateBackend``. Custom backends are explicitly skipped.
+
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 55 45
+
+ * - Signature
+ - Description
+ * - ``def get(self, scope: StateScope, key: str, *, session: Session | None = None) -> str | None``
+ - Return the stored value, or ``None``.
+ * - ``def set(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: Session | None = None) -> None``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``def delete(self, scope: StateScope, key: str, *, session: Session | None = None) -> None``
+ - Delete a single key; no-op if absent.
+ * - ``def clear(self, scope: StateScope, *, all_map_indices: bool = False, session: Session | None = None) -> None``
+ - Delete all keys under the scope.
+ * - ``async def aget(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> str | None``
+ - Async variant of ``get``.
+ * - ``async def aset(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: AsyncSession | None = None) -> None``
+ - Async variant of ``set``.
+ * - ``async def adelete(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> None``
+ - Async variant of ``delete``.
+ * - ``async def aclear(self, scope: StateScope, *, all_map_indices: bool = False, session: AsyncSession | None = None) -> None``
+ - Async variant of ``clear``.
Review Comment:
Now that I think of it, since these are signatures that can change, maybe we do not mention it at all like how secrets backend does: https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html?
They just mention that the interface must be implemented
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,255 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </core-concepts/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-states``. This process removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+.. important::
+
+ Garbage collection only works for the ``MetastoreStateBackend``. Custom backends are explicitly skipped.
+
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 55 45
+
+ * - Signature
+ - Description
+ * - ``def get(self, scope: StateScope, key: str, *, session: Session | None = None) -> str | None``
+ - Return the stored value, or ``None``.
+ * - ``def set(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: Session | None = None) -> None``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``def delete(self, scope: StateScope, key: str, *, session: Session | None = None) -> None``
+ - Delete a single key; no-op if absent.
+ * - ``def clear(self, scope: StateScope, *, all_map_indices: bool = False, session: Session | None = None) -> None``
+ - Delete all keys under the scope.
+ * - ``async def aget(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> str | None``
+ - Async variant of ``get``.
+ * - ``async def aset(self, scope: StateScope, key: str, value: str, *, expires_at: datetime | None = None, session: AsyncSession | None = None) -> None``
+ - Async variant of ``set``.
+ * - ``async def adelete(self, scope: StateScope, key: str, *, session: AsyncSession | None = None) -> None``
+ - Async variant of ``delete``.
+ * - ``async def aclear(self, scope: StateScope, *, all_map_indices: bool = False, session: AsyncSession | None = None) -> None``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if isinstance(scope, TaskScope):
+ return self._task_store.get(scope, key)
+ elif isinstance(scope, AssetScope):
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly using worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``def serialize_task_state_to_ref(self, *, value: str, key: str, ti_id: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``def deserialize_task_state_from_ref(self, stored: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``def serialize_asset_state_to_ref(self, *, value: str, key: str, asset_ref: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``def deserialize_asset_state_from_ref(self, stored: str) -> str``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
Review Comment:
Same thought as above
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
+
+Task and Asset State
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task state**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run
Review Comment:
Mention resume from progress made by past run?
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
+
+Task and Asset State
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task state**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run
+ * - **Asset state**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and JSON values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <state-store:worker-backends>`.
+
+When to use Task and Asset State
+--------------------------------
+
+Use this table to choose the right mechanism for your use case.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 22 78
+
+ * - Mechanism
+ - When to use it
+ * - **XCom**
+ - Pass data *between tasks* within a single Dag run (e.g. the output of one task consumed by a downstream task). XComs are cleared on retry, and should NOT be used to persist state across task retries or across runs.
Review Comment:
Across dag runs is possible, so we should not mention in a single dag run
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
+
+Task and Asset State
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task state**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run
+ * - **Asset state**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and JSON values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <state-store:worker-backends>`.
+
+When to use Task and Asset State
+--------------------------------
+
+Use this table to choose the right mechanism for your use case.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 22 78
+
+ * - Mechanism
+ - When to use it
+ * - **XCom**
+ - Pass data *between tasks* within a single Dag run (e.g. the output of one task consumed by a downstream task). XComs are cleared on retry, and should NOT be used to persist state across task retries or across runs.
+ * - **Variables**
+ - Dag-wide or installation-wide configuration that changes infrequently and is set by operators rather than by tasks themselves.
Review Comment:
```suggestion
- Deployment wide configuration store that changes infrequently and is set by deployment managers / users rather than by tasks themselves.
```
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
Review Comment:
```suggestion
from airflow.sdk import NEVER_EXPIRE
```
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
Review Comment:
```suggestion
from airflow.sdk import NEVER_EXPIRE
```
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,229 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+When using asset state within a task, ``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+If using asset state in a ``BaseEventTrigger``, the ``self.asset_state`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_state"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+To see asset state in-action in a real DAG, checkout the DAG in `example_asset_state.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_state.py>`_.
+
+Accessing asset state in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset state can be retrieved using the ``self.asset_state`` attribute.
+
+.. code-block:: python
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_state = self.asset_state[my_data]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_state = self.asset_state[my_data]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_state"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_state = context["asset_state"]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_state"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_state"][my_asset]``), the shorthand (``context["asset_state"]``) when the task has exactly one inlet, and when using the ``self.asset_state`` attribute.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_state"][my_asset].get("watermark")
+
+ # Using self.asset_state
+ watermark = self.asset_state[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike Task state, asset state has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using self.asset_state
+ self.asset_state[my_asset].set("watermark", "2024-06-01T00:00:00Z")
Review Comment:
Worth mentioning that None cannot be set or the types that can be set
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
Review Comment:
Or rather we shouldn't at all allow it
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_state = context["task_state"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_state.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_state.set("job_id", str(job_id), retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job. Another example of this sort of logic can be found in `example_task_state.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_state.py>`_.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ # Retrieve the task_state
+ task_state = context["task_state"]
+ raw = task_state.get("last_page")
+
+ start_page = raw + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_state.set("last_page", page) # Update the task_state for reuse later
+
+
+On a retry, the task reads ``last_page`` and skips pages that were already processed.
+
+Progress metadata
+~~~~~~~~~~~~~~~~~
+
+Task state can expose in-progress metrics for observability — row counts, status strings, or lightweight JSON payloads — without requiring XCom or an external system.
+
+.. code-block:: python
+
+ import json
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("row_ingest", schedule="@hourly"):
+
+ @task
+ def ingest_rows(**context):
+ task_state = context["task_state"]
+ total = 0
+
+ for batch in get_batches():
+ load(batch)
+ total += len(batch)
+ task_state.set(
+ "progress",
+ {
+ "rows_loaded": total,
+ "status": "running"
+ },
+ )
+
+ task_state.set(
+ "progress",
+ json.dumps({
Review Comment:
```suggestion
{
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3339652988
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
Review Comment:
It wasn't checked, fix here: https://github.com/apache/airflow/pull/67890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4635254279
@amoghrajesh - final feedback implement, and checks passing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3366363389
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
Review Comment:
The class is `BaseStoreBackend`, not `BaseStateBackend`. It is exported as `airflow.sdk.state.BaseStoreBackend` ([state.py](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/task-sdk/src/airflow/sdk/state.py#L21-L25)) and the backend resolver checks `issubclass(clazz, BaseStoreBackend)` ([state/__init__.py L36-L38](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/airflow-core/src/airflow/state/__init__.py#L36-L38)). `BaseStateBackend` does not exist, so `from airflow.sdk.state import BaseStateBackend` (lines 131 and 173) raises ImportError and the `:class:` xrefs will not resolve. Needs fixing at lines 39, 125, 131, 134, 156, 179. The metastore class is also `MetastoreStoreBackend`, not `MetastoreStateBackend` (lines 118 and 209).
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
Review Comment:
There is no `self.asset_store` attribute. Asset store is reached only through the task context (`context["asset_store"]`, optionally subscripted by the asset), implemented by `AssetStoreAccessors` ([context.py L774-L849](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/task-sdk/src/airflow/sdk/execution_time/context.py#L774-L849)); there is no operator-level `self` accessor. Every `self.asset_store[...]` example in this section (lines 93, 113, 126, 139) raises `AttributeError`. Suggest dropping the `self.asset_store` column and keeping `context["asset_store"]` as the single documented path.
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored JSON value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark", default="initial_watermark")
+
+ # Using self.asset_store
+ watermark = self.asset_store[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike task store, asset store has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated. Like with task store, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].set("watermark", default="2024-06-01T00:00:00Z")
Review Comment:
`set()` takes the value positionally and has no `default` parameter; the signature is `set(key, value)` ([context.py L835](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/task-sdk/src/airflow/sdk/execution_time/context.py#L835)). `set("watermark", default="...")` raises `TypeError`. It should be `set("watermark", "2024-06-01T00:00:00Z")`. Same on line 113.
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,278 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key", default="my_default_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+---------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored JSON value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get(
+ "job_id", default="123456789"
+ ) # returns the value associated with `job_id` or the default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped during garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
Review Comment:
The prose says to import `NEVER_EXPIRE` from `airflow.sdk.execution_time.context`, but the code block just below uses `from airflow.sdk import NEVER_EXPIRE`. Both resolve (it is defined in `execution_time.context` and re-exported via `airflow.sdk`), but `execution_time.context` is the internal module and `airflow.sdk` is the public export. Recommend the prose point at `airflow.sdk` to match the code and the other examples.
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
Review Comment:
This availability rule undercounts outlets. In `AssetStoreAccessors.__init__` the outlet loop adds an accessor for each `Asset`/`AssetNameRef`/`AssetUriRef` outlet, and `_total = len(by_name) + len(by_uri)` counts inlets and outlets together ([context.py L803-L825](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/task-sdk/src/airflow/sdk/execution_time/context.py#L803-L825)); the shorthand raises only when `_total != 1` ("Task has N concrete inlets and outlets"). So this should read "inlets or outlets", line 64 should be "exactly one concrete inlet or outlet", and line 74 likewise. It matters for the headline case: a watermark writer naturally declares the asset as an outlet only, and as written the docs tell that user the store is unavailable.
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days after which task store rows expire. When a key is written with no explicit retention, expires_at is computed on the worker as now + default_retention_days. Changing this setting does not affect already-written rows.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset store rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task store keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task store entries after success for observability (e.g. the submitted job ID or the last row count is still readable from the UI or REST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task store only**. It has no effect on asset store. Asset store is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_store`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+.. _task-and-asset-store:worker-backends:
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task store and asset store values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStoreAccessor.set()`` calls ``serialize_task_store_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_store_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-store``. This process removes store rows according to the following rules:
+
+**Time-based expiry (task store only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task store only)**
+ Keys written with no explicit retention get an ``expires_at`` of now + default_retention_days computed at write time. Garbage collection deletes rows where ``expires_at < now()``."
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset store)**
Review Comment:
I cannot find an orphan-sweep implementation. `MetastoreStoreBackend.cleanup()` deletes `task_store` rows only (`WHERE expires_at < now()`, [metastore.py L376-L404](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/airflow-core/src/airflow/state/metastore.py#L376-L404)); the `state-store cleanup-task-store` CLI just calls it, and nothing in the scheduler sweeps asset store. Asset store rows are removed only by the FK `ondelete=CASCADE` ([asset_store.py L56-L64](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/airflow-core/src/airflow/models/asset_store.py#L56-L64)) when the parent `asset` row is hard-deleted, which is not the same as "no `asset_active` record" (a deactivated asset keeps its `asset` row). So as written, deactivated or renamed assets keep their store rows indefinitely. Either the sweep needs implementing, or this bullet (and asset-store.rst's "removed during the next garbage-collection pass") should descri
be the cascade-on-delete behavior instead.
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require some amount of data to be persisted outside of a task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Task and Asset store provide two key/value stores to persist data like a job ID, watermark, or row count. These two stores are differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept JSON-able values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
Review Comment:
I cannot find where the 64 KB limit is enforced. `value` is `Text().with_variant(MEDIUMTEXT, "mysql")` (~1 GB on Postgres, 16 MB on MySQL), and the only `value` validator on the write path checks JSON-representability and rejects null/NaN/Inf, not size ([execution_api/datamodels/task_store.py L40-L46](https://github.com/apache/airflow/blob/d88504564c506e7edfbd64cb0589f6ac9d6263c5/airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_store.py#L40-L46)). If there is no size check, this states a hard limit that does not exist and points users at a worker-side backend they may not need. Could you point to the enforcement, or soften this to a recommended size?
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,278 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
Review Comment:
The examples reach the store via `context["task_store"]` inside `def my_task(**context):`, but the text never says the task has to accept `**context` (or call `airflow.sdk.get_current_context()`). A reader using the common TaskFlow form `def my_task():` will hit a `NameError`. One sentence by this first example, and showing the `get_current_context()` form once, would close the gap.
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,278 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key", default="my_default_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+---------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored JSON value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get(
+ "job_id", default="123456789"
+ ) # returns the value associated with `job_id` or the default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped during garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk import NEVER_EXPIRE
+
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_store.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* task store keys for this task instance.
+
+For :doc:`mapped tasks </authoring-and-scheduling/dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe the store across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_store.clear()
+
+ # clear all map indices (fleet-wide)
+ task_store.clear(all_map_indices=True)
+
+
+Some Example Use Cases
+----------------------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk import NEVER_EXPIRE
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_store = context["task_store"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_store.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job. Another example of this sort of logic can be found in `example_task_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_store.py>`_.
+
+For ``BaseOperator`` subclasses, the :class:`~airflow.sdk.bases.resumablemixin.ResumableJobMixin` encapsulates this pattern. It persists the external job ID to task store after submission and, on retry, reconnects to an active job or resubmits if the prior job reached a terminal failure state.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ # Retrieve the task_store
+ task_store = context["task_store"]
+ raw = task_store.get("last_page")
+
+ start_page = raw + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_store.set("last_page", page) # Update the task_store for reuse later
+
+
+On a retry, the task reads ``last_page`` and skips pages that were already processed.
+
+Progress metadata
+~~~~~~~~~~~~~~~~~
+
+Task store can expose in-progress metrics for observability — row counts, status strings, or lightweight JSON payloads — without requiring XCom or an external system.
+
+.. code-block:: python
+
+
+ from airflow.sdk import DAG, task
+
+ with DAG("row_ingest", schedule="@hourly"):
+
+ @task
+ def ingest_rows(**context):
+ task_store = context["task_store"]
+ total = 0
+
+ for batch in get_batches():
+ load(batch)
+ total += len(batch)
+ task_store.set(
+ "progress",
+ {"rows_loaded": total, "status": "running"},
+ )
+
+ task_store.set(
+ "progress",
+ {"rows_loaded": total, "status": "done"},
+ )
+
+The ``progress`` key is visible through the REST API and the Airflow UI while the task is running.
Review Comment:
This presents REST/UI visibility as a feature with no caveat. Unlike Variables and Connections, store values are persisted as a plain `Text` column with no Fernet encryption, and they are readable through the REST API and UI by anyone with access to the task or asset. Worth one explicit line here: do not put secrets or credentials in task or asset store.
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
Review Comment:
A reader meets "Task and Asset Store" throughout the concept pages, then lands on `[state_store]`, the `state-store` CLI, and `BaseStoreBackend`. This note only warns "not `[task_store]`"; the harder gap is that the whole config, CLI, and class namespace is `state_store`/`state-store`, which never appears in the concept docs. One sentence in the overview, something like "configuration, the CLI, and backend classes use the `state_store` name for this feature", would connect the two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3356184022
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
Review Comment:
I don't think we need to mention versions, cos docs are versioned?
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and JSON values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
+
+When to use Task and Asset Store
+--------------------------------
+
+Use this table to choose the right mechanism for your use case.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 22 78
+
+ * - Mechanism
+ - When to use it
+ * - **XCom**
+ - Pass data *between tasks* within a single Dag run (e.g. the output of one task consumed by a downstream task) or across different multiple Dag runs (referencing the data persisted from another run). XComs are cleared on retry, and should NOT be used to persist data across task retries or across runs.
+ * - **Variables**
+ - Dag-wide or installation-wide configuration that changes infrequently and is set by operators rather than by tasks themselves.
Review Comment:
Is it better to call this deployment wide?
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and JSON values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
+
+When to use Task and Asset Store
+--------------------------------
+
+Use this table to choose the right mechanism for your use case.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 22 78
+
+ * - Mechanism
+ - When to use it
+ * - **XCom**
+ - Pass data *between tasks* within a single Dag run (e.g. the output of one task consumed by a downstream task) or across different multiple Dag runs (referencing the data persisted from another run). XComs are cleared on retry, and should NOT be used to persist data across task retries or across runs.
+ * - **Variables**
+ - Dag-wide or installation-wide configuration that changes infrequently and is set by operators rather than by tasks themselves.
+ * - **Task store**
+ - Data that must survive a worker crash or a retry within the **same run**. An external job ID written before a long-running job completes is a perfect use case for task store.
Review Comment:
```suggestion
- Data that must survive a worker crash or data that must survive across retries within the **same run**. An external job ID written before a long-running job completes is a perfect use case for task store.
```
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
Review Comment:
```suggestion
value = task_store.get("job_id", default="123456789") # returns the value associated with `job_id` or the default value
```
Unclear if we dont pass `default` here
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk import NEVER_EXPIRE
+
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_store.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* task store keys for this task instance.
+
+For :doc:`mapped tasks </authoring-and-scheduling/dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe the store across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_store.clear()
+
+ # clear all map indices (fleet-wide)
+ task_store.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk import NEVER_EXPIRE
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_store = context["task_store"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_store.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job. Another example of this sort of logic can be found in `example_task_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_store.py>`_.
Review Comment:
Should we mention the `ResumableJobMixin` class here?
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
Review Comment:
```suggestion
An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
```
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark")
Review Comment:
```suggestion
# Using context
watermark = context["asset_store"][my_asset].get("watermark", default="initial_watermark")
```
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key)``
Review Comment:
```suggestion
``get(key, default)``
```
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and JSON values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
Review Comment:
```suggestion
Both stores accept JSON-able values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
```
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
Review Comment:
```suggestion
Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require some amount of data to be persisted outside of a yask's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
```
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
Review Comment:
```suggestion
my_value = task_store.get("my_key", default="my_default_key")
```
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
Review Comment:
UTC is irrelevant detail?
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
Review Comment:
```suggestion
* ``NEVER_EXPIRE``: the key never expires and is skipped during garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
```
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk import NEVER_EXPIRE
+
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_store.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* task store keys for this task instance.
+
+For :doc:`mapped tasks </authoring-and-scheduling/dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe the store across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_store.clear()
+
+ # clear all map indices (fleet-wide)
+ task_store.clear(all_map_indices=True)
+
+
+Use Cases
Review Comment:
```suggestion
Some Example Use Cases
```
IDK, is this better?
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk import NEVER_EXPIRE
+
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_store.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* task store keys for this task instance.
+
+For :doc:`mapped tasks </authoring-and-scheduling/dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe the store across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_store.clear()
+
+ # clear all map indices (fleet-wide)
+ task_store.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk import NEVER_EXPIRE
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_store = context["task_store"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_store.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job. Another example of this sort of logic can be found in `example_task_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_store.py>`_.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ # Retrieve the task_store
+ task_store = context["task_store"]
+ raw = task_store.get("last_page")
+
+ start_page = raw + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_store.set("last_page", page) # Update the task_store for reuse later
+
+
+On a retry, the task reads ``last_page`` and skips pages that were already processed.
+
+Progress metadata
+~~~~~~~~~~~~~~~~~
+
+Task store can expose in-progress metrics for observability — row counts, status strings, or lightweight JSON payloads — without requiring XCom or an external system.
+
+.. code-block:: python
+
+ import json
Review Comment:
```suggestion
```
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
Review Comment:
I suggest you move documentation related to `BaseEventTrigger` in a follow up OR in the PR where you add support for that since that PR has not been merged yet, we can close this initial framework sooner that way.
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark")
+
+ # Using self.asset_store
+ watermark = self.asset_store[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike task store, asset store has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated. Like with task store, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using self.asset_store
+ self.asset_store[my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].delete("watermark")
+
+ # Using self.asset_store
+ self.asset_store[my_asset].delete("watermark")
+
+``clear()``
+~~~~~~~~~~~
+
+Deletes *all* asset store keys for the asset.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].clear()
+
+ # Using self.asset_store
+ self.asset_store[my_asset].clear()
+
+Use cases
Review Comment:
```suggestion
Some Example Use cases
```
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
Review Comment:
We should also mention an example as to how it looks like when multiple inlets are present, aka the multi inlet pattern
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark")
+
+ # Using self.asset_store
+ watermark = self.asset_store[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike task store, asset store has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated. Like with task store, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using self.asset_store
+ self.asset_store[my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].delete("watermark")
+
+ # Using self.asset_store
+ self.asset_store[my_asset].delete("watermark")
+
+``clear()``
+~~~~~~~~~~~
+
+Deletes *all* asset store keys for the asset.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].clear()
+
+ # Using self.asset_store
+ self.asset_store[my_asset].clear()
+
+Use cases
+---------
+
+Watermark pattern
+~~~~~~~~~~~~~~~~~
+
+The canonical use case for asset store is an incremental-load task that advances a watermark on each run. The watermark is stored on the asset itself so any task that reads or writes that asset can access it. This use case is especially applicable when building things like asset "watchers" using ``BaseEventTrigger``'s.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ orders = Asset("orders", uri="s3://data/orders/")
+
+
+ with DAG("incremental_orders", schedule="@daily"):
+
+ @task(inlets=[orders], outlets=[orders])
+ def load_new_orders(**context):
+ asset_store = context["asset_store"] # single-inlet shorthand
+
+ # Read the last watermark, default to epoch if first run.
+ watermark = asset_store.get("watermark") or "1970-01-01T00:00:00Z"
Review Comment:
```suggestion
watermark = asset_store.get("watermark", default="1970-01-01T00:00:00Z")
```
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task store** rows after their last update. Rows older than this are deleted during the next garbage collection pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset store rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task store keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task store entries after success for observability (e.g. the submitted job ID or the last row count is still readable from the UI or REST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task store only**. It has no effect on asset store. Asset store is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_store`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+.. _task-and-asset-store:worker-backends:
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task store and asset store values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStoreAccessor.set()`` calls ``serialize_task_store_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_store_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The cleanup task, also known as "garbage collection" is triggered using the Airflow CLI. The command to trigger the cleanup task is ``airflow state-store cleanup-task-store``. This process removes store rows according to the following rules:
+
+**Time-based expiry (task store only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task store only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the garbage collection job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the garbage collection to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset store)**
+ Asset store rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up store entries for deactivated or renamed assets.
+
+.. important::
+
+ Garbage collection only works for the ``MetastoreStateBackend``. Custom backends are explicitly skipped.
+
+
+
+Custom backends
+---------------
+
+A custom backend must subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement its abstract methods: ``get``, ``set``, ``delete``, and ``clear`` for synchronous callers and the ``aget``, ``aset``, ``adelete``, and ``aclear`` async equivalents. Refer to :class:`~airflow.sdk.state.BaseStateBackend` for the full API.
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use ``isinstance`` to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if isinstance(scope, TaskScope):
+ return self._task_store.get(scope, key)
+ elif isinstance(scope, AssetScope):
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly using worker infrastructure while only a compact reference string is kept in the database.
+
+Override four serialization hooks from :class:`~airflow.sdk.state.BaseStateBackend`:
+
+* ``serialize_task_store_to_ref``: called by ``TaskStoreAccessor.set()`` before the value is sent to the Execution API; return a compact reference string (e.g. an S3 key) to be stored in the database instead of the raw value.
+* ``deserialize_task_store_from_ref``: called by ``TaskStoreAccessor.get()`` after retrieving the reference; return the actual value.
Review Comment:
```suggestion
* ``deserialize_task_store_from_ref``: called by ``TaskStoreAccessor.get()`` after retrieving the reference from the backend; return the actual value.
```
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored JSON value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_store"][my_asset].get("watermark")
+
+ # Using self.asset_store
+ watermark = self.asset_store[my_asset].get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike task store, asset store has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated. Like with task store, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using self.asset_store
+ self.asset_store[my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].delete("watermark")
+
+ # Using self.asset_store
+ self.asset_store[my_asset].delete("watermark")
+
+``clear()``
+~~~~~~~~~~~
+
+Deletes *all* asset store keys for the asset.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_store"][my_asset].clear()
+
+ # Using self.asset_store
+ self.asset_store[my_asset].clear()
+
+Use cases
+---------
+
+Watermark pattern
+~~~~~~~~~~~~~~~~~
+
+The canonical use case for asset store is an incremental-load task that advances a watermark on each run. The watermark is stored on the asset itself so any task that reads or writes that asset can access it. This use case is especially applicable when building things like asset "watchers" using ``BaseEventTrigger``'s.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ orders = Asset("orders", uri="s3://data/orders/")
+
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313056275
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,261 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key. Note, ``value`` must be a string.
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313412807
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
Review Comment:
Done in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313110680
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
Review Comment:
Waiting on response from other comment, will address then!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313107957
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
Review Comment:
Thanks, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4579210554
Removed `state-api` docs, added docs for cleanup! Ready for re-review (I think it's GTG).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3326454931
##########
airflow-core/docs/administration-and-deployment/state-api.rst:
##########
@@ -0,0 +1,142 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-api:
+
+State REST API
Review Comment:
Removed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342733213
##########
airflow-core/docs/administration-and-deployment/state-cleanup.rst:
##########
@@ -0,0 +1,109 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-cleanup:
+
+State Cleanup
+=============
+
+.. versionadded:: 3.3
+
+Airflow does not automatically purge task state rows on a schedule. Cleanup (also known as "garbage collection") is the responsibility of the user (you) and must be triggered explicitly via the CLI. This page explains what gets cleaned up, how to run it, and how to integrate it into a recurring maintenance workflow.
+
+
+What gets cleaned up
+--------------------
+
+The cleanup command operates only on **task state** rows in the ``MetastoreStateBackend``. Asset state rows are never touched by this command. Asset state rows are removed only by the orphan sweep when an asset is deactivated (see :ref:`state-store`).
+
+A task state row is eligible for deletion when its ``expires_at`` timestamp is in the past. ``expires_at`` is computed on the worker at write time:
+
+* Keys written with an explicit ``retention=timedelta(...)`` expire after that duration from the time of the write.
+* Keys written with ``retention=None`` (the default) pick up an expiry based on ``[state_store] default_retention_days``. If that value is ``> 0``, the key expires that many days after the write.
+* Keys written with ``retention=NEVER_EXPIRE`` have ``expires_at = NULL`` and a flag that marks them as permanent. They are **never** deleted by this command regardless of configuration.
+
+If ``[state_store] default_retention_days = 0``, keys written without an explicit retention have ``expires_at = NULL`` (no expiry) and are also skipped. Only keys with a non-null, past ``expires_at`` are removed.
+
+.. note::
+
+ Custom backends (``[state_store] backend`` set to anything other than the default) are **explicitly skipped**. The cleanup command prints a message and exits cleanly without deleting anything. If your custom backend needs its own retention logic, implement it in ``BaseStateBackend.cleanup()`` and call it from your own maintenance process.
+
+
+Running cleanup
+---------------
+
+The command is::
+
+ airflow state-store cleanup-task-states
+
+It reads ``[state_store] default_retention_days`` and ``[state_store] state_cleanup_batch_size`` from the ``airflow.cfg`` file, then deletes all eligible rows.
+
+**Dry run**
+
+Use ``--dry-run`` to preview what would be deleted without removing anything::
+
+ airflow state-store cleanup-task-states --dry-run
+
+The output lists every row that would be deleted, grouped by dag, run, task, map index, and key.
+
+**Batching**
+
+By default (``state_cleanup_batch_size = 0``) all eligible rows are deleted in a single statement. On deployments with large ``task_state`` tables, set a batch size to reduce lock duration per transaction::
+
+ # airflow.cfg
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+The command then deletes rows in batches of 10,000, committing after each batch, until no eligible rows remain.
+
+
+Scheduling regular cleanup
+--------------------------
+
+Because Airflow does not run cleanup automatically, you need to schedule it. The recommended approaches are:
+
+**System cron**
+
+Add a cron entry on the host that runs the Airflow scheduler or API server:
+
+.. code-block:: bash
+
+ # Run cleanup daily at 02:00
+ 0 2 * * * airflow state-store cleanup-task-states
+
+**Airflow Dag**
+
+You can also drive cleanup from a Dag so it is visible in the UI and benefits from Airflow's retry and alerting:
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ with DAG(
+ "state_store_cleanup",
+ schedule=timedelta(days=1),
+ catchup=False,
+ ):
+ BashOperator(
+ task_id="cleanup_task_states",
+ bash_command="airflow state-store cleanup-task-states",
+ )
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3357080333
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
Review Comment:
Updated in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3357121666
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,275 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+-------------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored string value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get("job_id", "123456789") # returns str or default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk import NEVER_EXPIRE
+
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_store.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* task store keys for this task instance.
+
+For :doc:`mapped tasks </authoring-and-scheduling/dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe the store across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_store.clear()
+
+ # clear all map indices (fleet-wide)
+ task_store.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk import NEVER_EXPIRE
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_store = context["task_store"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_store.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_store.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job. Another example of this sort of logic can be found in `example_task_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_store.py>`_.
Review Comment:
Added in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3366380556
##########
airflow-core/docs/core-concepts/task-and-asset-store.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-and-asset-store-overview:
+
+Task and Asset Store Overview
+=============================
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require some amount of data to be persisted outside of a task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task store and Asset store fill that gap without touching the XCom or Variable systems.
+
+Task and Asset Store
+--------------------
+
+Task and Asset store provide two key/value stores to persist data like a job ID, watermark, or row count. These two stores are differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task store**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run, resume progress from checkpoint set by a past run
+ * - **Asset store**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept JSON-able values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <task-and-asset-store:worker-backends>`.
Review Comment:
cc @amoghrajesh ^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367303701
##########
airflow-core/docs/core-concepts/task-store.rst:
##########
@@ -0,0 +1,278 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-store:
+
+.. spelling:word-list::
+
+ intra
+ Intra
+ checkpointing
+
+Task Store
+==========
+
+.. versionadded:: 3.3
+
+Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Data persisted via task store is accessed through the task context via ``context["task_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task store
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task store is available through the ``context`` dictionary via the ``task_store`` key. From there, it can be used to retrieve, set, delete, or clear data for a specific key-value pair. In this example, the ``job_id`` is retrieved from task store, then updated, before being deleted. All data for that task is then removed using the ``clear`` method.
+
+.. code-block:: python
+
+ from airflow.sdk import task
+ import random
+
+
+ @task
+ def my_task(**context):
+ # Retrieve task_store from context
+ task_store = context["task_store"]
+ my_value = task_store.get("my_key", default="my_default_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_store.set("my_key", new_value)
+
+ # Delete the value
+ task_store.delete("my_key")
+
+ # Clear all store entries for the task
+ task_store.clear()
+
+Reference
+---------
+
+``get(key, default)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the stored JSON value, or the ``default`` value if the key does not exist.
+
+.. code-block:: python
+
+ value = task_store.get(
+ "job_id", default="123456789"
+ ) # returns the value associated with `job_id` or the default value
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, except for ``None``. This includes:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped during garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_store.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_store.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313115446
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
+
+.. important::
+
+ **References must be deterministic.** Given the same inputs (``ti_id`` + ``key`` for task state; ``asset_ref`` + ``key`` for asset state), the serialization method must always return the same reference string. Do not embed timestamps, random UUIDs, or any other non-deterministic component in the reference path.
+
+ When a key is deleted or cleared, Airflow clears the database reference *first*, then calls the backend's ``delete()`` or ``clear()`` method. If backend cleanup fails after the DB row is gone, the external object is orphaned. Because the reference is deterministic, a subsequent ``set()`` for the same key will overwrite the orphaned object, making the failure recoverable. A non-deterministic reference would leave the external object permanently orphaned with no way to locate it.
+
+Example skeleton:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class S3StateBackend(BaseStateBackend):
+
+ def _task_ref(self, ti_id: str, key: str) -> str:
+ return f"airflow/task-state/{ti_id}/{key}"
+
+ def _asset_ref(self, asset_ref: str, key: str) -> str:
+ import hashlib
+
+ safe = hashlib.sha256(asset_ref.encode()).hexdigest()[:16]
+ return f"airflow/asset-state/{safe}/{key}"
+
+ def serialize_task_state_to_ref(self, *, value, key, ti_id):
+ s3_key = self._task_ref(ti_id, key)
+ s3_client.put_object(Bucket=BUCKET, Key=s3_key, Body=value.encode())
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313411850
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
Review Comment:
From other comment:
> outlets are never added to the accessors. The outlets-only KeyError behavior is a consequence of this, not a separate known issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313251911
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Accessing asset state using ``AssetState``
+------------------------------------------
+
+Asset state can also be retrieved for an asset using the ``airflow.sdk.AssetState`` class. This approach does NOT require that an asset is passed to a task using ``inlets``.
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(name="my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, the ``name`` of the asset is used to retrieve it's state. However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(uri="s3://bucket/my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
Review Comment:
Updated docs to not reference `AssetState` anymore!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367322862
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3357041637
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+------------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+If using asset store in a ``BaseEventTrigger``, the ``self.asset_store`` parameter can be used within the ``BaseEventTrigger``. It can be subscripted in the same way that ``context["asset_store"]`` can be.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_store"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+
+ # my_asset defined above ...
+
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_store"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release.
+
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_store"]`` with the asset object to retrieve the asset store.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Accessing asset store in a ``BaseEventTrigger``
+-----------------------------------------------
+
+When building Triggers used for asset "watching", asset store can be retrieved using the ``self.asset_store`` attribute.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(name="my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+In the example above, ``my_data`` is created using the ``name`` However, the ``uri`` can also be used:
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, BaseEventTrigger, TriggerEvent
+ from collections.abc import AsyncIterator
+
+
+ class GenericEventTrigger(BaseEventTrigger):
+ ...
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Logic that fires a TriggerEvent."""
+ my_data = Asset(uri="s3://bucket/my_data")
+ asset_store = self.asset_store[my_data]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
Review Comment:
Will be included in later PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313033321
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
+
+Task and Asset State
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task state**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run
+ * - **Asset state**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and string values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <state-store:worker-backends>`.
Review Comment:
Made the change to this:
> Both stores accept string keys and JSON values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313185111
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
Review Comment:
I think that this relates to the comment below. I'll leave it open for now until we decide how to tackle it.
> Guess we should just fix this instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313043555
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
Review Comment:
Correct, that is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3322562668
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
Review Comment:
This is the true signature:
```
async def aclear(
self, scope: StateScope, *, all_map_indices: bool = False, session: AsyncSession | None = None
)
```
having unnamed parameters kind of makes it confusing to me atleast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3324241160
##########
airflow-core/docs/administration-and-deployment/state-api.rst:
##########
@@ -0,0 +1,142 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-api:
+
+State REST API
Review Comment:
@Amogh, I think that makes sense (and is why I originally didn't include it). Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3325519123
##########
airflow-core/docs/administration-and-deployment/state-api.rst:
##########
@@ -0,0 +1,142 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-api:
+
+State REST API
Review Comment:
Yeah we should not have to need it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4512049705
cc: @amoghrajesh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342791952
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a value for the specified key. Note, ``value`` can be any JSON-compatible type, including:
+
+* ``str``
+* ``int``
+* ``float``
+* ``bool``
+* ``list``
+* ``dict``
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3309616194
##########
airflow-core/docs/core-concepts/task-and-asset-state.rst:
##########
@@ -0,0 +1,80 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:state-overview:
+
+Task and Asset State Overview
+========
+
+.. versionadded:: 3.3
+
+Airflow has always modeled tasks as stateless, idempotent units of work. A growing class of workloads, however, require a small amount of data to be persisted outside of a Task's return value, like a submitted job ID that must survive a worker crash, a watermark that advances run-by-run, or a row counter exposed for observability. Task state and Asset state fill that gap without touching the XCom or Variable systems.
+
+Task and Asset State
+--------------------
+
+Airflow 3.3 ships two persistent key/value stores, differentiated by *what* they are scoped to:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 25 25 30
+
+ * - Store
+ - Scope
+ - Default lifetime
+ - Primary use case
+ * - **Task state**
+ - A single task Instance (dag_id + run_id + task_id + map_index)
+ - Configurable retention; cleared on task success when ``clear_on_success = True``
+ - Survive retries, track in-flight jobs, checkpoint progress within a run
+ * - **Asset state**
+ - An asset (independent of any particular run)
+ - Persists indefinitely; removed only when the asset is deactivated
+ - Cross-run watermarks, incremental-load cursors, per-asset metadata
+
+Both stores accept string keys and string values. Values up to 64 KB are supported through the default metastore backend; larger payloads can be offloaded via a :ref:`custom worker-side backend <state-store:worker-backends>`.
Review Comment:
Values are `JsonValue` now, so we can probably call out as JSON strings instead. This will confuse users who try to store a dict or int.
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,261 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key. Note, ``value`` must be a string.
Review Comment:
Same JsonValue issue. We should say values can be any JSON-compatible type (str, int, float, bool, list, dict).
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
Review Comment:
Broken cross-reference. `asset-state.rst` lives at `core-concepts/asset-state`, not `authoring-and-scheduling/asset-state`?
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
Review Comment:
```suggestion
Number of rows deleted per batch during garbage collection cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
```
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
Review Comment:
Should use isinstance checks.
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
Review Comment:
```suggestion
When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the returned value (a reference to the actual storage) to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference from the Execution API. See `Custom worker-side backends`_ below.
```
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Accessing asset state using ``AssetState``
+------------------------------------------
+
+Asset state can also be retrieved for an asset using the ``airflow.sdk.AssetState`` class. This approach does NOT require that an asset is passed to a task using ``inlets``.
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(name="my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, the ``name`` of the asset is used to retrieve it's state. However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(uri="s3://bucket/my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
Review Comment:
Maybe remove this for now, you can revisit once the dependent PR lands.
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
Review Comment:
```suggestion
Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly using worker infrastructure while only a compact reference string is kept in the database.
```
Not sure if this is what you meant?
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
+
+.. important::
+
+ **References must be deterministic.** Given the same inputs (``ti_id`` + ``key`` for task state; ``asset_ref`` + ``key`` for asset state), the serialization method must always return the same reference string. Do not embed timestamps, random UUIDs, or any other non-deterministic component in the reference path.
+
+ When a key is deleted or cleared, Airflow clears the database reference *first*, then calls the backend's ``delete()`` or ``clear()`` method. If backend cleanup fails after the DB row is gone, the external object is orphaned. Because the reference is deterministic, a subsequent ``set()`` for the same key will overwrite the orphaned object, making the failure recoverable. A non-deterministic reference would leave the external object permanently orphaned with no way to locate it.
+
+Example skeleton:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class S3StateBackend(BaseStateBackend):
+
+ def _task_ref(self, ti_id: str, key: str) -> str:
+ return f"airflow/task-state/{ti_id}/{key}"
+
+ def _asset_ref(self, asset_ref: str, key: str) -> str:
+ import hashlib
+
+ safe = hashlib.sha256(asset_ref.encode()).hexdigest()[:16]
+ return f"airflow/asset-state/{safe}/{key}"
+
+ def serialize_task_state_to_ref(self, *, value, key, ti_id):
+ s3_key = self._task_ref(ti_id, key)
+ s3_client.put_object(Bucket=BUCKET, Key=s3_key, Body=value.encode())
+ return s3_key
+
+ def deserialize_task_state_from_ref(self, stored):
+ s3_object = s3_client.get_object(Bucket=BUCKET, Key=stored)
+ return s3_object["Body"].read().decode()
Review Comment:
```suggestion
def deserialize_task_state_from_ref(self, stored):
s3_object = s3_client.get_object(Bucket=BUCKET, Key=stored)
return json.loads(s3_object["Body"].read().decode())
```
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
Review Comment:
True signatures pls.
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
Review Comment:
Guess we should just fix this instead
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,261 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key. Note, ``value`` must be a string.
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_state = context["task_state"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_state.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_state.set("job_id", str(job_id), retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ task_state = context["task_state"]
+
+ raw = task_state.get("last_page")
+ start_page = int(raw) + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_state.set("last_page", str(page))
+
+On a retry, the task reads ``last_page`` and skips pages that were already processed.
+
+Progress metadata
+~~~~~~~~~~~~~~~~~
+
+Task state can expose in-progress metrics for observability — row counts, status strings, or lightweight JSON payloads — without requiring XCom or an external system.
+
+.. code-block:: python
+
+ import json
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("row_ingest", schedule="@hourly"):
+
+ @task
+ def ingest_rows(**context):
+ task_state = context["task_state"]
+ total = 0
+
+ for batch in get_batches():
+ load(batch)
+ total += len(batch)
+ task_state.set(
+ "progress",
+ json.dumps({
+ "rows_loaded": total,
+ "status": "running"
+ }),
+ )
+
+ task_state.set(
+ "progress",
+ json.dumps({
+ "rows_loaded": total,
+ "status": "done"
+ }),
Review Comment:
No need to do json.dumps anymore :)
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Accessing asset state using ``AssetState``
+------------------------------------------
+
+Asset state can also be retrieved for an asset using the ``airflow.sdk.AssetState`` class. This approach does NOT require that an asset is passed to a task using ``inlets``.
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(name="my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, the ``name`` of the asset is used to retrieve it's state. However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(uri="s3://bucket/my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_state"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_state = context["asset_state"]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_state"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_state"][my_asset]``), the shorthand (``context["asset_state"]``) when the task has exactly one inlet, and when using the ``AssetState`` Task SDK class.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
Review Comment:
Is returns a JsonValue - so maybe mention `Json` string?
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Accessing asset state using ``AssetState``
+------------------------------------------
+
+Asset state can also be retrieved for an asset using the ``airflow.sdk.AssetState`` class. This approach does NOT require that an asset is passed to a task using ``inlets``.
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(name="my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, the ``name`` of the asset is used to retrieve it's state. However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(uri="s3://bucket/my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_state"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_state = context["asset_state"]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_state"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_state"][my_asset]``), the shorthand (``context["asset_state"]``) when the task has exactly one inlet, and when using the ``AssetState`` Task SDK class.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_state"][my_asset].get("watermark")
+
+ # Using the Task SDK
+ AssetState(name="my_data").get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike Task state, asset state has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using the Task SDK class
+ AssetState(name="my_data").set("watermark", "2024-06-01T00:00:00Z")
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].delete("watermark")
+
+ # Using the Task SDK class
+ AssetState(name="my_data").delete("watermark")
+
+``clear()``
+~~~~~~~~~~~
+
+Deletes *all* state keys for the asset.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].clear()
+
+ # Using the Task SDK class
+ AssetState(name="my_data").clear()
+
+Use cases
+---------
+
+Watermark pattern
+~~~~~~~~~~~~~~~~~
+
+The canonical use case for asset state is an incremental-load task that advances a watermark on each run. The watermark is stored on the asset itself so any task that reads or writes that asset can access it. *This use case is especially applicable when building things like asset "watchers" using ``BaseEventTrigger``'s.
Review Comment:
```suggestion
The canonical use case for asset state is an incremental-load task that advances a watermark on each run. The watermark is stored on the asset itself so any task that reads or writes that asset can access it. This use case is especially applicable when building things like asset "watchers" using ``BaseEventTrigger``'s.
```
Formatting incident?
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,261 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key. Note, ``value`` must be a string.
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_state = context["task_state"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_state.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_state.set("job_id", str(job_id), retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ task_state = context["task_state"]
+
+ raw = task_state.get("last_page")
+ start_page = int(raw) + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_state.set("last_page", str(page))
+
Review Comment:
```suggestion
@task
def ingest_pages(**context):
task_state = context["task_state"]
raw = task_state.get("last_page")
start_page = raw + 1 if raw is not None else 1
for page in range(start_page, total_pages + 1):
fetch_and_load(page)
task_state.set("last_page", page)
```
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
Review Comment:
There's no scheduler side job here. Cleanup is triggered by a CLI command (`airflow state-store cleanup-task-states`) as done in: https://github.com/apache/airflow/pull/66463.
Also worth noting: the cleanup only works for MetastoreStateBackend. Custom backends are explicitly skipped
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
+
+.. important::
+
+ **References must be deterministic.** Given the same inputs (``ti_id`` + ``key`` for task state; ``asset_ref`` + ``key`` for asset state), the serialization method must always return the same reference string. Do not embed timestamps, random UUIDs, or any other non-deterministic component in the reference path.
+
+ When a key is deleted or cleared, Airflow clears the database reference *first*, then calls the backend's ``delete()`` or ``clear()`` method. If backend cleanup fails after the DB row is gone, the external object is orphaned. Because the reference is deterministic, a subsequent ``set()`` for the same key will overwrite the orphaned object, making the failure recoverable. A non-deterministic reference would leave the external object permanently orphaned with no way to locate it.
+
+Example skeleton:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class S3StateBackend(BaseStateBackend):
+
+ def _task_ref(self, ti_id: str, key: str) -> str:
+ return f"airflow/task-state/{ti_id}/{key}"
+
+ def _asset_ref(self, asset_ref: str, key: str) -> str:
+ import hashlib
+
+ safe = hashlib.sha256(asset_ref.encode()).hexdigest()[:16]
+ return f"airflow/asset-state/{safe}/{key}"
+
+ def serialize_task_state_to_ref(self, *, value, key, ti_id):
+ s3_key = self._task_ref(ti_id, key)
+ s3_client.put_object(Bucket=BUCKET, Key=s3_key, Body=value.encode())
Review Comment:
```suggestion
s3_client.put_object(Bucket=BUCKET, Key=s3_key, Body=json.dumps(value).encode())
```
`serialize_task_state_to_ref` receives value: JsonValue, which can be a dict, int, list, etc. Calling .encode() directly on those raises AttributeError
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
Review Comment:
We should mention true signatures here. This makes is confusing.
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
Review Comment:
outlets are never added to the accessors. The outlets-only KeyError behavior is a consequence of this, not a separate known issue.
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
Review Comment:
```suggestion
``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
```
AssetAlias inlets are resolved and their concrete assets are accessible, but you must subscript with the concrete Asset object, not the alias.
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
+
+
+Garbage collection semantics
+-----------------------------
+
+The GC job runs periodically and removes state rows according to the following rules:
+
+**Time-based expiry (task state only)**
+ Rows whose ``expires_at < now()`` are deleted. ``expires_at`` is computed on the *worker* at write time, not by the server.
+
+**``default_retention_days`` fallback (task state only)**
+ Keys written with no explicit ``retention`` (i.e. ``expires_at`` is ``NULL``) are governed by the global ``default_retention_days`` setting. When this setting is positive, the GC job treats such rows as expiring ``default_retention_days`` days after their last update.
+
+**``NEVER_EXPIRE`` keys**
+ Keys set with ``retention=NEVER_EXPIRE`` are stored with ``expires_at = NULL`` and a flag that tells the GC to skip them unconditionally. They are never deleted by time-based cleanup, regardless of ``default_retention_days``.
+
+**Orphan sweep (asset state)**
+ Asset state rows for assets that no longer have an ``asset_active`` record are deleted during the orphan-sweep pass. This cleans up state for deactivated or renamed assets.
+
+
+Custom backends
+---------------
+
+To replace the default metastore backend, subclass :class:`~airflow.sdk.state.BaseStateBackend` and implement all eight abstract methods.
+
+Abstract methods
+~~~~~~~~~~~~~~~~
+
+There are four synchronous methods and four async equivalents:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 30 70
+
+ * - Method
+ - Description
+ * - ``get(scope, key, *, session)``
+ - Return the stored value, or ``None``.
+ * - ``set(scope, key, value, *, expires_at, session)``
+ - Write or overwrite a key. ``expires_at`` is a UTC datetime or ``None``
+ for non-expiring keys.
+ * - ``delete(scope, key, *, session)``
+ - Delete a single key; no-op if absent.
+ * - ``clear(scope, *, all_map_indices, session)``
+ - Delete all keys under the scope.
+ * - ``aget(scope, key, *, session)``
+ - Async variant of ``get``.
+ * - ``aset(scope, key, value, *, expires_at, session)``
+ - Async variant of ``set``.
+ * - ``adelete(scope, key, *, session)``
+ - Async variant of ``delete``.
+ * - ``aclear(scope, *, all_map_indices, session)``
+ - Async variant of ``clear``.
+
+Dispatching on scope type
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each method receives a ``scope`` argument that is either a :class:`~airflow.sdk.state.TaskScope` or an :class:`~airflow.sdk.state.AssetScope`. Use a ``match`` statement to dispatch:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class MyBackend(BaseStateBackend):
+ def get(self, scope, key, *, session=None):
+ if scope == TaskScope():
+ return self._task_store.get(scope, key)
+ elif scope == AssetScope():
+ return self._asset_store.get(scope, key)
+
+:class:`~airflow.sdk.state.AssetScope` has three optional fields: ``asset_id`` (integer, server-side only), ``name``, and ``uri``. At least one must be set. Server-side operations (REST API calls) provide ``asset_id``. Worker-side operations provide ``name`` or ``uri`` (workers do not have access to the integer ``asset_id``).
+
+Configure the class via ``[state_store] backend``:
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.MyBackend
+
+
+Custom worker-side backends
+----------------------------
+
+Worker-side backends extend ``BaseStateBackend`` with two pairs of serialization hooks. They are configured separately via ``[workers] state_backend`` and run *on the worker process*, not on the API server. This lets you store large payloads or credentialed data directly on worker infrastructure while only a compact reference string is kept in the database.
+
+Override these four methods:
+
+``serialize_task_state_to_ref(*, value, key, ti_id)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.set()`` before sending the value to the Execution API. Return a reference string (e.g. an S3 key) that will be stored in the database instead of the raw value.
+
+``deserialize_task_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``TaskStateAccessor.get()`` after retrieving the reference string from the Execution API. Return the actual value.
+
+``serialize_asset_state_to_ref(*, value, key, asset_ref)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as the task variant, but for asset state. ``asset_ref`` is the asset name or URI, depending on how the accessor was constructed.
+
+``deserialize_asset_state_from_ref(stored)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``AssetStateAccessor.get()`` to resolve the stored reference back to the actual value.
+
+.. important::
+
+ **References must be deterministic.** Given the same inputs (``ti_id`` + ``key`` for task state; ``asset_ref`` + ``key`` for asset state), the serialization method must always return the same reference string. Do not embed timestamps, random UUIDs, or any other non-deterministic component in the reference path.
+
+ When a key is deleted or cleared, Airflow clears the database reference *first*, then calls the backend's ``delete()`` or ``clear()`` method. If backend cleanup fails after the DB row is gone, the external object is orphaned. Because the reference is deterministic, a subsequent ``set()`` for the same key will overwrite the orphaned object, making the failure recoverable. A non-deterministic reference would leave the external object permanently orphaned with no way to locate it.
+
+Example skeleton:
+
+.. code-block:: python
+
+ from airflow.sdk.state import BaseStateBackend, TaskScope, AssetScope
+
+
+ class S3StateBackend(BaseStateBackend):
Review Comment:
Type hints for methods pls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4623701782
Feedback implemented - should be good to go!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367301816
##########
airflow-core/docs/core-concepts/asset-store.rst:
##########
@@ -0,0 +1,189 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-store:
+
+.. spelling:word-list::
+
+ subscripted
+ subscripting
+
+Asset Store
+===========
+
+.. versionadded:: 3.3
+
+Asset store is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task store </core-concepts/task-store>`, which is tied to a single task instance, asset store persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset store is accessed through the task context via ``context["asset_store"]``.
+
+
+When is ``asset_store`` available?
+----------------------------------
+
+When using asset store within a task, ``context["asset_store"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. A task must declare at least one concrete inlet or outlet for ``asset_store`` to contain any entries.
+
+Accessing asset store using ``context``
+---------------------------------------
+
+An asset becomes available through context["asset_store"] when it is included in inlets (or in both inlets and outlets). You can then retrieve its asset store by subscripting context["asset_store"] with the asset object.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_store", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_store = context["asset_store"][my_asset]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+To see asset store in-action in a real DAG, checkout the DAG in `example_asset_store.py <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_asset_store.py>`_.
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_store"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_store = context["asset_store"]
+ watermark = asset_store.get("watermark")
+ asset_store.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_store"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_store"][my_asset]``), the shorthand (``context["asset_store"]``) when the task has exactly one inlet, and when using the ``self.asset_store`` attribute.
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Documentation for Task and Asset Store [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3367299687
##########
airflow-core/docs/administration-and-deployment/task-and-asset-store.rst:
##########
@@ -0,0 +1,210 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _task-and-asset-store:
+
+Task and Asset Store Configuration
+====================================
+
+.. versionadded:: 3.3
+
+The task and asset store is the persistence layer for :doc:`task store </core-concepts/task-store>` and :doc:`asset store </core-concepts/asset-store>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_store]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
Review Comment:
Resolved in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3324241160
##########
airflow-core/docs/administration-and-deployment/state-api.rst:
##########
@@ -0,0 +1,142 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-api:
+
+State REST API
Review Comment:
@amoghrajesh, I think that makes sense (and is why I originally didn't include it). Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342786264
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-103: Initial framework for Task State and Asset State documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3342787441
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,270 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
Review Comment:
Addressed in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on PR #67299:
URL: https://github.com/apache/airflow/pull/67299#issuecomment-4554590846
> And a few things that I think are missing.
>
> * A rest API page
> * Maybe as a follow up task we should be mentioning the new ResumableJobMixin: [Add `ResumableJobMixin` with `SparkSubmitOperator` as a case study for surviving worker failures (standalone) #67118](https://github.com/apache/airflow/pull/67118), maybe task sdk docs
> * `example_task_state.py` and `example_asset_state.py` can probably be mentioned
I agree - where do you think that the REST API page should live?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313076271
##########
airflow-core/docs/administration-and-deployment/state-store.rst:
##########
@@ -0,0 +1,247 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _state-store:
+
+State Store Configuration
+==========================
+
+.. versionadded:: 3.3
+
+The state store is the persistence layer for :doc:`task state </core-concepts/task-state>` and :doc:`asset state </authoring-and-scheduling/asset-state>`. By default, both are stored in the Airflow metadata database. This page describes the available configuration options, garbage-collection semantics, and how to provide a custom backend.
+
+Configuration reference
+-----------------------
+
+All options live under the ``[state_store]`` section of ``airflow.cfg``.
+
+.. note::
+
+ The config section is ``[state_store]``, **not** ``[task_state]``.
+
+``backend``
+~~~~~~~~~~~
+
+Full dotted path to a class that implements :class:`~airflow.sdk.state.BaseStateBackend`. Defaults to the built-in metastore backend.
+
+.. code-block:: ini
+
+ [state_store]
+ backend = mypackage.state.CustomStateBackend
+
+``default_retention_days``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of days to retain **task state** rows after their last update. Rows older than this are deleted during the next GC pass.
+
+* Set to ``0`` to disable time-based cleanup entirely.
+* Default: ``30``.
+* This setting does **not** apply to asset state rows.
+
+.. code-block:: ini
+
+ [state_store]
+ default_retention_days = 30
+
+``clear_on_success``
+~~~~~~~~~~~~~~~~~~~~
+
+When ``True``, all task state keys for a task instance are automatically deleted when that task instance moves to the ``success`` state. Defaults to ``False``, which preserves task state after success for observability (e.g.the submitted job ID or the last row count is still readable from the UI orREST API after the run completes).
+
+.. important::
+
+ ``clear_on_success`` clears **task state only**. It has no effect on asset state. Asset state is scoped to the asset rather than the task instance and must be cleared explicitly.
+
+.. code-block:: ini
+
+ [state_store]
+ clear_on_success = False
+
+``state_cleanup_batch_size``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Number of rows deleted per batch during GC cleanup. Set to ``0`` (default) to delete all matching rows in a single statement. Tune this on deployments with large ``task_state`` tables to reduce lock contention.
+
+.. code-block:: ini
+
+ [state_store]
+ state_cleanup_batch_size = 10000
+
+Worker-side backend (``[workers] state_backend``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A separate, optional config key under ``[workers]`` lets you route task state and asset state values through a worker-side backend before they reach the API server.
+
+.. code-block:: ini
+
+ [workers]
+ state_backend = mypackage.state.S3StateBackend
+
+When this is set, ``TaskStateAccessor.set()`` calls ``serialize_task_state_to_ref()`` on the worker-side backend before sending the value to the Execution API, and ``get()`` calls ``deserialize_task_state_from_ref()`` after receiving the stored reference. See `Custom worker-side backends`_ below.
Review Comment:
Committed locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313203584
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,261 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key. Note, ``value`` must be a string.
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_state = context["task_state"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_state.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_state.set("job_id", str(job_id), retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ task_state = context["task_state"]
+
+ raw = task_state.get("last_page")
+ start_page = int(raw) + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_state.set("last_page", str(page))
+
+On a retry, the task reads ``last_page`` and skips pages that were already processed.
+
+Progress metadata
+~~~~~~~~~~~~~~~~~
+
+Task state can expose in-progress metrics for observability — row counts, status strings, or lightweight JSON payloads — without requiring XCom or an external system.
+
+.. code-block:: python
+
+ import json
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("row_ingest", schedule="@hourly"):
+
+ @task
+ def ingest_rows(**context):
+ task_state = context["task_state"]
+ total = 0
+
+ for batch in get_batches():
+ load(batch)
+ total += len(batch)
+ task_state.set(
+ "progress",
+ json.dumps({
+ "rows_loaded": total,
+ "status": "running"
+ }),
+ )
+
+ task_state.set(
+ "progress",
+ json.dumps({
+ "rows_loaded": total,
+ "status": "done"
+ }),
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313197715
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
+
+Asset state is also available using ``AssetState``, provided by the Task SDK. This approach is more commonly used when building ``BaseEventTrigger``'s for things like asset-watching.
+
+.. warning::
+
+ **Outlets-only tasks**: if a task declares only ``outlets`` (no ``inlets``), ``context["asset_state"][my_asset]`` may raise a ``KeyError`` at runtime. The workaround is to declare the asset in **both** ``inlets`` and ``outlets``.
+
+ .. code-block:: python
+ # my_asset defined above ...
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def write_asset(**context):
+ context["asset_state"][my_asset].set("watermark", "2024-01-01")
+
+ This known issue will be resolved in a future release. A workaround for this is to use the ``AssetState`` class, provided as part of the Task SDK. More details on that below!
+
+
+Accessing asset state using ``context``
+---------------------------------------
+
+An asset can be brought into "scope" (for lack of a better phrase) by including it in ``inlets`` (or both ``inlets`` and ``outlets``). Then subscript ``context["asset_state"]`` with the asset object to retrieve the asset state.
+
+.. code-block:: python
+
+ from airflow.sdk import Asset, DAG, task
+
+ my_asset = Asset("my_data", uri="s3://bucket/my_data")
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process(**context):
+ asset_state = context["asset_state"][my_asset]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Accessing asset state using ``AssetState``
+------------------------------------------
+
+Asset state can also be retrieved for an asset using the ``airflow.sdk.AssetState`` class. This approach does NOT require that an asset is passed to a task using ``inlets``.
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(name="my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+In the example above, the ``name`` of the asset is used to retrieve it's state. However, the ``uri`` can also be used:
+
+.. code-block:: python
+ from airflow.sdk import DAG, task, AssetState
+
+ with DAG("example_asset_state", schedule=None):
+
+ @task()
+ def process():
+ asset_state = AssetState(uri="s3://bucket/my_data")
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+Single-inlet shorthand
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks with exactly **one** concrete inlet, you can call ``get``, ``set``, ``delete``, and ``clear`` directly on ``context["asset_state"]`` without subscripting.
+
+.. code-block:: python
+
+ @task(inlets=[my_asset], outlets=[my_asset])
+ def process_single(**context):
+ asset_state = context["asset_state"]
+ watermark = asset_state.get("watermark")
+ asset_state.set("watermark", "2024-06-01")
+
+If the task has more than one concrete inlet, calling the shorthand raises a ``ValueError``. Use the subscript form (``context["asset_state"][my_asset]``) whenever a task has multiple inlets.
+
+
+API reference
+-------------
+
+The following methods are available on both the per-asset accessor (``context["asset_state"][my_asset]``), the shorthand (``context["asset_state"]``) when the task has exactly one inlet, and when using the ``AssetState`` Task SDK class.
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ watermark = context["asset_state"][my_asset].get("watermark")
+
+ # Using the Task SDK
+ AssetState(name="my_data").get("watermark")
+
+``set(key, value)``
+~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key-value pair. Unlike Task state, asset state has no ``retention`` parameter. Values persist until explicitly deleted or until the asset is deactivated.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].set("watermark", "2024-06-01T00:00:00Z")
+
+ # Using the Task SDK class
+ AssetState(name="my_data").set("watermark", "2024-06-01T00:00:00Z")
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].delete("watermark")
+
+ # Using the Task SDK class
+ AssetState(name="my_data").delete("watermark")
+
+``clear()``
+~~~~~~~~~~~
+
+Deletes *all* state keys for the asset.
+
+.. code-block:: python
+
+ # Using context
+ context["asset_state"][my_asset].clear()
+
+ # Using the Task SDK class
+ AssetState(name="my_data").clear()
+
+Use cases
+---------
+
+Watermark pattern
+~~~~~~~~~~~~~~~~~
+
+The canonical use case for asset state is an incremental-load task that advances a watermark on each run. The watermark is stored on the asset itself so any task that reads or writes that asset can access it. *This use case is especially applicable when building things like asset "watchers" using ``BaseEventTrigger``'s.
Review Comment:
Yes, thanks for the catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313211170
##########
airflow-core/docs/core-concepts/task-state.rst:
##########
@@ -0,0 +1,261 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:task-state:
+
+Task State
+==========
+
+.. versionadded:: 3.3
+
+Task state is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata.
+
+Task state is accessed through the task context via ``context["task_state"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
+
+
+Accessing task state
+--------------------
+
+Inside any ``@task``-decorated function or ``BaseOperator.execute()`` method, task state is available through the ``context`` dictionary via the ``task_state`` key. From there, it can be used to retrieve, set, delete, or clear task state for a specific key-value pair. In this example, the ``job_id`` is retrieved from task state, then upated.
+
+.. code-block:: python
+ from airflow.sdk import task
+ import random
+
+ @task
+ def my_task(**context):
+ # Retrieve task_state from context
+ task_state = context["task_state"]
+ my_value = task_state.get("my_key")
+
+ # Set the new value
+ new_value = f"It is {random.randint(1, 12 + 1)} o'clock"
+ task_state.set("my_key", new_value)
+
+Reference
+-------------
+
+``get(key)``
+~~~~~~~~~~~~
+
+Returns the stored string value, or ``None`` if the key does not exist.
+
+.. code-block:: python
+
+ value = task_state.get("job_id") # returns str or None
+
+``set(key, value, *, retention=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes or overwrites a key. Note, ``value`` must be a string.
+
+The optional ``retention`` argument controls when the key expires:
+
+* ``timedelta(...)``: expire after the given duration from the time of the write (e.g. ``timedelta(hours=6)``). The expiry timestamp is computed on the worker in UTC before the value is sent to the API server.
+* ``NEVER_EXPIRE``: the key never expires and is skipped by garbage collection, regardless of the global ``[state_store] default_retention_days`` setting.
+* ``None`` (default): fall back to the global ``[state_store] default_retention_days`` config.
+
+.. important::
+
+ ``retention`` accepts only a :class:`~datetime.timedelta`, not a plain integer number of days. Passing an integer raises a ``TypeError``.
+
+ .. code-block:: python
+
+ # correct
+ task_state.set("key", "val", retention=timedelta(days=7))
+
+ # wrong — raises TypeError
+ task_state.set("key", "val", retention=7)
+
+``NEVER_EXPIRE`` sentinel
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Import ``NEVER_EXPIRE`` from ``airflow.sdk.execution_time.context``:
+
+.. code-block:: python
+
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+ task_state.set("job_id", job_id, retention=NEVER_EXPIRE)
+
+``delete(key)``
+~~~~~~~~~~~~~~~
+
+Deletes a single key. No-op if the key does not exist.
+
+.. code-block:: python
+
+ task_state.delete("job_id")
+
+``clear(all_map_indices=False)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes *all* state keys for this task instance.
+
+For :doc:`mapped tasks <dynamic-task-mapping>`, the default clears only the current map index. Pass ``all_map_indices=True`` to wipe state across **every** mapped instance of the task (fleet-wide reset).
+
+.. code-block:: python
+
+ # clear only this map index
+ task_state.clear()
+
+ # clear all map indices (fleet-wide)
+ task_state.clear(all_map_indices=True)
+
+
+Use Cases
+---------
+
+External job resumption
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A common pattern for long-running external jobs: check whether a job ID is already stored before submitting, and use ``NEVER_EXPIRE`` so the key outlives
+the default retention window.
+
+.. code-block:: python
+
+ from datetime import timedelta
+
+ from airflow.sdk import DAG, task
+ from airflow.sdk.execution_time.context import NEVER_EXPIRE
+
+
+ with DAG("spark_job_dag", schedule=None):
+
+ @task
+ def run_spark_job(**context):
+ task_state = context["task_state"]
+
+ # Check for an already-submitted job from a previous attempt.
+ job_id = task_state.get("job_id")
+ if job_id is None:
+ job_id = spark_client.submit_job(...)
+ # Store with NEVER_EXPIRE so the key is not garbage-collected before the job finishes
+ task_state.set("job_id", str(job_id), retention=NEVER_EXPIRE)
+
+ # Reattach to the job and wait for completion.
+ result = spark_client.wait_for_completion(job_id)
+ return result
+
+On a retry, the task finds the stored ``job_id`` and reattaches instead of submitting a duplicate job.
+
+Intra-task checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For tasks that process paginated or batched data, store the last-completed offset so a retry can resume mid-stream rather than restarting from the beginning.
+
+.. code-block:: python
+
+ from airflow.sdk import DAG, task
+
+
+ with DAG("paginated_ingest", schedule="@daily"):
+
+ @task
+ def ingest_pages(**context):
+ task_state = context["task_state"]
+
+ raw = task_state.get("last_page")
+ start_page = int(raw) + 1 if raw is not None else 1
+
+ for page in range(start_page, total_pages + 1):
+ fetch_and_load(page)
+ task_state.set("last_page", str(page))
+
Review Comment:
Updated locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313176853
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
Review Comment:
Added, thanks for the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] `docs/issue-65782`: Initial framework for AIP-103 documentation [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #67299:
URL: https://github.com/apache/airflow/pull/67299#discussion_r3313410711
##########
airflow-core/docs/core-concepts/asset-state.rst:
##########
@@ -0,0 +1,221 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _concepts:asset-state:
+
+Asset State
+===========
+
+.. versionadded:: 3.3
+
+Asset state is a persistent key/value store scoped to an *asset*, independent of any particular DAG run. Unlike :doc:`task state </core-concepts/task-state>`, which is tied to a single task instance, asset state persists across runs and is logically owned by the asset itself. It is the natural home for cross-run metadata such as watermarks, incremental-load cursors, and per-asset configuration.
+
+Asset state is accessed through the task context via ``context["asset_state"]``, or via the ``AssetState`` Task SDK mechanism.
+
+
+When is ``asset_state`` available?
+------------------------------------
+
+``context["asset_state"]`` is populated for **concrete** :class:`~airflow.sdk.definitions.asset.Asset` inlets and outlets. It is *not* available for :class:`~airflow.sdk.definitions.asset.AssetAlias` inlets. Aliases are resolved at runtime, but asset state accessors are only created for concrete assets declared directly on the task. A task must declare at least one concrete inlet or outlet for ``asset_state`` to contain any entries.
Review Comment:
This is technically resolved, deferring to comment below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org