(airflow) branch main updated: Document effect of state-store cleanup for operators with durable execution (#70721)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new f5f431378f1 Document effect of state-store cleanup for operators with durable execution (#70721)
f5f431378f1 is described below

commit f5f431378f19eef122ca220361c4f009272981c1
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Thu Jul 30 15:10:38 2026 +0530

    Document effect of state-store cleanup for operators with durable execution (#70721)
---
 providers/amazon/docs/operators/redshift/redshift_data.rst | 7 +++++++
 providers/databricks/docs/operators/run_now.rst            | 6 ++++++
 providers/databricks/docs/operators/submit_run.rst         | 6 ++++++
 providers/google/docs/operators/cloud/bigquery.rst         | 6 ++++++
 providers/snowflake/docs/operators/snowflake.rst           | 7 +++++++
 5 files changed, 32 insertions(+)

diff --git a/providers/amazon/docs/operators/redshift/redshift_data.rst b/providers/amazon/docs/operators/redshift/redshift_data.rst
index e48b09ce60d..c9e64aeb662 100644
--- a/providers/amazon/docs/operators/redshift/redshift_data.rst
+++ b/providers/amazon/docs/operators/redshift/redshift_data.rst
@@ -92,6 +92,13 @@ earlier Airflow versions the flag is a no-op and the operator always submits fre
 exactly as before. If the task state store is unavailable at runtime, the operator logs that
 crash recovery is disabled and behaves the same way.
 
+Like the persisted state itself, the stored statement id isn't deleted automatically, that only
+happens when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer
+than ``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between,
+the statement id won't be there for the next retry, and the operator will submit the SQL fresh
+instead of reconnecting. Avoid running cleanup on a schedule shorter than your longest
+``retry_delay``.
+
 To opt out and always submit fresh SQL on retry, set ``durable=False``:
 
 .. code-block:: python
diff --git a/providers/databricks/docs/operators/run_now.rst b/providers/databricks/docs/operators/run_now.rst
index 78e5a1788aa..00ee6ed0fec 100644
--- a/providers/databricks/docs/operators/run_now.rst
+++ b/providers/databricks/docs/operators/run_now.rst
@@ -99,6 +99,12 @@ Airflow versions the flag is a no-op and the operator always triggers a fresh ru
 exactly as before. If the task state store is unavailable at runtime, the operator logs that crash
 recovery is disabled and behaves the same way.
 
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the
+run id won't be there for the next retry, and the operator will trigger a fresh run instead of
+reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``.
+
 To opt out and always trigger a fresh run on retry, set ``durable=False``:
 
 .. code-block:: python
diff --git a/providers/databricks/docs/operators/submit_run.rst b/providers/databricks/docs/operators/submit_run.rst
index 808f7534995..f3afd104619 100644
--- a/providers/databricks/docs/operators/submit_run.rst
+++ b/providers/databricks/docs/operators/submit_run.rst
@@ -184,6 +184,12 @@ earlier Airflow versions the flag is a no-op and the operator always submits a f
 run on retry, exactly as before. If the task state store is unavailable at runtime, the
 operator logs that crash recovery is disabled and behaves the same way.
 
+Like the persisted state itself, the stored run id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the
+run id won't be there for the next retry, and the operator will submit a fresh run instead of
+reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``.
+
 To opt out and always submit a fresh run on retry, set ``durable=False``:
 
 .. code-block:: python
diff --git a/providers/google/docs/operators/cloud/bigquery.rst b/providers/google/docs/operators/cloud/bigquery.rst
index 9f46ef351fa..6f42b4e841a 100644
--- a/providers/google/docs/operators/cloud/bigquery.rst
+++ b/providers/google/docs/operators/cloud/bigquery.rst
@@ -408,6 +408,12 @@ exactly as before -- including the pre-existing ``reattach_states``/``Conflict``
 unchanged. If the task state store is unavailable at runtime, the operator logs that crash
 recovery is disabled and behaves the same way.
 
+Like the persisted state itself, the stored job id isn't deleted automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the
+job id won't be there for the next retry, and the operator will submit a fresh job instead of
+reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``.
+
 To opt out and always submit a fresh job on retry, set ``durable=False``:
 
 .. code-block:: python
diff --git a/providers/snowflake/docs/operators/snowflake.rst b/providers/snowflake/docs/operators/snowflake.rst
index 5bb2beb8dac..6f12197cd12 100644
--- a/providers/snowflake/docs/operators/snowflake.rst
+++ b/providers/snowflake/docs/operators/snowflake.rst
@@ -182,6 +182,13 @@ earlier Airflow versions the flag is a no-op and the operator always submits fre
 exactly as before. If the task state store is unavailable at runtime, the operator logs that crash
 recovery is disabled and behaves the same way.
 
+Like the persisted state itself, the stored statement handles aren't deleted automatically, that
+only happens when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is
+longer than ``[state_store] default_retention_days`` (30 days by default) and cleanup runs in
+between, the handles won't be there for the next retry, and the operator will submit the SQL
+fresh instead of reconnecting. Avoid running cleanup on a schedule shorter than your longest
+``retry_delay``.
+
 To opt out and always submit fresh SQL on retry, set ``durable=False``:
 
 .. code-block:: python