You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2026/06/06 12:15:53 UTC

[PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

amoghrajesh opened a new pull request, #68136:
URL: https://github.com/apache/airflow/pull/68136

    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull request.
   Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the documentation.
   
   Feel free to ping (in general) for the review if you do not see reaction for a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
   -->
   
   - [x] Yes - claude sonnet 4.6
   
   <!--
   Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   closes: https://github.com/apache/airflow/issues/67706
   
   
   ---
   
   ### What?
   
   Operators that submit work to external systems such as Spark, BigQuery, EMR, Kubernetes share a common failure mode: the worker holds its slot for the full polling duration, and if the worker crashes the task retries from scratch, often submitting a duplicate job. Airflow 3.3 now introduces `ResumableJobMixin` to solve the crash-and-duplicate problem, and the task state store enables the broader checkpoint-and-resume pattern. Neither had documentation, and there was no guidance on when to reach for these over deferrable operators or async tasks.
   
   ### Current behaviour
   
   The only comparison page (`task-sdk/docs/deferred-vs-async-operators.rst`) covered deferrable and async operators but made no mention of resumable tasks. `ResumableJobMixin` had no user-facing documentation at all. Users experiencing repeated job submission failures had no clear path to a solution.
   
   ### Proposed change
   
   - Adds `airflow-core/docs/core-concepts/resumable-tasks.rst` - a decision guide covering all three patterns (deferrable, resumable, async) with trade-off descriptions, a three-way comparison table, and a general checkpoint example using `task_store` directly.
   - Adds `task-sdk/docs/resumable-job-mixin.rst`- a reference page for `ResumableJobMixin` covering the interface, method descriptions with inline examples, the retry flow, the pre-submit crash window limitation, and the `external_id_key` renaming warning.
   - Updates `task-sdk/docs/deferred-vs-async-operators.rst` with a `versionchanged:: 3.3.0` note pointing to both new pages.
   
   
   ---
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #68136:
URL: https://github.com/apache/airflow/pull/68136#issuecomment-4647237643

   Thanks for review folks, merging it now. We can revisit and handle changes if any in follow ups.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3367364175


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a separate ``Trigger`` class in

Review Comment:
   ```suggestion
   * Writing a custom deferrable operator requires implementing a ``Trigger`` class in
   ```



##########
task-sdk/docs/resumable-job-mixin.rst:
##########
@@ -0,0 +1,167 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _sdk-resumable-job-mixin:
+
+ResumableJobMixin
+=================
+
+.. versionadded:: 3.3.0
+
+:class:`~airflow.sdk.ResumableJobMixin` is a mixin for operators that submit long-running jobs
+to an external system and poll for its completion. It makes the operator crash-safe by persisting
+the external job identifier to the task state store before polling begins. If the worker is restarted

Review Comment:
   ```suggestion
   the external job identifier to task state store before polling begins. If the worker is restarted
   ```



##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a separate ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, just plain old context. The state store is just

Review Comment:
   ```suggestion
   This pattern works without any additional work, relying only on ``context``. The state store is just
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #68136:
URL: https://github.com/apache/airflow/pull/68136


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3370892737


##########
airflow-core/docs/authoring-and-scheduling/deferring.rst:
##########
@@ -34,6 +34,9 @@ This is where *Deferrable Operators* can be used. When it has nothing to do but
 
    For guidance on when to use deferred operators versus async tasks,
    see `Deferred vs Async Operators <https://airflow.apache.org/docs/task-sdk/stable/deferred-vs-async-operators.html>`__.
+   For guidance on when to use deferrable operators versus resumable tasks

Review Comment:
   Would it be better to add another `..note` section  or subheader for the `concepts-resumable-tasks`?
   
   In case users thought the `Resumable Tasks` feature is introduced in Airflow 3.2 as well. Since the `Resumable Tasks` is under same section as `Deferred vs Async Operators`.



##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a dedicated ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, relying only on ``context``. The state store is just
+a key-value store scoped to the task instance, and what you checkpoint is up to you.
+
+**Resumable operators for external jobs**
+
+When the task submits a job to an external system and then polls for completion, there is an
+additional problem: on retry, the task would submit a second job even though the first one may
+still be running. Instead of having to handle this manually, the :class:`~airflow.sdk.ResumableJobMixin`
+solves this by persisting the external job identifier before polling starts, and reconnecting to the
+existing job on retry instead of submitting a new one.
+
+For more details and a working example, see :class:`~airflow.sdk.ResumableJobMixin`.
+
+.. _concepts-resumable-tasks-async:
+
+Asynchronous Tasks
+------------------
+
+.. note::
+
+   Async task support applies to Python tasks only: ``@task`` decorated ``async def`` functions
+   and class-based operators that subclass ``BaseAsyncOperator``. It is not available for
+   other operator types.
+
+Python tasks support ``async``/``await`` syntax. When the decorated callable is an async
+function, Airflow runs it inside an event loop, which lets you fan out many concurrent I/O
+operations (HTTP requests, database queries, file reads) within a single task execution without
+blocking the event loop while waiting for each one.
+
+The worker slot is held for the full duration of the task. Async tasks are not designed for
+long external waits or crash recovery; they are designed for high-throughput I/O work that
+completes within a single execution.

Review Comment:
   Out of curious, non-blocking, not sure if this is a valid point.
   
   From user point of view, the resumable feature and async task feature doesn't conflict, they could even combine together to support stateful async-native processing.
   
   Would it be a valid point to support `await task_store.aget`, `await task_store.aset`  by wiring up the async method of `BaseStoreBackend` on `TaskStoreAccessor`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3371161513


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a dedicated ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, relying only on ``context``. The state store is just
+a key-value store scoped to the task instance, and what you checkpoint is up to you.
+
+**Resumable operators for external jobs**
+
+When the task submits a job to an external system and then polls for completion, there is an
+additional problem: on retry, the task would submit a second job even though the first one may
+still be running. Instead of having to handle this manually, the :class:`~airflow.sdk.ResumableJobMixin`
+solves this by persisting the external job identifier before polling starts, and reconnecting to the
+existing job on retry instead of submitting a new one.
+
+For more details and a working example, see :class:`~airflow.sdk.ResumableJobMixin`.
+
+.. _concepts-resumable-tasks-async:
+
+Asynchronous Tasks
+------------------
+
+.. note::
+
+   Async task support applies to Python tasks only: ``@task`` decorated ``async def`` functions
+   and class-based operators that subclass ``BaseAsyncOperator``. It is not available for
+   other operator types.
+
+Python tasks support ``async``/``await`` syntax. When the decorated callable is an async
+function, Airflow runs it inside an event loop, which lets you fan out many concurrent I/O
+operations (HTTP requests, database queries, file reads) within a single task execution without
+blocking the event loop while waiting for each one.
+
+The worker slot is held for the full duration of the task. Async tasks are not designed for
+long external waits or crash recovery; they are designed for high-throughput I/O work that
+completes within a single execution.

Review Comment:
   No problem and not blocking at all, or just having an issue to keep track of would be sufficient for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3371144428


##########
airflow-core/docs/authoring-and-scheduling/deferring.rst:
##########
@@ -34,6 +34,9 @@ This is where *Deferrable Operators* can be used. When it has nothing to do but
 
    For guidance on when to use deferred operators versus async tasks,
    see `Deferred vs Async Operators <https://airflow.apache.org/docs/task-sdk/stable/deferred-vs-async-operators.html>`__.
+   For guidance on when to use deferrable operators versus resumable tasks

Review Comment:
   We already have a `versionchanged:: 3.3.0` note at the top of this page that calls out the resumable tasks addition and links to the concept page. Does that cover your concern, or would you want something more prominent?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3368795968


##########
task-sdk/docs/deferred-vs-async-operators.rst:
##########
@@ -22,6 +22,15 @@ Deferred vs Async Operators
 
 .. versionadded:: 3.2.0
 
+.. versionchanged:: 3.3.0

Review Comment:
   Thanks, fixed in dd99782989



##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a separate ``Trigger`` class in

Review Comment:
   Handled in dd99782989
   



##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a separate ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, just plain old context. The state store is just

Review Comment:
   Handled in dd99782989



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3371031012


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a dedicated ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, relying only on ``context``. The state store is just
+a key-value store scoped to the task instance, and what you checkpoint is up to you.
+
+**Resumable operators for external jobs**
+
+When the task submits a job to an external system and then polls for completion, there is an
+additional problem: on retry, the task would submit a second job even though the first one may
+still be running. Instead of having to handle this manually, the :class:`~airflow.sdk.ResumableJobMixin`
+solves this by persisting the external job identifier before polling starts, and reconnecting to the
+existing job on retry instead of submitting a new one.
+
+For more details and a working example, see :class:`~airflow.sdk.ResumableJobMixin`.
+
+.. _concepts-resumable-tasks-async:
+
+Asynchronous Tasks
+------------------
+
+.. note::
+
+   Async task support applies to Python tasks only: ``@task`` decorated ``async def`` functions
+   and class-based operators that subclass ``BaseAsyncOperator``. It is not available for
+   other operator types.
+
+Python tasks support ``async``/``await`` syntax. When the decorated callable is an async
+function, Airflow runs it inside an event loop, which lets you fan out many concurrent I/O
+operations (HTTP requests, database queries, file reads) within a single task execution without
+blocking the event loop while waiting for each one.
+
+The worker slot is held for the full duration of the task. Async tasks are not designed for
+long external waits or crash recovery; they are designed for high-throughput I/O work that
+completes within a single execution.

Review Comment:
   Agree with @jason810496, I will even use that for AIP-104.  For AIP-104, I will use the async aset on task store, so I can keep track of each individual processed task iteration.  At start of task iteration, I would use the sync method to retrieve list of already processed tasks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3371157858


##########
airflow-core/docs/authoring-and-scheduling/deferring.rst:
##########
@@ -34,6 +34,9 @@ This is where *Deferrable Operators* can be used. When it has nothing to do but
 
    For guidance on when to use deferred operators versus async tasks,
    see `Deferred vs Async Operators <https://airflow.apache.org/docs/task-sdk/stable/deferred-vs-async-operators.html>`__.
+   For guidance on when to use deferrable operators versus resumable tasks

Review Comment:
   > We already have a versionchanged:: 3.3.0 note at the top of this page that calls out the resumable tasks addition and links to the concept page. 
   
   Then it should be fine, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3367431456


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a separate ``Trigger`` class in

Review Comment:
   Or: requires implementing a dedicated Trigger class...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3368796127


##########
task-sdk/docs/resumable-job-mixin.rst:
##########
@@ -0,0 +1,167 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _sdk-resumable-job-mixin:
+
+ResumableJobMixin
+=================
+
+.. versionadded:: 3.3.0
+
+:class:`~airflow.sdk.ResumableJobMixin` is a mixin for operators that submit long-running jobs
+to an external system and poll for its completion. It makes the operator crash-safe by persisting
+the external job identifier to the task state store before polling begins. If the worker is restarted

Review Comment:
   Handled in dd99782989



##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a separate ``Trigger`` class in

Review Comment:
   Handled in dd99782989



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3367375898


##########
task-sdk/docs/deferred-vs-async-operators.rst:
##########
@@ -22,6 +22,15 @@ Deferred vs Async Operators
 
 .. versionadded:: 3.2.0
 
+.. versionchanged:: 3.3.0

Review Comment:
   This breaks the current structure the
   `.. versionadded:: 3.2.0`
   Is related to the pargraph below the new additon



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3371031012


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a dedicated ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, relying only on ``context``. The state store is just
+a key-value store scoped to the task instance, and what you checkpoint is up to you.
+
+**Resumable operators for external jobs**
+
+When the task submits a job to an external system and then polls for completion, there is an
+additional problem: on retry, the task would submit a second job even though the first one may
+still be running. Instead of having to handle this manually, the :class:`~airflow.sdk.ResumableJobMixin`
+solves this by persisting the external job identifier before polling starts, and reconnecting to the
+existing job on retry instead of submitting a new one.
+
+For more details and a working example, see :class:`~airflow.sdk.ResumableJobMixin`.
+
+.. _concepts-resumable-tasks-async:
+
+Asynchronous Tasks
+------------------
+
+.. note::
+
+   Async task support applies to Python tasks only: ``@task`` decorated ``async def`` functions
+   and class-based operators that subclass ``BaseAsyncOperator``. It is not available for
+   other operator types.
+
+Python tasks support ``async``/``await`` syntax. When the decorated callable is an async
+function, Airflow runs it inside an event loop, which lets you fan out many concurrent I/O
+operations (HTTP requests, database queries, file reads) within a single task execution without
+blocking the event loop while waiting for each one.
+
+The worker slot is held for the full duration of the task. Async tasks are not designed for
+long external waits or crash recovery; they are designed for high-throughput I/O work that
+completes within a single execution.

Review Comment:
   Agree with @jason810496, i will even use that for AIP-104.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3371146385


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a dedicated ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, relying only on ``context``. The state store is just
+a key-value store scoped to the task instance, and what you checkpoint is up to you.
+
+**Resumable operators for external jobs**
+
+When the task submits a job to an external system and then polls for completion, there is an
+additional problem: on retry, the task would submit a second job even though the first one may
+still be running. Instead of having to handle this manually, the :class:`~airflow.sdk.ResumableJobMixin`
+solves this by persisting the external job identifier before polling starts, and reconnecting to the
+existing job on retry instead of submitting a new one.
+
+For more details and a working example, see :class:`~airflow.sdk.ResumableJobMixin`.
+
+.. _concepts-resumable-tasks-async:
+
+Asynchronous Tasks
+------------------
+
+.. note::
+
+   Async task support applies to Python tasks only: ``@task`` decorated ``async def`` functions
+   and class-based operators that subclass ``BaseAsyncOperator``. It is not available for
+   other operator types.
+
+Python tasks support ``async``/``await`` syntax. When the decorated callable is an async
+function, Airflow runs it inside an event loop, which lets you fan out many concurrent I/O
+operations (HTTP requests, database queries, file reads) within a single task execution without
+blocking the event loop while waiting for each one.
+
+The worker slot is held for the full duration of the task. Async tasks are not designed for
+long external waits or crash recovery; they are designed for high-throughput I/O work that
+completes within a single execution.

Review Comment:
   Yes, async-native stateful processing with a`wait task_store.aget/aset` would be a natural evolution once the async I/O path matures. Nothing blocking it architecturally. Outside the scope of this PR but worth a follow-up issue if there is enough interest. ( I dont have enough bandwidth this week, maybe I can attempt next week)
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3371148094


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a dedicated ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, relying only on ``context``. The state store is just
+a key-value store scoped to the task instance, and what you checkpoint is up to you.
+
+**Resumable operators for external jobs**
+
+When the task submits a job to an external system and then polls for completion, there is an
+additional problem: on retry, the task would submit a second job even though the first one may
+still be running. Instead of having to handle this manually, the :class:`~airflow.sdk.ResumableJobMixin`
+solves this by persisting the external job identifier before polling starts, and reconnecting to the
+existing job on retry instead of submitting a new one.
+
+For more details and a working example, see :class:`~airflow.sdk.ResumableJobMixin`.
+
+.. _concepts-resumable-tasks-async:
+
+Asynchronous Tasks
+------------------
+
+.. note::
+
+   Async task support applies to Python tasks only: ``@task`` decorated ``async def`` functions
+   and class-based operators that subclass ``BaseAsyncOperator``. It is not available for
+   other operator types.
+
+Python tasks support ``async``/``await`` syntax. When the decorated callable is an async
+function, Airflow runs it inside an event loop, which lets you fan out many concurrent I/O
+operations (HTTP requests, database queries, file reads) within a single task execution without
+blocking the event loop while waiting for each one.
+
+The worker slot is held for the full duration of the task. Async tasks are not designed for
+long external waits or crash recovery; they are designed for high-throughput I/O work that
+completes within a single execution.

Review Comment:
   > From user point of view, the resumable feature and async task feature doesn't conflict, they could even combine together to support stateful async-native processing.
   
   That is correct, but good to clarify to users as users can come from all sorts of backgrounds



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Documentation for ResumableJobMixin and resumable tasks [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #68136:
URL: https://github.com/apache/airflow/pull/68136#discussion_r3372050501


##########
airflow-core/docs/core-concepts/resumable-tasks.rst:
##########
@@ -0,0 +1,187 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _concepts-resumable-tasks:
+
+Resumable Tasks
+===============
+
+.. versionadded:: 3.3.0
+
+Many data engineering workflows involve submitting work to an external system and waiting for it
+to complete. A Spark job, a BigQuery query, a Kubernetes batch pod, an EMR step: these are all
+tasks where the real work happens outside Airflow, and the operator's job is mostly to submit,
+poll, and collect the result.
+
+These tasks share a common failure mode. In classic operator cases, the worker slot is held for the
+entire polling duration, and if the worker process is restarted or the host is preempted, the task
+retries from scratch, losing all the progress made. Depending on the operator, that means the external
+job is submitted again, creating a duplicate run in context of the external system.
+
+Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs
+between them helps you choose the right one for your situation.
+
+.. _concepts-resumable-tasks-deferrable:
+
+Deferrable Operators
+--------------------
+
+A deferrable operator pauses itself at the point where it would otherwise start polling, hands
+the polling work to the Triggerer component, and releases its worker slot. When the external
+condition is met, the Triggerer wakes the task and the worker resumes from where the operator
+left off.
+
+This is the most resource-efficient option. A single Triggerer process can concurrently watch
+thousands of conditions, so the rest of the worker pool stays free for other tasks.
+
+The trade-offs are:
+
+* A Triggerer component must be running. Deployments that do not include a Triggerer cannot use this pattern.
+* Writing a custom deferrable operator requires implementing a dedicated ``Trigger`` class in
+  addition to the operator itself.
+* The polling logic runs inside the Triggerer's async event loop. Blocking calls inside a
+  Trigger stall the entire Triggerer process.
+
+If a deferrable operator already exists for your use case, or your team is comfortable
+implementing one, this is the recommended path considering its resource efficiency.
+
+For more details, see :doc:`../authoring-and-scheduling/deferring`.
+
+.. _concepts-resumable-tasks-resumable:
+
+Resumable Tasks
+---------------
+
+A resumable task uses the task state store to persist a checkpoint before
+it would otherwise lose progress. On retry, the task reads that checkpoint and continues from
+where it left off rather than starting over.
+
+The worker slot is held for the full duration of the task, the same as a standard synchronous
+operator. The benefit is crash safety and continuity, not resource efficiency.
+
+Resumable tasks are useful when:
+
+* No deferrable operator exists for your external system and writing one is not practical.
+* You want crash recovery without operating a Triggerer.
+* The task processes work incrementally (for example, reading files from a list or paginating
+  through API results) and should be able to resume from its last completed batch.
+
+**General pattern**
+
+The task reads a checkpoint from ``task_store`` at the start, does some work, writes an updated
+checkpoint, and either continues or finishes. On the next run (whether due to a retry after a
+crash or a deliberate reschedule), it reads the checkpoint again and picks up from there.
+
+.. code-block:: python
+
+    from airflow.sdk import dag, task
+
+
+    @dag(schedule=None)
+    def process_files_dag():
+
+        @task(retries=5)
+        def process_files(context=None):
+            task_store = context["task_store"]
+            files = ["a.csv", "b.csv", "c.csv", "d.csv"]
+
+            last_processed = task_store.get("last_processed")
+            start_index = 0
+            if last_processed is not None:
+                start_index = files.index(last_processed) + 1
+
+            for file in files[start_index:]:
+                # ... process the file ...
+                task_store.set("last_processed", file)
+
+        process_files()
+
+
+    process_files_dag()
+
+This pattern works without any additional work, relying only on ``context``. The state store is just
+a key-value store scoped to the task instance, and what you checkpoint is up to you.
+
+**Resumable operators for external jobs**
+
+When the task submits a job to an external system and then polls for completion, there is an
+additional problem: on retry, the task would submit a second job even though the first one may
+still be running. Instead of having to handle this manually, the :class:`~airflow.sdk.ResumableJobMixin`
+solves this by persisting the external job identifier before polling starts, and reconnecting to the
+existing job on retry instead of submitting a new one.
+
+For more details and a working example, see :class:`~airflow.sdk.ResumableJobMixin`.
+
+.. _concepts-resumable-tasks-async:
+
+Asynchronous Tasks
+------------------
+
+.. note::
+
+   Async task support applies to Python tasks only: ``@task`` decorated ``async def`` functions
+   and class-based operators that subclass ``BaseAsyncOperator``. It is not available for
+   other operator types.
+
+Python tasks support ``async``/``await`` syntax. When the decorated callable is an async
+function, Airflow runs it inside an event loop, which lets you fan out many concurrent I/O
+operations (HTTP requests, database queries, file reads) within a single task execution without
+blocking the event loop while waiting for each one.
+
+The worker slot is held for the full duration of the task. Async tasks are not designed for
+long external waits or crash recovery; they are designed for high-throughput I/O work that
+completes within a single execution.

Review Comment:
   Here you go: https://github.com/apache/airflow/issues/68214



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org