You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/06/24 16:40:34 UTC
[airflow] branch main updated: added difference between Deferrable and Non-Deferrable Operators (#31840)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 0db0ff14da added difference between Deferrable and Non-Deferrable Operators (#31840)
0db0ff14da is described below
commit 0db0ff14da449dc3dbfe9577ccdb12db946b9647
Author: Rohan Anand <96...@users.noreply.github.com>
AuthorDate: Sat Jun 24 22:10:26 2023 +0530
added difference between Deferrable and Non-Deferrable Operators (#31840)
---------
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
---
.../authoring-and-scheduling/deferring.rst | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/docs/apache-airflow/authoring-and-scheduling/deferring.rst b/docs/apache-airflow/authoring-and-scheduling/deferring.rst
index a20e3dafe6..4755566670 100644
--- a/docs/apache-airflow/authoring-and-scheduling/deferring.rst
+++ b/docs/apache-airflow/authoring-and-scheduling/deferring.rst
@@ -166,3 +166,25 @@ Airflow tries to only run triggers in one place at once, and maintains a heartbe
This means it's possible, but unlikely, for triggers to run in multiple places at once; this is designed into the Trigger contract, however, and entirely expected. Airflow will de-duplicate events fired when a trigger is running in multiple places simultaneously, so this process should be transparent to your Operators.
Note that every extra ``triggerer`` you run will result in an extra persistent connection to your database.
+
+
+Difference between Mode='reschedule' and Deferrable=True in Sensors
+-------------------------------------------------------------------
+
+In Airflow, Sensors wait for specific conditions to be met before proceeding with downstream tasks. Sensors have two options for managing idle periods: mode='reschedule' and deferrable=True. As mode='reschedule' is a parameter specific to the BaseSensorOperator in Airflow, which allows the sensor to reschedule itself if the condition is not met, whereas, 'deferrable=True' is a convention used by some operators to indicate that the task can be retried (or deferred) later, but it is not a [...]
+
++--------------------------------------------------------+--------------------------------------------------------+
+| mode='reschedule' | deferrable=True |
++========================================================+========================================================+
+| Continuously reschedules itself until condition is met | Pauses execution when idle, resumes when condition |
+| | changes |
++--------------------------------------------------------+--------------------------------------------------------+
+| Resource Usage is Higher (repeated execution) | Resource Usage is Lower (pauses when idle, frees |
+| | up worker slots) |
++--------------------------------------------------------+--------------------------------------------------------+
+| Conditions expected to change over time | Waiting for external events or resources |
+| (e.g. file creation) | (e.g. API response) |
++--------------------------------------------------------+--------------------------------------------------------+
+| Built-in functionality for rescheduling | Requires custom logic to defer task and handle |
+| | external changes |
++--------------------------------------------------------+--------------------------------------------------------+