You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/04 17:51:39 UTC
[GitHub] [airflow] xinbinhuang commented on a change in pull request #12803: Add sensors section to describe different modes of sensors
xinbinhuang commented on a change in pull request #12803:
URL: https://github.com/apache/airflow/pull/12803#discussion_r536273111
##########
File path: docs/apache-airflow/concepts.rst
##########
@@ -448,7 +447,81 @@ Operators are only loaded by Airflow if they are assigned to a DAG.
.. seealso::
- :ref:`List Airflow operators <pythonapi:operators>`
- - :doc:`How-to guides for some Airflow operators<howto/operator/index>`.
+ - :doc:`How-to guides for some Airflow operators<howto/operator/index>`
+
+.. _concepts:sensors:
+
+Sensors
+-------
+
+``Sensor`` is an Operator that waits (polls) for a certain time, file, database row, S3 key, , another DAG/task, etc...
+
+There are currently 3 different modes for how a sensor operates:
+
++--------------------+-----------------------+-----------------------+
+| Schedule Mode | Description | Use case |
++====================+=======================+=======================+
+| ``poke`` (default) | The sensor is taking | Use this mode if the |
+| | up a worker slot for | expected runtime of |
+| | its whole execution | the sensor is short |
+| | time and sleeps | or if a short poke |
+| | between pokes. | interval is required. |
+| | | Note that the sensor |
+| | | will hold onto a |
+| | | worker slot and a |
+| | | pool slot for the |
+| | | duration of the |
+| | | sensor's runtime in |
+| | | this mode. |
++--------------------+-----------------------+-----------------------+
+| ``reschedule`` | The sensor task frees | Use this mode if the |
+| | the worker slot when | time before the |
+| | the criteria is not | criteria is met is |
+| | yet met and it's | expected to be quite |
+| | rescheduled at a | long. The poke |
+| | later time. | interval should be |
+| | | more than one minute |
+| | | to prevent too much |
+| | | load on the |
+| | | scheduler. |
++--------------------+-----------------------+-----------------------+
+| ``smart sensor`` | smart sensor is a | Use this mode if you |
+| | service (run by a | have a large amount |
+| | builtin DAG) which | of sensor tasks |
+| | consolidate the | running in your |
+| | execution of sensors | airflow cluster. This |
+| | in batches. Instead | can largely reduce |
+| | of holding a long | airflow’s |
+| | running process for | infrastructure cost |
+| | each sensor and | and improve cluster |
+| | poking periodically, | stability - reduce |
+| | a sensor will only | meta database load. |
+| | store poke context at | |
+| | ``sensor_instance`` | |
+| | table and then exits | |
+| | with a 'sensing' | |
+| | state. | |
++--------------------+-----------------------+-----------------------+
Review comment:
For this table, it's shown with a scrollbar horizontally and it's a bit long. Is there a way to wrap the text inside the cell instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org