[airflow] branch main updated: Doc: Add Callbacks Section to Logging & Monitoring (#18842)

[ka...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new eae7b2e  Doc: Add Callbacks Section to Logging & Monitoring (#18842)
eae7b2e is described below

commit eae7b2ebd9df1029a9975af817555b29ef5dc627
Author: john-jac <75...@users.noreply.github.com>
AuthorDate: Mon Oct 11 12:16:37 2021 -0700

    Doc: Add Callbacks Section to Logging & Monitoring (#18842)
    
    Airflow users often ask for an easy way to monitor the success or failure of tasks and/or DAGs. The use of callbacks, such as on_success_callback, helps users monitor DAG runs in an easily configurable way.
    
    This document provides an overview of available callbacks, linked to the relevant section in the Tasks document. It also provides a simple example of calling a function on any task failure in the DAG, as well as a separate function that is called upon the success of the last task in the DAG.
---
 .../logging-monitoring/callbacks.rst               | 81 ++++++++++++++++++++++
 docs/apache-airflow/logging-monitoring/index.rst   |  1 +
 2 files changed, 82 insertions(+)

diff --git a/docs/apache-airflow/logging-monitoring/callbacks.rst b/docs/apache-airflow/logging-monitoring/callbacks.rst
new file mode 100644
index 0000000..77ac594
--- /dev/null
+++ b/docs/apache-airflow/logging-monitoring/callbacks.rst
@@ -0,0 +1,81 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+
+
+Callbacks
+=========
+
+A valuable component of logging and monitoring is the use of task callbacks to act upon changes in state of a given task, or across all tasks in a given DAG.
+For example, you may wish to alert when certain tasks have failed, or have the last task in your DAG invoke a callback when it succeeds.
+
+.. note::
+
+    Callback functions are only invoked when the task state changes due to execution by a worker.
+    As such, task changes set by the command line interface (:doc:`CLI <../usage-cli>`) or user interface (:doc:`UI <../ui>`) do not
+    execute callback functions.
+
+Callback Types
+--------------
+
+There are four types of task events that can trigger a callback:
+
+=========================================== ================================================================
+Name                                        Description
+=========================================== ================================================================
+``on_success_callback``                     Invoked when the task :ref:`succeeds <concepts:task-instances>`
+``on_failure_callback``                     Invoked when the task :ref:`fails <concepts:task-instances>`
+``sla_miss_callback``                       Invoked when a task misses its defined :ref:`SLA <concepts:slas>`
+``on_retry_callback``                       Invoked when the task is :ref:`up for retry <concepts:task-instances>`
+=========================================== ================================================================
+
+
+Example
+-------
+
+In the following example, failures in any task call the ``task_failure_alert`` function, and success in the last task calls the ``dag_success_alert`` function:
+
+.. code-block:: python
+
+    from datetime import datetime, timedelta
+    from airflow import DAG
+    from airflow.operators.dummy import DummyOperator
+
+
+    def task_failure_alert(context):
+        print(f"Task has failed, task_instance_key_str: {context['task_instance_key_str']}")
+
+
+    def dag_success_alert(context):
+        print(f"DAG has succeeded, run_id: {context['run_id']}")
+
+
+    with DAG(
+        dag_id="example_callback",
+        schedule_interval=None,
+        start_date=datetime(2021, 1, 1),
+        dagrun_timeout=timedelta(minutes=60),
+        catchup=False,
+        on_success_callback=None,
+        on_failure_callback=task_failure_alert,
+        tags=["example"],
+    ) as dag:
+
+        task1 = DummyOperator(task_id="task1")
+        task2 = DummyOperator(task_id="task2")
+        task3 = DummyOperator(task_id="task3", on_success_callback=dag_success_alert)
+        task1 >> task2 >> task3
diff --git a/docs/apache-airflow/logging-monitoring/index.rst b/docs/apache-airflow/logging-monitoring/index.rst
index 47c0d5f..58edec4 100644
--- a/docs/apache-airflow/logging-monitoring/index.rst
+++ b/docs/apache-airflow/logging-monitoring/index.rst
@@ -31,6 +31,7 @@ In addition to the standard logging and metrics capabilities, Airflow supports t
     logging-architecture
     logging-tasks
     metrics
+    callbacks
 
     check-health
     errors