[airflow] 07/37: Add example for list-import-errors as a CI check (#32811)

[ep...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ephraimanierobi pushed a commit to branch v2-7-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 0ed56c84ef2001eb6760a65e500a2daeac370f42
Author: Francesco Macagno <fr...@foursquare.com>
AuthorDate: Mon Aug 7 02:44:30 2023 -0500

    Add example for list-import-errors as a CI check (#32811)
    
    (cherry picked from commit 1b2b666ae358f6f8839610a1026f709b1c951f23)
---
 docs/apache-airflow/howto/usage-cli.rst | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/docs/apache-airflow/howto/usage-cli.rst b/docs/apache-airflow/howto/usage-cli.rst
index db8c9faa97..abf17ea7d1 100644
--- a/docs/apache-airflow/howto/usage-cli.rst
+++ b/docs/apache-airflow/howto/usage-cli.rst
@@ -374,3 +374,34 @@ JSON example output:
 
     airflow_db={"conn_type": "mysql", "login": "root", "password": "plainpassword", "host": "mysql", "schema": "airflow"}
     druid_broker_default={"conn_type": "druid", "host": "druid-broker", "port": 8082, "extra": "{\"endpoint\": \"druid/v2/sql\"}"}
+
+Testing for DAG Import Errors
+-----------------------------
+The CLI can be used to check whether any discovered DAGs have import errors via the ``list-import-errors`` subcommand. It is possible to create an automation step which fails if any DAGs cannot be imported by checking the command output, particularly when used with ``--output`` to generate a standard file format.
+For example, the default output when there are no errors is ``No data found``, and the json output is ``[]``. The check can then be run in CI or pre-commit to speed up the review process and testing.
+
+Example command that fails if there are any errors, using `jq <https://jqlang.github.io/jq/>`__ to parse the output:
+
+.. code-block:: bash
+
+    airflow dags list-import-errors --output=json | jq -e 'select(type=="array" and length == 0)'
+
+The line can be added to automation as-is, or if you want to print the output you can use ``tee``:
+
+.. code-block:: bash
+
+    airflow dags list-import-errors | tee import_errors.txt && jq -e 'select(type=="array" and length == 0)' import_errors.txt
+
+Example in a Jenkins pipeline:
+
+.. code-block:: groovy
+
+    stage('All DAGs are loadable') {
+        steps {
+            sh 'airflow dags list-import-errors | tee import_errors.txt && jq -e \'select(type=="array" and length == 0)\' import_errors.txt'
+        }
+    }
+
+.. note::
+
+  For this to work accurately, you must ensure Airflow does not log any additional text to stdout. For example, you may need to fix any deprecation warnings, add ``2>/dev/null`` to your command, or set ``lazy_load_plugins = True`` in the Airflow config if you have a plugin that generates logs when loaded.