You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2025/04/23 18:43:12 UTC
(airflow) branch v3-0-test updated: Update standard provider doc operators in core operators-and-hooks-ref.rst (#49401)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-0-test by this push:
new 3ee386f272a Update standard provider doc operators in core operators-and-hooks-ref.rst (#49401)
3ee386f272a is described below
commit 3ee386f272a6cd6ea97382dce6d174b0ef28887c
Author: GPK <go...@gmail.com>
AuthorDate: Wed Apr 23 19:30:19 2025 +0100
Update standard provider doc operators in core operators-and-hooks-ref.rst (#49401)
* move standard provider doc references from core area
* Remove standard provider reference doc link in core
* Update operators-and-hooks-ref.rst with standard provider docs
* Add rst file ref to how-to-guide section
(cherry picked from commit f73f895c28bc773efb614ef80f9b95b3b9c62838)
---
airflow-core/docs/operators-and-hooks-ref.rst | 38 ++++++++++---
.../airflow/example_dags/example_latest_only.py | 2 +
.../example_dags/example_trigger_controller_dag.py | 4 ++
providers/common/sql/docs/operators.rst | 14 +++++
.../system/common/sql/example_generic_transfer.py | 62 ++++++++++++++++++++++
providers/standard/docs/operators/latest_only.rst | 31 +++++++++++
.../standard/docs/operators/trigger_dag_run.rst | 31 +++++++++++
providers/standard/provider.yaml | 7 +++
.../providers/standard/get_provider_info.py | 7 +++
9 files changed, 188 insertions(+), 8 deletions(-)
diff --git a/airflow-core/docs/operators-and-hooks-ref.rst b/airflow-core/docs/operators-and-hooks-ref.rst
index 511775d117e..5036ace9137 100644
--- a/airflow-core/docs/operators-and-hooks-ref.rst
+++ b/airflow-core/docs/operators-and-hooks-ref.rst
@@ -1,3 +1,4 @@
+
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -50,20 +51,26 @@ For details see: :doc:`apache-airflow-providers:operators-and-hooks-ref/index`.
* - Operators
- Guides
- * - :mod:`airflow.operators.branch`
- -
+ * - :mod:`airflow.provider.standard.operators.bash`
+ - :doc:`How to use <apache-airflow-providers-standard:operators/bash>`
+
+ * - :mod:`airflow.provider.standard.operators.python`
+ - :doc:`How to use <apache-airflow-providers-standard:operators/python>`
+
+ * - :mod:`airflow.provider.standard.operators.datetime`
+ - :doc:`How to use <apache-airflow-providers-standard:operators/datetime>`
* - :mod:`airflow.providers.standard.operators.empty`
-
- * - :mod:`airflow.operators.generic_transfer`
- -
+ * - :mod:`airflow.providers.common.sql.operators.generic_transfer.GenericTransfer`
+ - :doc:`How to use <apache-airflow-providers-common-sql:operators>`
* - :mod:`airflow.providers.standard.operators.latest_only`
- -
+ - :doc:`How to use <apache-airflow-providers-standard:operators/latest_only>`
* - :mod:`airflow.providers.standard.operators.trigger_dagrun`
- -
+ - :doc:`How to use <apache-airflow-providers-standard:operators/trigger_dag_run>`
**Sensors:**
@@ -73,8 +80,23 @@ For details see: :doc:`apache-airflow-providers:operators-and-hooks-ref/index`.
* - Sensors
- Guides
- * - :mod:`airflow.sensors.base`
- -
+ * - :mod:`airflow.providers.standard.sensors.bash`
+ - :doc:`How to use <apache-airflow-providers-standard:sensors/bash>`
+
+ * - :mod:`airflow.providers.standard.sensors.python`
+ - :doc:`How to use <apache-airflow-providers-standard:sensors/python>`
+
+ * - :mod:`airflow.providers.standard.sensors.filesystem`
+ - :doc:`How to use <apache-airflow-providers-standard:sensors/file>`
+
+ * - :mod:`airflow.providers.standard.sensors.date_time`
+ - :doc:`How to use <apache-airflow-providers-standard:sensors/datetime>`
+
+ * - :mod:`airflow.providers.standard.sensors.external_task`
+ - :doc:`How to use <apache-airflow-providers-standard:sensors/external_task_sensor>`
+
+
+
**Hooks:**
diff --git a/airflow-core/src/airflow/example_dags/example_latest_only.py b/airflow-core/src/airflow/example_dags/example_latest_only.py
index 8881d7d1f80..745100def2a 100644
--- a/airflow-core/src/airflow/example_dags/example_latest_only.py
+++ b/airflow-core/src/airflow/example_dags/example_latest_only.py
@@ -32,7 +32,9 @@ with DAG(
catchup=False,
tags=["example2", "example3"],
) as dag:
+ # [START howto_operator_latest_only]
latest_only = LatestOnlyOperator(task_id="latest_only")
+ # [END howto_operator_latest_only]
task1 = EmptyOperator(task_id="task1")
latest_only >> task1
diff --git a/airflow-core/src/airflow/example_dags/example_trigger_controller_dag.py b/airflow-core/src/airflow/example_dags/example_trigger_controller_dag.py
index 6adac540957..8ca49e34b3d 100644
--- a/airflow-core/src/airflow/example_dags/example_trigger_controller_dag.py
+++ b/airflow-core/src/airflow/example_dags/example_trigger_controller_dag.py
@@ -35,8 +35,12 @@ with DAG(
schedule="@once",
tags=["example"],
) as dag:
+ # [START howto_operator_trigger_dagrun]
+
trigger = TriggerDagRunOperator(
task_id="test_trigger_dagrun",
trigger_dag_id="example_trigger_target_dag", # Ensure this equals the dag_id of the DAG to trigger
conf={"message": "Hello World"},
)
+
+ # [END howto_operator_trigger_dagrun]
diff --git a/providers/common/sql/docs/operators.rst b/providers/common/sql/docs/operators.rst
index 9674dc28d69..fa62b6fc11f 100644
--- a/providers/common/sql/docs/operators.rst
+++ b/providers/common/sql/docs/operators.rst
@@ -173,3 +173,17 @@ The below example demonstrates how to instantiate the SQLThresholdCheckOperator
:end-before: [END howto_operator_sql_threshold_check]
If the value returned by the query, is within the thresholds, the task passes. Otherwise, it fails.
+
+.. _howto/operator:GenericTransfer:
+
+Generic Transfer
+~~~~~~~~~~~~~~~~
+
+Use the :class:`~airflow.providers.common.sql.operators.generic_transfer.GenericTransfer` to transfer data between
+between two connections.
+
+.. exampleinclude:: /../tests/system/common/sql/example_generic_transfer.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_generic_transfer]
+ :end-before: [END howto_operator_generic_transfer]
diff --git a/providers/common/sql/tests/system/common/sql/example_generic_transfer.py b/providers/common/sql/tests/system/common/sql/example_generic_transfer.py
new file mode 100644
index 00000000000..672838d4f42
--- /dev/null
+++ b/providers/common/sql/tests/system/common/sql/example_generic_transfer.py
@@ -0,0 +1,62 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from airflow import DAG
+from airflow.providers.common.sql.operators.generic_transfer import GenericTransfer
+from airflow.utils import timezone
+
+connection_args = {
+ "conn_id": "airflow_db",
+ "conn_type": "Postgres",
+ "host": "postgres",
+ "schema": "postgres",
+ "login": "postgres",
+ "password": "postgres",
+ "port": 5432,
+}
+
+with DAG(
+ "example_generic_transfer",
+ description="Example DAG for GenericTransfer.",
+ default_args=connection_args,
+ start_date=timezone.datetime(2021, 1, 1),
+ schedule=None,
+ catchup=False,
+) as dag:
+ # [START howto_operator_generic_transfer]
+ sql = "SELECT * FROM connection LIMIT 10;"
+
+ generic_transfer = GenericTransfer(
+ task_id="generic_transfer",
+ preoperator=[
+ "DROP TABLE IF EXISTS test_mysql_to_mysql",
+ "CREATE TABLE IF NOT EXISTS test_mysql_to_mysql LIKE connection",
+ ],
+ source_conn_id="airflow_db",
+ destination_conn_id="airflow_db",
+ destination_table="test_mysql_to_mysql",
+ sql=sql,
+ )
+ # [END howto_operator_generic_transfer]
+
+
+from tests_common.test_utils.system_tests import get_test_run # noqa: E402
+
+# Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
+test_run = get_test_run(dag)
diff --git a/providers/standard/docs/operators/latest_only.rst b/providers/standard/docs/operators/latest_only.rst
new file mode 100644
index 00000000000..d3fc30937a2
--- /dev/null
+++ b/providers/standard/docs/operators/latest_only.rst
@@ -0,0 +1,31 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/operator:LatestOnlyOperator:
+
+LatestOnlyOperator
+=======================
+
+Use the :class:`~airflow.providers.standard.operators.latest_only.LatestOnlyOperator`.
+
+.. exampleinclude:: /../../../airflow-core/src/airflow/example_dags/example_latest_only.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_latest_only]
+ :end-before: [END howto_operator_latest_only]
diff --git a/providers/standard/docs/operators/trigger_dag_run.rst b/providers/standard/docs/operators/trigger_dag_run.rst
new file mode 100644
index 00000000000..b23e79f0ff3
--- /dev/null
+++ b/providers/standard/docs/operators/trigger_dag_run.rst
@@ -0,0 +1,31 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/operator:TriggerDagRunOperator:
+
+TriggerDagRunOperator
+=======================
+
+Use the :class:`~airflow.providers.standard.operators.trigger_dagrun.TriggerDagRunOperator` to trigger dag from another dag.
+
+.. exampleinclude:: /../../../airflow-core/src/airflow/example_dags/example_trigger_controller_dag.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_trigger_dagrun]
+ :end-before: [END howto_operator_trigger_dagrun]
diff --git a/providers/standard/provider.yaml b/providers/standard/provider.yaml
index bc3d533abcf..27f3a6905df 100644
--- a/providers/standard/provider.yaml
+++ b/providers/standard/provider.yaml
@@ -42,6 +42,13 @@ integrations:
- /docs/apache-airflow-providers-standard/operators/bash.rst
- /docs/apache-airflow-providers-standard/operators/python.rst
- /docs/apache-airflow-providers-standard/operators/datetime.rst
+ - /docs/apache-airflow-providers-standard/operators/trigger_dagrun.rst
+ - /docs/apache-airflow-providers-standard/operators/latest_only.rst
+ - /docs/apache-airflow-providers-standard/sensors/bash.rst
+ - /docs/apache-airflow-providers-standard/sensors/python.rst
+ - /docs/apache-airflow-providers-standard/sensors/datetime.rst
+ - /docs/apache-airflow-providers-standard/sensors/file.rst
+ - /docs/apache-airflow-providers-standard/sensors/external_task_sensor.rst
operators:
- integration-name: Standard
diff --git a/providers/standard/src/airflow/providers/standard/get_provider_info.py b/providers/standard/src/airflow/providers/standard/get_provider_info.py
index d477fa47112..744b0fdc420 100644
--- a/providers/standard/src/airflow/providers/standard/get_provider_info.py
+++ b/providers/standard/src/airflow/providers/standard/get_provider_info.py
@@ -35,6 +35,13 @@ def get_provider_info():
"/docs/apache-airflow-providers-standard/operators/bash.rst",
"/docs/apache-airflow-providers-standard/operators/python.rst",
"/docs/apache-airflow-providers-standard/operators/datetime.rst",
+ "/docs/apache-airflow-providers-standard/operators/trigger_dagrun.rst",
+ "/docs/apache-airflow-providers-standard/operators/latest_only.rst",
+ "/docs/apache-airflow-providers-standard/sensors/bash.rst",
+ "/docs/apache-airflow-providers-standard/sensors/python.rst",
+ "/docs/apache-airflow-providers-standard/sensors/datetime.rst",
+ "/docs/apache-airflow-providers-standard/sensors/file.rst",
+ "/docs/apache-airflow-providers-standard/sensors/external_task_sensor.rst",
],
}
],