You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/14 18:05:08 UTC
[GitHub] [airflow] jedcunningham commented on a diff in pull request #26371: Sync v2-4-stable with v2-4-test to release 2.4.0
jedcunningham commented on code in PR #26371:
URL: https://github.com/apache/airflow/pull/26371#discussion_r971140772
##########
RELEASE_NOTES.rst:
##########
@@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08)
Significant Changes
^^^^^^^^^^^^^^^^^^^
-- The DB related classes: ``DBApiHook``, ``SQLSensor`` have been moved to ``apache-airflow-providers-common-sql`` provider. (NEW)
-- DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
+Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example") as dag:
- ...
+ with DAG(dag_id="example") as dag:
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag2 = dag_maker()
+ dag2 = dag_maker()
- can become
+can become
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example"):
- ...
+ with DAG(dag_id="example"):
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag_maker()
+ dag_maker()
- If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
+If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
- .. code-block:: python
+.. code-block:: python
- # This dag will not be picked up by Airflow as it's not assigned to a variable
- with DAG(dag_id="example", auto_register=False):
- ...
+ # This dag will not be picked up by Airflow as it's not assigned to a variable
+ with DAG(dag_id="example", auto_register=False):
+ ...
-- DAG runs sorting logic changed in grid view (#25410)
+Deprecation of ``schedule_interval`` and ``timetable`` arguments (#25410)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- The ordering of DAG runs in the grid view has been changed to be more "natural".
- The new logic generally orders by data interval, but a custom ordering can be
- applied by setting the DAG to use a custom timetable. (#25090)
-- Deprecation of ``schedule_interval`` and ``timetable`` arguments
+We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
- We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
+If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
- If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule_interval='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule_interval='@daily',
- ):
- ...
+Going forward, you should use the ``schedule`` argument instead:
- Going forward, you should use the ``schedule`` argument instead:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule='@daily',
- ):
- ...
+The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
- The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Now you should use the ``schedule`` argument:
- Now you should use the ``schedule`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Removal of experimental Smart Sensors (#25507)
+""""""""""""""""""""""""""""""""""""""""""""""
-- Removal of experimental Smart Sensors (#25507)
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
- Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
-- The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes. (#26153, #26179, #26167)
+``airflow.contrib`` packages and deprecated modules are dynamically generated (#26153, #26179, #26167)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes.
-Features
-^^^^^^^^
+``DBApiHook`` and ``SQLSensor`` have moved (#24836)
+"""""""""""""""""""""""""""""""""""""""""""""""""""
-- DbApiHook accepts log_sql to turn off logging SQL queries. (#24570)
+``DBApiHook``, ``SQLSensor`` have been moved to the ``apache-airflow-providers-common-sql`` provider.
+DAG runs sorting logic changed in grid view (#25090)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
-Improvements
+The ordering of DAG runs in the grid view has been changed to be more "natural".
+The new logic generally orders by data interval, but a custom ordering can be
+applied by setting the DAG to use a custom timetable.
+
+
+New Features
^^^^^^^^^^^^
-- Default value for [core] hostname_callable is ``airflow.utils.net.getfqdn`` which should provide more stable canonical host name. You can still use ``socket.getfqdn``or any other ``hostname_callable`` you had configured.. (#24981)
+- Add ``@task.short_circuit`` TaskFlow decorator (#25752)
+- Make ``execution_date_or_run_id`` optional in ``tasks test`` command (#26114)
+- Automatically register DAGs that are used in a context manager (#23592)
+- Add option of sending DAG parser logs to stdout. (#25754)
+- Support multiple ``DagProcessors`` parsing files from different locations. (#25935)
+- Implement ``ExternalPythonOperator`` (#25780)
+- Make execution_date optional for command ``dags test`` (#26111)
+- Implement ``expand_kwargs()`` against a literal list (#25925)
+- Add trigger rule tooltip (#26043)
+- Add conf parameter to CLI for airflow dags test (#25900)
+- Include scheduled slots in pools view (#26006)
+- Add ``output`` property to ``MappedOperator`` (#25604)
+- Add roles delete command to cli (#25854)
+- Add Airflow specific warning classes (#25799)
+- Add support for ``TaskGroup`` in ``ExternalTaskSensor`` (#24902)
+- Add ``@task.kubernetes`` taskflow decorator (#25663)
+- Add a way to import Airflow without side-effects (#25832)
+- Let timetables control generated run_ids. (#25795)
+- Allow per-timetable ordering override in grid view (#25633)
+- Grid logs for mapped instances (#25610)
+- Consolidate to one ``schedule`` param (#25410)
+- DAG regex flag in backfill command (#23870)
+- Adding support for owner links in the Dags view UI (#25280)
+- clear specific dag run TI (#23516)
+- Possibility to document DAG with a separated markdown file (#25509)
+- Add parsing context to DAG Parsing (#25161)
+- Implement ``CronTriggerTimetable`` (#23662)
+- Add option to mask sensitive data in UI configuration page (#25346)
+- Create new databases from the ORM (#24156)
+- Implement ``XComArg.zip(*xcom_args)`` (#25176)
+- Introduce ``sla_miss`` metric (#23402)
+- Migrate files to ts (#25267)
Review Comment:
I'm not sure this should be a new feature. Maybe improvement or misc?
##########
RELEASE_NOTES.rst:
##########
@@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08)
Significant Changes
^^^^^^^^^^^^^^^^^^^
-- The DB related classes: ``DBApiHook``, ``SQLSensor`` have been moved to ``apache-airflow-providers-common-sql`` provider. (NEW)
-- DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
+Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example") as dag:
- ...
+ with DAG(dag_id="example") as dag:
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag2 = dag_maker()
+ dag2 = dag_maker()
- can become
+can become
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example"):
- ...
+ with DAG(dag_id="example"):
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag_maker()
+ dag_maker()
- If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
+If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
- .. code-block:: python
+.. code-block:: python
- # This dag will not be picked up by Airflow as it's not assigned to a variable
- with DAG(dag_id="example", auto_register=False):
- ...
+ # This dag will not be picked up by Airflow as it's not assigned to a variable
+ with DAG(dag_id="example", auto_register=False):
+ ...
-- DAG runs sorting logic changed in grid view (#25410)
+Deprecation of ``schedule_interval`` and ``timetable`` arguments (#25410)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- The ordering of DAG runs in the grid view has been changed to be more "natural".
- The new logic generally orders by data interval, but a custom ordering can be
- applied by setting the DAG to use a custom timetable. (#25090)
-- Deprecation of ``schedule_interval`` and ``timetable`` arguments
+We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
- We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
+If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
- If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule_interval='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule_interval='@daily',
- ):
- ...
+Going forward, you should use the ``schedule`` argument instead:
- Going forward, you should use the ``schedule`` argument instead:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule='@daily',
- ):
- ...
+The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
- The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Now you should use the ``schedule`` argument:
- Now you should use the ``schedule`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Removal of experimental Smart Sensors (#25507)
+""""""""""""""""""""""""""""""""""""""""""""""
-- Removal of experimental Smart Sensors (#25507)
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
- Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
-- The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes. (#26153, #26179, #26167)
+``airflow.contrib`` packages and deprecated modules are dynamically generated (#26153, #26179, #26167)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes.
-Features
-^^^^^^^^
+``DBApiHook`` and ``SQLSensor`` have moved (#24836)
+"""""""""""""""""""""""""""""""""""""""""""""""""""
-- DbApiHook accepts log_sql to turn off logging SQL queries. (#24570)
+``DBApiHook``, ``SQLSensor`` have been moved to the ``apache-airflow-providers-common-sql`` provider.
+DAG runs sorting logic changed in grid view (#25090)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
-Improvements
+The ordering of DAG runs in the grid view has been changed to be more "natural".
+The new logic generally orders by data interval, but a custom ordering can be
+applied by setting the DAG to use a custom timetable.
+
+
+New Features
^^^^^^^^^^^^
-- Default value for [core] hostname_callable is ``airflow.utils.net.getfqdn`` which should provide more stable canonical host name. You can still use ``socket.getfqdn``or any other ``hostname_callable`` you had configured.. (#24981)
+- Add ``@task.short_circuit`` TaskFlow decorator (#25752)
+- Make ``execution_date_or_run_id`` optional in ``tasks test`` command (#26114)
+- Automatically register DAGs that are used in a context manager (#23592)
+- Add option of sending DAG parser logs to stdout. (#25754)
+- Support multiple ``DagProcessors`` parsing files from different locations. (#25935)
+- Implement ``ExternalPythonOperator`` (#25780)
+- Make execution_date optional for command ``dags test`` (#26111)
+- Implement ``expand_kwargs()`` against a literal list (#25925)
+- Add trigger rule tooltip (#26043)
+- Add conf parameter to CLI for airflow dags test (#25900)
+- Include scheduled slots in pools view (#26006)
+- Add ``output`` property to ``MappedOperator`` (#25604)
+- Add roles delete command to cli (#25854)
+- Add Airflow specific warning classes (#25799)
+- Add support for ``TaskGroup`` in ``ExternalTaskSensor`` (#24902)
+- Add ``@task.kubernetes`` taskflow decorator (#25663)
+- Add a way to import Airflow without side-effects (#25832)
+- Let timetables control generated run_ids. (#25795)
+- Allow per-timetable ordering override in grid view (#25633)
+- Grid logs for mapped instances (#25610)
+- Consolidate to one ``schedule`` param (#25410)
+- DAG regex flag in backfill command (#23870)
+- Adding support for owner links in the Dags view UI (#25280)
+- clear specific dag run TI (#23516)
+- Possibility to document DAG with a separated markdown file (#25509)
Review Comment:
```suggestion
- Possibility to document DAG with a separate markdown file (#25509)
```
##########
RELEASE_NOTES.rst:
##########
@@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08)
Significant Changes
^^^^^^^^^^^^^^^^^^^
-- The DB related classes: ``DBApiHook``, ``SQLSensor`` have been moved to ``apache-airflow-providers-common-sql`` provider. (NEW)
-- DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
+Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example") as dag:
- ...
+ with DAG(dag_id="example") as dag:
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag2 = dag_maker()
+ dag2 = dag_maker()
- can become
+can become
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example"):
- ...
+ with DAG(dag_id="example"):
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag_maker()
+ dag_maker()
- If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
+If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
- .. code-block:: python
+.. code-block:: python
- # This dag will not be picked up by Airflow as it's not assigned to a variable
- with DAG(dag_id="example", auto_register=False):
- ...
+ # This dag will not be picked up by Airflow as it's not assigned to a variable
+ with DAG(dag_id="example", auto_register=False):
+ ...
-- DAG runs sorting logic changed in grid view (#25410)
+Deprecation of ``schedule_interval`` and ``timetable`` arguments (#25410)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- The ordering of DAG runs in the grid view has been changed to be more "natural".
- The new logic generally orders by data interval, but a custom ordering can be
- applied by setting the DAG to use a custom timetable. (#25090)
-- Deprecation of ``schedule_interval`` and ``timetable`` arguments
+We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
- We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
+If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
- If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule_interval='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule_interval='@daily',
- ):
- ...
+Going forward, you should use the ``schedule`` argument instead:
- Going forward, you should use the ``schedule`` argument instead:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule='@daily',
- ):
- ...
+The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
- The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Now you should use the ``schedule`` argument:
- Now you should use the ``schedule`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Removal of experimental Smart Sensors (#25507)
+""""""""""""""""""""""""""""""""""""""""""""""
-- Removal of experimental Smart Sensors (#25507)
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
- Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
-- The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes. (#26153, #26179, #26167)
+``airflow.contrib`` packages and deprecated modules are dynamically generated (#26153, #26179, #26167)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes.
-Features
-^^^^^^^^
+``DBApiHook`` and ``SQLSensor`` have moved (#24836)
+"""""""""""""""""""""""""""""""""""""""""""""""""""
-- DbApiHook accepts log_sql to turn off logging SQL queries. (#24570)
+``DBApiHook``, ``SQLSensor`` have been moved to the ``apache-airflow-providers-common-sql`` provider.
+DAG runs sorting logic changed in grid view (#25090)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
-Improvements
+The ordering of DAG runs in the grid view has been changed to be more "natural".
+The new logic generally orders by data interval, but a custom ordering can be
+applied by setting the DAG to use a custom timetable.
+
+
+New Features
^^^^^^^^^^^^
-- Default value for [core] hostname_callable is ``airflow.utils.net.getfqdn`` which should provide more stable canonical host name. You can still use ``socket.getfqdn``or any other ``hostname_callable`` you had configured.. (#24981)
+- Add ``@task.short_circuit`` TaskFlow decorator (#25752)
+- Make ``execution_date_or_run_id`` optional in ``tasks test`` command (#26114)
+- Automatically register DAGs that are used in a context manager (#23592)
+- Add option of sending DAG parser logs to stdout. (#25754)
+- Support multiple ``DagProcessors`` parsing files from different locations. (#25935)
+- Implement ``ExternalPythonOperator`` (#25780)
+- Make execution_date optional for command ``dags test`` (#26111)
+- Implement ``expand_kwargs()`` against a literal list (#25925)
+- Add trigger rule tooltip (#26043)
+- Add conf parameter to CLI for airflow dags test (#25900)
+- Include scheduled slots in pools view (#26006)
+- Add ``output`` property to ``MappedOperator`` (#25604)
+- Add roles delete command to cli (#25854)
+- Add Airflow specific warning classes (#25799)
+- Add support for ``TaskGroup`` in ``ExternalTaskSensor`` (#24902)
+- Add ``@task.kubernetes`` taskflow decorator (#25663)
+- Add a way to import Airflow without side-effects (#25832)
+- Let timetables control generated run_ids. (#25795)
+- Allow per-timetable ordering override in grid view (#25633)
+- Grid logs for mapped instances (#25610)
+- Consolidate to one ``schedule`` param (#25410)
+- DAG regex flag in backfill command (#23870)
+- Adding support for owner links in the Dags view UI (#25280)
+- clear specific dag run TI (#23516)
Review Comment:
```suggestion
- Ability to clear a specific DAG Run's task instances via REST API (#23516)
```
##########
RELEASE_NOTES.rst:
##########
@@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08)
Significant Changes
^^^^^^^^^^^^^^^^^^^
-- The DB related classes: ``DBApiHook``, ``SQLSensor`` have been moved to ``apache-airflow-providers-common-sql`` provider. (NEW)
-- DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
+Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example") as dag:
- ...
+ with DAG(dag_id="example") as dag:
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag2 = dag_maker()
+ dag2 = dag_maker()
- can become
+can become
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example"):
- ...
+ with DAG(dag_id="example"):
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag_maker()
+ dag_maker()
- If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
+If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
- .. code-block:: python
+.. code-block:: python
- # This dag will not be picked up by Airflow as it's not assigned to a variable
- with DAG(dag_id="example", auto_register=False):
- ...
+ # This dag will not be picked up by Airflow as it's not assigned to a variable
+ with DAG(dag_id="example", auto_register=False):
+ ...
-- DAG runs sorting logic changed in grid view (#25410)
+Deprecation of ``schedule_interval`` and ``timetable`` arguments (#25410)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- The ordering of DAG runs in the grid view has been changed to be more "natural".
- The new logic generally orders by data interval, but a custom ordering can be
- applied by setting the DAG to use a custom timetable. (#25090)
-- Deprecation of ``schedule_interval`` and ``timetable`` arguments
+We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
- We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
+If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
- If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule_interval='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule_interval='@daily',
- ):
- ...
+Going forward, you should use the ``schedule`` argument instead:
- Going forward, you should use the ``schedule`` argument instead:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule='@daily',
- ):
- ...
+The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
- The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Now you should use the ``schedule`` argument:
- Now you should use the ``schedule`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Removal of experimental Smart Sensors (#25507)
+""""""""""""""""""""""""""""""""""""""""""""""
-- Removal of experimental Smart Sensors (#25507)
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
- Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
-- The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes. (#26153, #26179, #26167)
+``airflow.contrib`` packages and deprecated modules are dynamically generated (#26153, #26179, #26167)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes.
-Features
-^^^^^^^^
+``DBApiHook`` and ``SQLSensor`` have moved (#24836)
+"""""""""""""""""""""""""""""""""""""""""""""""""""
-- DbApiHook accepts log_sql to turn off logging SQL queries. (#24570)
+``DBApiHook``, ``SQLSensor`` have been moved to the ``apache-airflow-providers-common-sql`` provider.
+DAG runs sorting logic changed in grid view (#25090)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
-Improvements
+The ordering of DAG runs in the grid view has been changed to be more "natural".
+The new logic generally orders by data interval, but a custom ordering can be
+applied by setting the DAG to use a custom timetable.
+
+
+New Features
^^^^^^^^^^^^
-- Default value for [core] hostname_callable is ``airflow.utils.net.getfqdn`` which should provide more stable canonical host name. You can still use ``socket.getfqdn``or any other ``hostname_callable`` you had configured.. (#24981)
+- Add ``@task.short_circuit`` TaskFlow decorator (#25752)
+- Make ``execution_date_or_run_id`` optional in ``tasks test`` command (#26114)
+- Automatically register DAGs that are used in a context manager (#23592)
+- Add option of sending DAG parser logs to stdout. (#25754)
+- Support multiple ``DagProcessors`` parsing files from different locations. (#25935)
+- Implement ``ExternalPythonOperator`` (#25780)
+- Make execution_date optional for command ``dags test`` (#26111)
+- Implement ``expand_kwargs()`` against a literal list (#25925)
+- Add trigger rule tooltip (#26043)
+- Add conf parameter to CLI for airflow dags test (#25900)
+- Include scheduled slots in pools view (#26006)
+- Add ``output`` property to ``MappedOperator`` (#25604)
+- Add roles delete command to cli (#25854)
+- Add Airflow specific warning classes (#25799)
+- Add support for ``TaskGroup`` in ``ExternalTaskSensor`` (#24902)
+- Add ``@task.kubernetes`` taskflow decorator (#25663)
+- Add a way to import Airflow without side-effects (#25832)
+- Let timetables control generated run_ids. (#25795)
+- Allow per-timetable ordering override in grid view (#25633)
+- Grid logs for mapped instances (#25610)
+- Consolidate to one ``schedule`` param (#25410)
+- DAG regex flag in backfill command (#23870)
+- Adding support for owner links in the Dags view UI (#25280)
+- clear specific dag run TI (#23516)
+- Possibility to document DAG with a separated markdown file (#25509)
+- Add parsing context to DAG Parsing (#25161)
+- Implement ``CronTriggerTimetable`` (#23662)
+- Add option to mask sensitive data in UI configuration page (#25346)
+- Create new databases from the ORM (#24156)
+- Implement ``XComArg.zip(*xcom_args)`` (#25176)
+- Introduce ``sla_miss`` metric (#23402)
+- Migrate files to ts (#25267)
+- Implement ``map()`` semantic (#25085)
+- Add override method to TaskGroupDecorator (#25160)
+- Implement ``expand_kwargs()`` (#24989)
+- Add parameter to turn off SQL query logging (#24570)
+- Add ``DagWarning`` model, and a check for missing pools (#23317)
+- Add Task Logs to Grid details panel (#24249)
+- Added small health check server and endpoint in scheduler(#23905)
+- Add built-in External Link for ``ExternalTaskMarker`` operator (#23964)
+- Add default task retry delay config (#23861)
+- Add clear DagRun endpoint. (#23451)
+- Add support for timezone as string in cron interval timetable (#23279)
+- add auto refresh to dags home page (#22900)
Review Comment:
```suggestion
- Add auto-refresh to dags home page (#22900)
```
Nit: the only non-cap one here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org