You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/14 20:09:10 UTC
[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #26371: Sync v2-4-stable with v2-4-test to release 2.4.0
ephraimbuddy commented on code in PR #26371:
URL: https://github.com/apache/airflow/pull/26371#discussion_r971249686
##########
RELEASE_NOTES.rst:
##########
@@ -27,123 +27,354 @@ Airflow 2.4.0beta1 (2022-09-08)
Significant Changes
^^^^^^^^^^^^^^^^^^^
-- The DB related classes: ``DBApiHook``, ``SQLSensor`` have been moved to ``apache-airflow-providers-common-sql`` provider. (NEW)
-- DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
+Previously you had do assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example") as dag:
- ...
+ with DAG(dag_id="example") as dag:
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag2 = dag_maker()
+ dag2 = dag_maker()
- can become
+can become
- .. code-block:: python
+.. code-block:: python
- with DAG(dag_id="example"):
- ...
+ with DAG(dag_id="example"):
+ ...
- @dag
- def dag_maker():
- ...
+ @dag
+ def dag_maker():
+ ...
- dag_maker()
+ dag_maker()
- If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
+If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
- .. code-block:: python
+.. code-block:: python
- # This dag will not be picked up by Airflow as it's not assigned to a variable
- with DAG(dag_id="example", auto_register=False):
- ...
+ # This dag will not be picked up by Airflow as it's not assigned to a variable
+ with DAG(dag_id="example", auto_register=False):
+ ...
-- DAG runs sorting logic changed in grid view (#25410)
+Deprecation of ``schedule_interval`` and ``timetable`` arguments (#25410)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
- The ordering of DAG runs in the grid view has been changed to be more "natural".
- The new logic generally orders by data interval, but a custom ordering can be
- applied by setting the DAG to use a custom timetable. (#25090)
-- Deprecation of ``schedule_interval`` and ``timetable`` arguments
+We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
- We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
+If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
- If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule_interval='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule_interval='@daily',
- ):
- ...
+Going forward, you should use the ``schedule`` argument instead:
- Going forward, you should use the ``schedule`` argument instead:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule='@daily',
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule='@daily',
- ):
- ...
+The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
- The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Now you should use the ``schedule`` argument:
- Now you should use the ``schedule`` argument:
+.. code-block:: python
- .. code-block:: python
+ with DAG(
+ dag_id='my_example',
+ start_date=datetime(2021, 1, 1),
+ schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
+ ):
+ ...
- with DAG(
- dag_id='my_example',
- start_date=datetime(2021, 1, 1),
- schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
- ):
- ...
+Removal of experimental Smart Sensors (#25507)
+""""""""""""""""""""""""""""""""""""""""""""""
-- Removal of experimental Smart Sensors (#25507)
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
- Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
-- The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes. (#26153, #26179, #26167)
+``airflow.contrib`` packages and deprecated modules are dynamically generated (#26153, #26179, #26167)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages, have now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to non-deprecated classes.
-Features
-^^^^^^^^
+``DBApiHook`` and ``SQLSensor`` have moved (#24836)
+"""""""""""""""""""""""""""""""""""""""""""""""""""
-- DbApiHook accepts log_sql to turn off logging SQL queries. (#24570)
+``DBApiHook``, ``SQLSensor`` have been moved to the ``apache-airflow-providers-common-sql`` provider.
+DAG runs sorting logic changed in grid view (#25090)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
-Improvements
+The ordering of DAG runs in the grid view has been changed to be more "natural".
+The new logic generally orders by data interval, but a custom ordering can be
+applied by setting the DAG to use a custom timetable.
+
+
+New Features
^^^^^^^^^^^^
-- Default value for [core] hostname_callable is ``airflow.utils.net.getfqdn`` which should provide more stable canonical host name. You can still use ``socket.getfqdn``or any other ``hostname_callable`` you had configured.. (#24981)
+- Add ``@task.short_circuit`` TaskFlow decorator (#25752)
+- Make ``execution_date_or_run_id`` optional in ``tasks test`` command (#26114)
+- Automatically register DAGs that are used in a context manager (#23592)
+- Add option of sending DAG parser logs to stdout. (#25754)
+- Support multiple ``DagProcessors`` parsing files from different locations. (#25935)
+- Implement ``ExternalPythonOperator`` (#25780)
+- Make execution_date optional for command ``dags test`` (#26111)
+- Implement ``expand_kwargs()`` against a literal list (#25925)
+- Add trigger rule tooltip (#26043)
+- Add conf parameter to CLI for airflow dags test (#25900)
+- Include scheduled slots in pools view (#26006)
+- Add ``output`` property to ``MappedOperator`` (#25604)
+- Add roles delete command to cli (#25854)
+- Add Airflow specific warning classes (#25799)
+- Add support for ``TaskGroup`` in ``ExternalTaskSensor`` (#24902)
+- Add ``@task.kubernetes`` taskflow decorator (#25663)
+- Add a way to import Airflow without side-effects (#25832)
+- Let timetables control generated run_ids. (#25795)
+- Allow per-timetable ordering override in grid view (#25633)
+- Grid logs for mapped instances (#25610)
+- Consolidate to one ``schedule`` param (#25410)
+- DAG regex flag in backfill command (#23870)
+- Adding support for owner links in the Dags view UI (#25280)
+- clear specific dag run TI (#23516)
+- Possibility to document DAG with a separated markdown file (#25509)
+- Add parsing context to DAG Parsing (#25161)
+- Implement ``CronTriggerTimetable`` (#23662)
+- Add option to mask sensitive data in UI configuration page (#25346)
+- Create new databases from the ORM (#24156)
+- Implement ``XComArg.zip(*xcom_args)`` (#25176)
+- Introduce ``sla_miss`` metric (#23402)
+- Migrate files to ts (#25267)
+- Implement ``map()`` semantic (#25085)
+- Add override method to TaskGroupDecorator (#25160)
+- Implement ``expand_kwargs()`` (#24989)
+- Add parameter to turn off SQL query logging (#24570)
+- Add ``DagWarning`` model, and a check for missing pools (#23317)
+- Add Task Logs to Grid details panel (#24249)
+- Added small health check server and endpoint in scheduler(#23905)
+- Add built-in External Link for ``ExternalTaskMarker`` operator (#23964)
+- Add default task retry delay config (#23861)
+- Add clear DagRun endpoint. (#23451)
+- Add support for timezone as string in cron interval timetable (#23279)
+- add auto refresh to dags home page (#22900)
+
+Improvements
+^^^^^^^^^^^^
+- Add more weekday operator and sensor examples #26071 (#26098)
+- Add subdir parameter to dags reserialize command (#26170)
+- Update zombie message to be more descriptive (#26141)
+- Only send an ``SlaCallbackRequest`` if the DAG is scheduled (#26089)
+- Promote ``Operator.output`` more (#25617)
+- Upgrade API files to typescript (#25098)
+- Less hacky double-rendering prevention in mapped task (#25924)
+- Improve Audit log (#25856)
+- Remove mapped operator validation code (#25870)
+- More ``DAG(schedule=...)`` improvements (#25648)
+- Reduce ``operator_name`` dupe in serialised JSON (#25819)
+- make grid view group/mapped summary UI more consistent (#25723)
+- Remove useless statement in ``task_group_to_grid`` (#25654)
+- Add auto refresh to grid logs (#25621)
+- Grid Logs, add visual indicator of selected attempt (#25611)
+- Add optional data interval to ``CronTriggerTimetable`` (#25503)
+- Simply json responses (#25518)
+- Remove unused code in ``/grid`` endpoint (#25481)
+- Add and document description fields (#25370)
+- Improve Airflow logging for operator Jinja template processing (#25452)
+- Update core example DAGs to use ``@task.branch`` decorator (#25242)
+- Fix DAG ``audit_log`` route (#25415)
+- Add missing import in best-practices code example (#25391)
+- Change stdout and stderr access mode to append in commands (#25253)
+- Remove ``getTasks`` from Grid view (#25359)
+- Improve taskflow type hints with ParamSpec (#25173)
+- Use tables in grid details panes (#25258)
+- Disallow any dag tags longer than 100 char (#25196)
+- Explicitly list ``@dag`` arguments (#25044)
+- Only refresh active dags on dags page (#24770)
+- More typing in ``SchedulerJob`` and ``TaskInstance`` (#24912)
+- Patch ``getfqdn`` with more resilient version (#24981)
+- Replace all ``NBSP`` characters by whitespaces (#24797)
+- Have consistent types between the ORM and the migration files (#24044)
+- Re-serialize all DAGs on ``airflow db upgrade`` (#24518)
+- Rework contract of try_adopt_task_instances method (#23188)
+- Make ``expand()`` error vague so it's not misleading (#24018)
+- Add enum validation for ``[webserver]analytics_tool`` (#24032)
+- Split contributor's quick start into separate guides. (#23762)
+- Add ``dttm`` searchable field in audit log (#23794)
+- Allow more parameters to be piped through via ``execute_in_subprocess`` (#23286)
+- Use ``func.count`` to count rows (#23657)
+- remove stale serialized dags (#22917)
+- AIP45 Remove dag parsing in airflow run local (#21877)
+- Add support for queued state in DagRun update endpoint. (#23481)
+- Add fields to dagrun endpoint (#23440)
Bug Fixes
^^^^^^^^^
-- ``ExternalTaskSensor`` now supports the ``soft_fail`` flag to skip if external task or DAG enters a failed state. (#23647)
+- Add the dag_id to ``AirflowDagCycleException`` message (#26204)
+- Properly build URL to retrieve logs independently from system (#26337)
+- Fix ``from airflow import version`` lazy import (#26239)
+- For worker log servers only bind to IPV6 when dual stack is available (#26222)
+- Fix ``TaskInstance.task`` not defined before ``handle_failure`` (#26040)
+- Undo secrets backend config caching (#26223)
+- Fix faulty executor config serialization logic (#26191)
+- Show ``DAGs`` and ``Datasets`` menu links based on role permission (#26183)
+- Allow setting ``TaskGroup`` tooltip via function docstring (#26028)
+- Rewrite recursion into iteration (#26175)
+- Fix backfill occasional deadlocking (#26161)
+- Fix ``DagRun.start_date`` not set during backfill with ``--reset-dagruns`` True (#26135)
+- use label instead of id for dynamic task labels in graph (#26108)
+- Don't fail DagRun when leaf ``mapped_task`` is SKIPPED (#25995)
+- Add group prefix to decorated mapped task (#26081)
+- Fix UI flash when triggering with dup logical date (#26094)
+- Fix Make items nullable for ``TaskInstance`` related endpoints to avoid API errors (#26076)
+- Fix ``BranchDateTimeOperator`` to be ``timezone-awreness-insensitive`` (#25944)
+- Fix legacy timetable schedule interval params (#25999)
+- Fix response schema for ``list-mapped-task-instance`` (#25965)
+- Properly check the existence of missing mapped TIs (#25788)
+- fix broken auto-refresh (#25950)
+- Use per-timetable ordering in grid UI (#25880)
+- Rewrite recursion when parsing DAG into iteration (#25898)
+- Find cross-group tasks in ``iter_mapped_dependants`` (#25793)
+- Fail task if mapping upstream fails (#25757)
+- Support ``/`` in variable get endpoint (#25774)
+- Use cfg default_wrap value for grid logs (#25731)
+- Add origin request args when triggering a run (#25729)
+- Operator name separate from class (#22834)
+- Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
+- Return None if an ``XComArg`` fails to resolve (#25661)
+- Correct ``json`` arg help in ``airflow variables set`` command (#25726)
+- Added MySQL index hint to use ``ti_state`` on ``find_zombies`` query (#25725)
+- Only excluded actually expanded fields from render (#25599)
+- Grid, fix toast for axios errors (#25703)
+- Check for queued states for dags autorefresh (#25695)
+- Fix upgrade code for the ``dag_owner_attributes`` table (#25579)
+- Add map index to task logs api (#25568)
+- Ensure that zombie tasks for dags with errors get cleaned up (#25550)
+- Make extra link work in UI (#25500)
+- Sync up plugin API schema and definition (#25524)
+- First/last names can be empty (#25476)
+- Refactor DAG pages to be consistent (#25402)
+- Check ``expand_kwargs()`` input type before unmapping (#25355)
+- Filter XCOM by key when calculating map lengths (#24530)
+- Fix ``ExternalTaskSensor`` not working with dynamic task (#25215)
+- Added exception catching to send default email if template file raises any exception (#24943)
+- Use ``sql_alchemy_conn`` for celery result backend when ``result_backend`` is not set (#24496)
+- Bring ``MappedOperator`` members in sync with ``BaseOperator`` (#24034)
+
+
+Misc/Internal
+^^^^^^^^^^^^^
+
+- Add automatically generated ``ERD`` schema for the ``MetaData`` DB (#26217)
Review Comment:
This one seems like it's good to be in misc. I take doc only to mean that the files are all rst?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org