You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/16 14:00:05 UTC
[GitHub] [airflow-site] paolaperaza commented on a diff in pull request #659: Add posts about 2.4 release
paolaperaza commented on code in PR #659:
URL: https://github.com/apache/airflow-site/pull/659#discussion_r973034021
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
Review Comment:
```suggestion
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
Review Comment:
```suggestion
But enough talking, lets have a short example. First lets write a simple DAG with a task called `my_task` that creates a dataset called `my-dataset`:
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
Review Comment:
```suggestion
What does this mean, exactly? It means that if one task updates a dataset in your ecosystem, for example, you can write a second task that runs _only when_ that dataset has been updated. While the `ExternalTaskSensor` or the `TriggerDagRunOperator` are able to perform that function today, datasets enable faster scheduling and more autonomous data pipelines. It enables DAG authors create smaller, self-contained DAGs that can chain together into a larger data-based workflow that is aware of the underlying data you're working with.
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
Review Comment:
```suggestion
This is just the beginning for data-aware functionality in Airflow. We know that what we're releasing today won't accomplish all use cases, but we're committed to expanding and building upon this foundation in future releases.
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
+
+For more info on dynamic task mapping please the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
+
+[task-mapping-zip]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#combining-upstream-data-aka-zipping
+[transforming-mapped-data]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#transforming-mapped-data
+[expand-kwargs]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#assigning-multiple-parameters-to-a-non-taskflow-operator
+
+## Auto-register DAGS used in a context manager (no more `as dag:` needed)
+
+This one is a small quality of life improvement, and I don't want to admit how many times I forgot the `as dag:`, or worse, had `as dag:` repeated (meaning only the last use showed up)
Review Comment:
```suggestion
This one is a small quality of life improvement, and I don't want to admit how many times I forgot the `as dag:`, or worse, had `as dag:` repeated.
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
+
+For more info on dynamic task mapping please the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
+
+[task-mapping-zip]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#combining-upstream-data-aka-zipping
+[transforming-mapped-data]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#transforming-mapped-data
+[expand-kwargs]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#assigning-multiple-parameters-to-a-non-taskflow-operator
+
+## Auto-register DAGS used in a context manager (no more `as dag:` needed)
+
+This one is a small quality of life improvement, and I don't want to admit how many times I forgot the `as dag:`, or worse, had `as dag:` repeated (meaning only the last use showed up)
+
+```python
+
+with DAG(dag_id="example") as dag:
+ ...
+
+
+@dag
+def dag_maker():
+ ...
+
+
+dag2 = dag_maker()
+
+can become
+
+```python
+
+with DAG(dag_id="example"):
+ ...
+
+
+@dag
+def my_dag():
+ ...
+
+
+my_dag()
+
+If you want to disable the behaviour for any reason then set `auto_register=False` on the dag:
+
+```python
+# This dag will not be picked up by Airflow as it's not assigned to a variable
+with DAG(dag_id="example", auto_register=False):
+ ...
+```
+
+## Removal of experimental Smart Sensors feature
+
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed. If you were using smart sensors you will have to switch to using deferrable operators before you can upgrade.
+
+We're sorry to remove this feature (and we didn't do it lightly) but to enable us to continue to grow and evolve Airflow we needed to remove this experimental code. We will only do this sort of change in a minor release for features marked as experimental, for anything that is fully supported it won't be removed or broken until the next major release.
Review Comment:
```suggestion
We're sorry to remove this feature (we didn't do it lightly) but to enable us to continue to grow and evolve Airflow we needed to remove this experimental code. We will only do this sort of change in a minor release for features marked as experimental. Any feature that is fully supported will only ever be removed in a major release.
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
+
+For more info on dynamic task mapping please the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
+
+[task-mapping-zip]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#combining-upstream-data-aka-zipping
+[transforming-mapped-data]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#transforming-mapped-data
+[expand-kwargs]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#assigning-multiple-parameters-to-a-non-taskflow-operator
+
+## Auto-register DAGS used in a context manager (no more `as dag:` needed)
+
+This one is a small quality of life improvement, and I don't want to admit how many times I forgot the `as dag:`, or worse, had `as dag:` repeated (meaning only the last use showed up)
+
+```python
+
+with DAG(dag_id="example") as dag:
+ ...
+
+
+@dag
+def dag_maker():
+ ...
+
+
+dag2 = dag_maker()
+
+can become
+
+```python
+
+with DAG(dag_id="example"):
+ ...
+
+
+@dag
+def my_dag():
+ ...
+
+
+my_dag()
+
+If you want to disable the behaviour for any reason then set `auto_register=False` on the dag:
+
+```python
+# This dag will not be picked up by Airflow as it's not assigned to a variable
+with DAG(dag_id="example", auto_register=False):
+ ...
+```
+
+## Removal of experimental Smart Sensors feature
+
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed. If you were using smart sensors you will have to switch to using deferrable operators before you can upgrade.
+
+We're sorry to remove this feature (and we didn't do it lightly) but to enable us to continue to grow and evolve Airflow we needed to remove this experimental code. We will only do this sort of change in a minor release for features marked as experimental, for anything that is fully supported it won't be removed or broken until the next major release.
+
+## Other small features
+
+With over 650 commits the full list of features, fixes and changes is too big to go in to here (check out the release notes for a full list), but some noteworthy or interesting small features include:
+
+- Auto-refresh on the home page
+- Add `@task.short_circuit` TaskFlow decorator
Review Comment:
PRs?
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
Review Comment:
```suggestion
Apache Airflow 2.4.0 contains over 650 "user-facing" commits (excluding commits to providers or chart) and over 870 total. That includes 50 new features, 99 improvements, 85 bug fixes, and several documentation changes.
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
Review Comment:
```suggestion
Datasets are defined as a URI. Now, we can create a second DAG (`consumer`) that gets scheduled whenever this dataset changes:
```
Do we want to add a sentence or so on how this dataset object is actually "connected" to... your literal dataset? How does Airflow know what this is?
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
Review Comment:
```suggestion
You asked, we listened. Dynamic task mapping now includes support for:
- `expand_kwargs`: To assign multiple parameters to a non-TaskFlow operator.
- `zip`: To combine multiple things without cross-product.
- `map`: To transform the parameters just before the task is run.
```
@ashb What does `without cross-product` mean?
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
Review Comment:
```suggestion
Now, the instant that `my_task` finishes and the dataset is created with the first DAG, Airflow will schedule and create a DAG run of the `dataset-consumer` DAG that then perhaps updates or writes to that dataset.
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
Review Comment:
```suggestion
This is just the beginning for data-aware functionality in Airflow. For now, a dataset in Airflow represents the abstract concept of a dataset, versus a specific database or table that you can specify and Airflow can read from or write to.
We know that what we're releasing today won't accomplish all use cases, but we're committed to expanding and building upon this foundation in future releases.
```
I'd just combine these two paragraphs
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
Review Comment:
```suggestion
For more information on datasets, see [documentation Data-aware scheduling][data-aware-scheduling]. That includes details on how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what a dataset is. (Hint: don't include "date partitions" in a dataset, it's higher level than that).
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
+
+For more info on dynamic task mapping please the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
Review Comment:
```suggestion
For more information on dynamic task mapping, see the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
+
+For more info on dynamic task mapping please the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
+
+[task-mapping-zip]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#combining-upstream-data-aka-zipping
+[transforming-mapped-data]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#transforming-mapped-data
+[expand-kwargs]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#assigning-multiple-parameters-to-a-non-taskflow-operator
+
+## Auto-register DAGS used in a context manager (no more `as dag:` needed)
+
+This one is a small quality of life improvement, and I don't want to admit how many times I forgot the `as dag:`, or worse, had `as dag:` repeated (meaning only the last use showed up)
+
+```python
+
+with DAG(dag_id="example") as dag:
+ ...
+
+
+@dag
+def dag_maker():
+ ...
+
+
+dag2 = dag_maker()
+
+can become
+
+```python
+
+with DAG(dag_id="example"):
+ ...
+
+
+@dag
+def my_dag():
+ ...
+
+
+my_dag()
+
+If you want to disable the behaviour for any reason then set `auto_register=False` on the dag:
+
+```python
+# This dag will not be picked up by Airflow as it's not assigned to a variable
+with DAG(dag_id="example", auto_register=False):
+ ...
+```
+
+## Removal of experimental Smart Sensors feature
+
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed. If you were using smart sensors you will have to switch to using deferrable operators before you can upgrade.
Review Comment:
```suggestion
Smart Sensors were added in Airflow 2.0 and are now deprecated in favor of Deferrable operators, which were released in Airflow 2.2. If you were using smart sensors, you will have to switch to using deferrable operators before you can upgrade to Airflow 2.4.
```
Link to deferrable operators docs here? Brief statement that they "perform the same function" or have the same benefits?
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
+
+For more info on dynamic task mapping please the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
+
+[task-mapping-zip]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#combining-upstream-data-aka-zipping
+[transforming-mapped-data]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#transforming-mapped-data
+[expand-kwargs]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#assigning-multiple-parameters-to-a-non-taskflow-operator
+
+## Auto-register DAGS used in a context manager (no more `as dag:` needed)
+
+This one is a small quality of life improvement, and I don't want to admit how many times I forgot the `as dag:`, or worse, had `as dag:` repeated (meaning only the last use showed up)
+
+```python
+
+with DAG(dag_id="example") as dag:
+ ...
+
+
+@dag
+def dag_maker():
+ ...
+
+
+dag2 = dag_maker()
+
+can become
+
+```python
+
+with DAG(dag_id="example"):
+ ...
+
+
+@dag
+def my_dag():
+ ...
+
+
+my_dag()
+
+If you want to disable the behaviour for any reason then set `auto_register=False` on the dag:
Review Comment:
```suggestion
If you want to disable the behaviour for any reason, set the `auto_register=False` on the DAG:
```
##########
landing-pages/site/content/en/blog/airflow-2.4.0/index.md:
##########
@@ -0,0 +1,140 @@
+---
+title: "Apache Airflow 2.4.0: Data
+linkTitle: "Apache Airflow 2.4.0"
+author: "Ash Berlin-Taylor"
+github: "ashberlin"
+linkedin: "ashberlin-taylor"
+description: "We're proud to announce that Apache Airflow 2.4.0 has been released."
+tags: [Release]
+date: "2022-09-19"
+---
+
+Apache Airflow
+Apache Airflow 2.4.0 contains over 650 "user-facing" commits in this release (excluding commits to providers or chart) and over 870 in total since 2.3.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.
+
+**Details**:
+
+📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/ \
+📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/ \
+🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html \
+🐳 Docker Image: docker pull apache/airflow:2.4.0 \
+🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0
+
+
+## Data-aware scheduling (AIP-48)
+
+This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.
+
+What does this mean, exactly? This is a great new feature and lets DAG authors create smaller, more self-contained DAGs, that can chain together into a larger data-based workflow. If you are currently using `ExternalTaskSensor` or `TriggerDagRunOperator` you should take a look at datasets -- in most cases you can replace them with something that will speed up the scheduling!
+
+But enough talking, lets have a short example. First lets create the task that produces the dataset:
+
+```python
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='producer', ...)
+ @task(outlets=[dataset])
+ def my_task():
+ ...
+```
+
+And then we can tell Airflow to schedule a DAG whenever this Dataset changes:
+
+```python
+
+from airflow import Dataset
+
+
+dataset = Dataset(uri='my-dataset')
+
+with DAG(dag_id='dataset-consumer', schedule=[dataset]):
+ ...
+```
+
+With these two dags, the instant that the `my_task` finishes Airflow will create the DAG run for the `dataset-consumer` workflow.
+
+(If you have the produce and consumer in different files you do not need to use the same Dataset object, two `Dataset()`s created with the same URI are equal.)
+
+We know that what exists right now won't fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation
+
+Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in future - and it's part of our goal to have smaller releases to get new features in your hands sooner!
+
+For more info on Datasets please see the [docs on Data-aware scheduling][data-aware-scheduling], including how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what is a Dataset (hint: don't include "date partitions" in a dataset, it's higher level than that.)
+
+[data-aware-scheduling]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/datasets.html
+
+## More improvments to Dynamic Task Mapping (AIP-42)
+
+You asked, we listened. Dynamic task mapping now includes support for `expand_kwargs` (to assign multiple parameters to a non-TaskFlow operator), `zip` (to combine multiple things without cross-product) and `map` (to transform the parameters on just before the task is run).
+
+For more info on dynamic task mapping please the new sections of the doc on [Transforming Mapped Data][transforming-mapped-data], [Combining upstream data (aka "zipping")][task-mapping-zip], and [Assigning multiple parameters to a non-TaskFlow operator][expand-kwargs].
+
+[task-mapping-zip]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#combining-upstream-data-aka-zipping
+[transforming-mapped-data]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#transforming-mapped-data
+[expand-kwargs]: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html#assigning-multiple-parameters-to-a-non-taskflow-operator
+
+## Auto-register DAGS used in a context manager (no more `as dag:` needed)
+
+This one is a small quality of life improvement, and I don't want to admit how many times I forgot the `as dag:`, or worse, had `as dag:` repeated (meaning only the last use showed up)
+
+```python
+
+with DAG(dag_id="example") as dag:
+ ...
+
+
+@dag
+def dag_maker():
+ ...
+
+
+dag2 = dag_maker()
+
+can become
+
+```python
+
+with DAG(dag_id="example"):
+ ...
+
+
+@dag
+def my_dag():
+ ...
+
+
+my_dag()
+
+If you want to disable the behaviour for any reason then set `auto_register=False` on the dag:
+
+```python
+# This dag will not be picked up by Airflow as it's not assigned to a variable
+with DAG(dag_id="example", auto_register=False):
+ ...
+```
+
+## Removal of experimental Smart Sensors feature
+
+Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed. If you were using smart sensors you will have to switch to using deferrable operators before you can upgrade.
+
+We're sorry to remove this feature (and we didn't do it lightly) but to enable us to continue to grow and evolve Airflow we needed to remove this experimental code. We will only do this sort of change in a minor release for features marked as experimental, for anything that is fully supported it won't be removed or broken until the next major release.
+
+## Other small features
Review Comment:
```suggestion
## Additional improvements
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org