You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by gi...@apache.org on 2025/09/25 21:56:32 UTC
(airflow-site) branch publish updated: Docs updated via refs/heads/main:a80ecf8c5af6b656e04385a47b2b0fc5789cbb61
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch publish
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/publish by this push:
new 0c517946b9 Docs updated via refs/heads/main:a80ecf8c5af6b656e04385a47b2b0fc5789cbb61
0c517946b9 is described below
commit 0c517946b9854ed54c96995e882a07e7640fa696
Author: GitHub Actions <ac...@users.noreply.github.com>
AuthorDate: Thu Sep 25 21:56:26 2025 +0000
Docs updated via refs/heads/main:a80ecf8c5af6b656e04385a47b2b0fc5789cbb61
---
404.html | 35 +-
_gen/indexes/en/blog-index.json | 2 +-
announcements/index.html | 56 +-
announcements/index.xml | 1 +
blocktype/index.html | 36 +-
blocktype/index.xml | 1 +
blocktype/testimonial/index.html | 36 +-
blocktype/testimonial/index.xml | 5 +-
blocktype/use-case/index.html | 36 +-
blocktype/use-case/index.xml | 5 +-
blog/airflow-1.10.10/index.html | 60 +-
blog/airflow-1.10.12/index.html | 64 +-
blog/airflow-1.10.8-1.10.9/index.html | 62 +-
blog/airflow-2.10.0/index.html | 176 +-
blog/airflow-2.2.0/index.html | 48 +-
blog/airflow-2.3.0/index.html | 132 +-
blog/airflow-2.4.0/index.html | 152 +-
blog/airflow-2.5.0/index.html | 48 +-
blog/airflow-2.6.0/index.html | 80 +-
blog/airflow-2.7.0/index.html | 52 +-
blog/airflow-2.8.0/index.html | 86 +-
blog/airflow-2.9.0/index.html | 139 +-
blog/airflow-3.1.0/index.html | 164 +-
blog/airflow-survey-2020/index.html | 2378 +++----
blog/airflow-survey-2022/index.html | 3064 ++++----
blog/airflow-survey-2023/index.html | 62 +-
blog/airflow-survey-2024/index.html | 52 +-
blog/airflow-survey/index.html | 1354 ++--
blog/airflow-three-point-oh-is-here/index.html | 54 +-
blog/airflow-two-point-oh-is-here/index.html | 108 +-
blog/airflow_summit_2021/index.html | 53 +-
blog/airflow_summit_2022/index.html | 55 +-
blog/announcing-new-website/index.html | 52 +-
blog/apache-airflow-for-newcomers/index.html | 68 +-
.../index.html | 52 +-
.../index.html | 48 +-
.../index.html | 52 +-
.../index.html | 56 +-
blog/fab-oid-vulnerability/index.html | 110 +-
.../index.html | 50 +-
blog/index.html | 42 +-
blog/index.xml | 7464 ++++++++++----------
blog/introducing_setup_teardown/index.html | 70 +-
.../index.html | 48 +-
blog/tags/airflow-summit/index.html | 41 +-
blog/tags/airflow-summit/index.xml | 3 +-
blog/tags/community/index.html | 41 +-
blog/tags/community/index.xml | 6679 +++++++++---------
blog/tags/development/index.html | 37 +-
blog/tags/development/index.xml | 1 +
blog/tags/documentation/index.html | 37 +-
blog/tags/documentation/index.xml | 1 +
blog/tags/release/index.html | 37 +-
blog/tags/release/index.xml | 710 +-
blog/tags/rest-api/index.html | 39 +-
blog/tags/rest-api/index.xml | 1 +
blog/tags/survey/index.html | 37 +-
blog/tags/survey/index.xml | 6703 +++++++++---------
blog/tags/users/index.html | 37 +-
blog/tags/users/index.xml | 6703 +++++++++---------
blog/tags/vulnerabilities/index.html | 37 +-
blog/tags/vulnerabilities/index.xml | 61 +-
categories/index.html | 35 +-
categories/index.xml | 1 +
chunk-1.60e9d.js | 1 -
chunk-187.8cb1f.js | 1 +
chunk-291.8cb1f.js | 2 +
chunk-291.8cb1f.js.LICENSE.txt | 61 +
chunk-3.60e9d.js | 1 -
chunk-4.60e9d.js | 19 -
chunk-5.60e9d.js | 51 -
chunk-537.8cb1f.js | 1 +
chunk-608.8cb1f.js | 1 +
chunk-744.8cb1f.js | 1 +
chunk-863.8cb1f.js | 2 +
chunk-863.8cb1f.js.LICENSE.txt | 1 +
code-of-conduct/index.html | 39 +-
code-of-conduct/index.xml | 1 +
community/index.html | 123 +-
community/index.xml | 56 +
community/resources.html | 22 -
.../experity => community/resources}/index.html | 130 +-
docs.60e9d.js | 1 -
docs.8cb1f.js | 1 +
docs/index.html | 39 +-
docs/index.xml | 1 +
ecosystem/index.html | 39 +-
ecosystem/index.xml | 1 +
index.html | 470 +-
index.xml | 7464 ++++++++++----------
...f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js | 1 -
...a5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js | 1 +
main.60e9d.js | 1 -
main.8cb1f.js | 1 +
meetups/index.html | 36 +-
meetups/index.xml | 1 +
privacy-notice/index.html | 37 +-
privacy-notice/index.xml | 1 +
roadmap/index.html | 79 +-
roadmap/index.xml | 1 +
...b92c623dfad7cedd424c3202bfec29db4dcb0326427.css | 7 +
...ab812b30fb885ece0ac9dedd4969f14edf568b76157.css | 7 -
search/index.html | 40 +-
sitemap.xml | 152 +-
survey/_index.html | 14 -
{search => survey}/index.html | 63 +-
survey/index.xml | 18 +
tags/index.html | 37 +-
tags/index.xml | 1 +
use-cases/adjoe/index.html | 47 +-
use-cases/adobe/index.html | 43 +-
use-cases/adyen/index.html | 43 +-
use-cases/big-fish-games/index.html | 43 +-
use-cases/business_operations/index.html | 47 +-
use-cases/dish/index.html | 43 +-
use-cases/etl_analytics/index.html | 47 +-
use-cases/experity/index.html | 47 +-
use-cases/index.html | 36 +-
use-cases/index.xml | 1 +
use-cases/infrastructure-management/index.html | 47 +-
use-cases/mlops/index.html | 43 +-
use-cases/onefootball/index.html | 47 +-
use-cases/plarium-krasnodar/index.html | 43 +-
use-cases/seniorlink/index.html | 47 +-
use-cases/sift/index.html | 43 +-
use-cases/snapp/index.html | 43 +-
use-cases/suse/index.html | 49 +-
127 files changed, 24051 insertions(+), 23744 deletions(-)
diff --git a/404.html b/404.html
index b463180c22..79dd58a63b 100644
--- a/404.html
+++ b/404.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,21 +31,26 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>404 Page not found | Apache Airflow</title><meta property="og:title" content="404 Page not found" />
-<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
-<meta property="og:type" content="website" />
-<meta property="og:url" content="/404.html" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
+<title>404 Page not found | Apache Airflow</title><meta property="og:url" content="/404.html">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="404 Page not found">
+ <meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="website">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="404 Page not found">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="name" content="404 Page not found">
+ <meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="image" content="/images/feature-image.png">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -81,10 +86,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -400,7 +405,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//404>
@@ -552,12 +557,12 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/_gen/indexes/en/blog-index.json b/_gen/indexes/en/blog-index.json
index b733e02542..e98c3af8f6 100644
--- a/_gen/indexes/en/blog-index.json
+++ b/_gen/indexes/en/blog-index.json
@@ -1 +1 @@
-{"version":"2.3.8","fields":["title","description","author","content","tags","url"],"fieldVectors":[["title/It's a \"Breeze\" to develop Apache Airflow",[0,1.557,1,1.557,2,0.476,3,0.388,4,0.501]],["description/It's a \"Breeze\" to develop Apache Airflow",[2,0.583,4,0.415,5,2.434,6,1.421,7,2.434,8,1.577,9,1.421,10,1.179,11,1.179,12,0.916,13,2.434,14,0.846,15,2.434,16,0.085]],["author/It's a \"Breeze\" to develop Apache Airflow",[11,1.521,17,1.665]],["content/It's a \"Breeze\" to develop A [...]
\ No newline at end of file
+{"version":"2.3.9","fields":["title","description","author","content","tags","url"],"fieldVectors":[["title/It's a \"Breeze\" to develop Apache Airflow",[0,1.557,1,1.557,2,0.476,3,0.388,4,0.501]],["description/It's a \"Breeze\" to develop Apache Airflow",[2,0.583,4,0.415,5,2.434,6,1.421,7,2.434,8,1.577,9,1.421,10,1.179,11,1.179,12,0.916,13,2.434,14,0.846,15,2.434,16,0.085]],["author/It's a \"Breeze\" to develop Apache Airflow",[11,1.521,17,1.665]],["content/It's a \"Breeze\" to develop A [...]
\ No newline at end of file
diff --git a/announcements/index.html b/announcements/index.html
index 27b78f333e..d24fc171b3 100644
--- a/announcements/index.html
+++ b/announcements/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,21 +32,37 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Announcements | Apache Airflow</title><meta property="og:title" content="Announcements" />
-<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
-<meta property="og:type" content="website" />
-<meta property="og:url" content="/announcements/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
+<title>Announcements | Apache Airflow</title><meta property="og:url" content="/announcements/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Announcements">
+ <meta property="og:description" content=" September 25, 2025 We’ve just released Apache Airflow 3.1.0 and Task SDK 1.1.0.
+📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ 🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0
+August 29, 2025 We’ve just released Apache Airflow 3.0.6 and Task SDK 1.0.6.
+📦 PyPI: https://pypi.org/project/apache-airflow/3.0.6/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.0.6/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.0.6/ 🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.0.6/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.0.6/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.0.6
+August 20, 2025 We’ve just released Apache Airflow 3.0.5 and Task SDK 1.0.5.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="website">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Announcements">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="name" content="Announcements">
+ <meta itemprop="description" content=" September 25, 2025 We’ve just released Apache Airflow 3.1.0 and Task SDK 1.1.0.
+📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/ 🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0
+August 29, 2025 We’ve just released Apache Airflow 3.0.6 and Task SDK 1.0.6.
+📦 PyPI: https://pypi.org/project/apache-airflow/3.0.6/ 📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.0.6/ 📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.0.6/ 🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.0.6/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.0.6/installation/installing-from-sources.html 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.0.6
+August 20, 2025 We’ve just released Apache Airflow 3.0.5 and Task SDK 1.0.5.">
+ <meta itemprop="datePublished" content="2020-08-08T05:30:20+05:30">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="5483">
+ <meta itemprop="image" content="/images/feature-image.png">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -82,10 +98,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -868,7 +884,7 @@ There will also be a series of presentations on non-code contributions driving t
you can take part in local in-person events organized worldwide for data communities
to watch the event and network.</p>
<h3 id="interested">Interested?</h3>
-<p>🪶 <a href="https://www.crowdcast.io/e/airflowsummit2022/register?utm_campaign=Astronomer_marketing&utm_source=Astronomer%20website&utm_medium=website&utm_term=Airflow%20Summit">Register for Airflow Summit 2022</a> today! <br>
+<p>🪶 <a href="https://www.crowdcast.io/e/airflowsummit2022/register?utm_campaign=Astronomer_marketing&utm_source=Astronomer%20website&utm_medium=website&utm_term=Airflow%20Summit">Register for Airflow Summit 2022</a> today! <br>
🗣️ If you have an Airflow story to share, <a href="https://sessionize.com/airflow-summit-2022/">join as a speaker</a> <br>
✨ <a href="https://www.linkedin.com/company/airflow-summit/">Follow Airflow Summit on LinkedIn</a> to stay current with the latest updates. <br>
🤝 <a href="https://airflowsummit.org/in-person-events/">Check out the in-person events</a> planned for Airflow Summit 2022.</p>
@@ -1212,7 +1228,7 @@ Changelog: <a href="https://airflow.readthedocs.io/en/1.10.13/changelog.html">ht
<p>Jarek has been one of the most active community members and has spread the word about Airflow Well deserved Jarek, congratulations</p>
<h1 id="sep-24-2019">Sep 24, 2019</h1>
<p>What do you think of our new logo? <em>#AirflowLogo</em> <em>#ApacheAirflow</em></p>
-<p><img src="/images/airflow_dark_bg.png" alt="Airflow"></p>
+<p><img alt="Airflow" src="/images/airflow_dark_bg.png"></p>
<h1 id="sep-18-2019">Sep 18, 2019</h1>
<p>Two meetups are happening soon:</p>
<ol>
@@ -1345,7 +1361,7 @@ Changelog: <a href="https://airflow.readthedocs.io/en/1.10.13/changelog.html">ht
<p>The Apache Airflow Meetup group in Amsterdam will be hosting its <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/events/245382408/?eventId=245382408">1st meetup (Heineken) on Dec 21</a>.</p>
<p>Niels Zeilemaker from GoDataDriven will present Using Azure Container Instances as a cheap method to run Heineken data science workloads, and Daniel van der Ende from ING Wholesale banking Advanced Analytics will talk about Data Tests using Apache Airflow.</p>
<h1 id="nov-5-2017">Nov 5, 2017</h1>
-<p>We'’re happy to welcome a new Apache Airflow Meetup group in <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/">Amsterdam</a></p>
+<p>We’’re happy to welcome a new Apache Airflow Meetup group in <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/">Amsterdam</a></p>
<h1 id="nov-1-2017">Nov 1, 2017</h1>
<p>Next Bay Area Airflow meet-up hosted by <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/">Airbnb this December (Dec 4)</a></p>
<p>Slides are available <a href="https://drive.google.com/drive/folders/154jnUADKfrHXLUDvJBQOl8aiBwpaw9-w">here</a>.</p>
@@ -1357,7 +1373,7 @@ Changelog: <a href="https://airflow.readthedocs.io/en/1.10.13/changelog.html">ht
<p>Release Notes - <a href="http://bit.ly/2gH9QFx">http://bit.ly/2gH9QFx</a></p>
<p>Special Thanks to Maxime Beauchemin (<a href="https://github.com/mistercrunch">@mistercrunch</a>) for tirelessly shepherding this release!</p>
<h1 id="april-12-2017">April 12, 2017</h1>
-<p>We'’re happy to welcome a new Apache Meetup group in <a href="https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/">Tokyo</a></p>
+<p>We’’re happy to welcome a new Apache Meetup group in <a href="https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/">Tokyo</a></p>
<p>Their first event is being held on <a href="https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/events/238731591/">May 11</a></p>
<p>Slides :</p>
<ul>
@@ -1388,7 +1404,7 @@ Changelog: <a href="https://airflow.readthedocs.io/en/1.10.13/changelog.html">ht
<h1 id="mar-14-2017">Mar 14, 2017</h1>
<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Alex Guziel</strong> (<a href="https://github.com/saguziel">@saguziel</a>)</p>
<h1 id="feb-3-2017">Feb 3, 2017</h1>
-<p>We'’re happy to welcome a new <a href="https://www.meetup.com/NYC-Apache-Airflow-incubating-Meetup/">Apache Meetup group in NYC</a></p>
+<p>We’’re happy to welcome a new <a href="https://www.meetup.com/NYC-Apache-Airflow-incubating-Meetup/">Apache Meetup group in NYC</a></p>
<h1 id="feb-2-2017">Feb 2, 2017</h1>
<p>Next Bay Area Apache Airflow Meet-up (at PayPal) on Mar 14 : <a href="http://bit.ly/2knyNHv">http://bit.ly/2knyNHv</a></p>
<h1 id="jan-30-2017">Jan 30, 2017</h1>
@@ -1501,7 +1517,7 @@ yet have a viable 1.7.1 release candidate. On April 12, <a href="https://github.
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/announcements/_index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//announcements>
@@ -1653,12 +1669,12 @@ yet have a viable 1.7.1 release candidate. On April 12, <a href="https://github.
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/announcements/index.xml b/announcements/index.xml
index 319a2c5e02..01e02c9d03 100644
--- a/announcements/index.xml
+++ b/announcements/index.xml
@@ -4,6 +4,7 @@
<link>/announcements/</link>
<description>Recent content in Announcements on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
+ <language>en</language>
<lastBuildDate>Sat, 08 Aug 2020 05:30:20 +0530</lastBuildDate>
<atom:link href="/announcements/index.xml" rel="self" type="application/rss+xml" />
diff --git a/blocktype/index.html b/blocktype/index.html
index 8d00a10d99..0a97899c09 100644
--- a/blocktype/index.html
+++ b/blocktype/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,21 +32,27 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Blocktype | Apache Airflow</title><meta property="og:title" content="Blocktype" />
-<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
-<meta property="og:type" content="website" />
-<meta property="og:url" content="/blocktype/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
+<title>Blocktype | Apache Airflow</title><meta property="og:url" content="/blocktype/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Blocktype">
+ <meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="website">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Blocktype">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="name" content="Blocktype">
+ <meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="image" content="/images/feature-image.png">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -82,10 +88,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -395,7 +401,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blocktype>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blocktype>
@@ -547,13 +553,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blocktype/index.xml b/blocktype/index.xml
index c538358a1a..7553fb1939 100644
--- a/blocktype/index.xml
+++ b/blocktype/index.xml
@@ -4,6 +4,7 @@
<link>/blocktype/</link>
<description>Recent content in Blocktype on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
+ <language>en</language>
<atom:link href="/blocktype/index.xml" rel="self" type="application/rss+xml" />
diff --git a/blocktype/testimonial/index.html b/blocktype/testimonial/index.html
index d328ff925c..fd35465d79 100644
--- a/blocktype/testimonial/index.html
+++ b/blocktype/testimonial/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,21 +32,27 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>testimonial | Apache Airflow</title><meta property="og:title" content="testimonial" />
-<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
-<meta property="og:type" content="website" />
-<meta property="og:url" content="/blocktype/testimonial/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
+<title>Testimonial | Apache Airflow</title><meta property="og:url" content="/blocktype/testimonial/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Testimonial">
+ <meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="website">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="testimonial">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="name" content="Testimonial">
+ <meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="image" content="/images/feature-image.png">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -82,10 +88,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -395,7 +401,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blocktype/testimonial>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blocktype/testimonial>
@@ -547,13 +553,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blocktype/testimonial/index.xml b/blocktype/testimonial/index.xml
index b1d5c574e7..9473e994a8 100644
--- a/blocktype/testimonial/index.xml
+++ b/blocktype/testimonial/index.xml
@@ -1,9 +1,10 @@
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
- <title>Apache Airflow – testimonial</title>
+ <title>Apache Airflow – Testimonial</title>
<link>/blocktype/testimonial/</link>
- <description>Recent content in testimonial on Apache Airflow</description>
+ <description>Recent content in Testimonial on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
+ <language>en</language>
<atom:link href="/blocktype/testimonial/index.xml" rel="self" type="application/rss+xml" />
diff --git a/blocktype/use-case/index.html b/blocktype/use-case/index.html
index 1b630f7431..d0edbfa28b 100644
--- a/blocktype/use-case/index.html
+++ b/blocktype/use-case/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,21 +32,27 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>use-case | Apache Airflow</title><meta property="og:title" content="use-case" />
-<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
-<meta property="og:type" content="website" />
-<meta property="og:url" content="/blocktype/use-case/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
+<title>Use-Case | Apache Airflow</title><meta property="og:url" content="/blocktype/use-case/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Use-Case">
+ <meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="website">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="use-case">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="name" content="Use-Case">
+ <meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="image" content="/images/feature-image.png">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -82,10 +88,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -395,7 +401,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blocktype/use-case>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blocktype/use-case>
@@ -547,13 +553,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blocktype/use-case/index.xml b/blocktype/use-case/index.xml
index c534b1ab0e..377e84ec48 100644
--- a/blocktype/use-case/index.xml
+++ b/blocktype/use-case/index.xml
@@ -1,9 +1,10 @@
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
- <title>Apache Airflow – use-case</title>
+ <title>Apache Airflow – Use-Case</title>
<link>/blocktype/use-case/</link>
- <description>Recent content in use-case on Apache Airflow</description>
+ <description>Recent content in Use-Case on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
+ <language>en</language>
<atom:link href="/blocktype/use-case/index.xml" rel="self" type="application/rss+xml" />
diff --git a/blog/airflow-1.10.10/index.html b/blog/airflow-1.10.10/index.html
index 8c56a03cd0..a2eb535b43 100644
--- a/blog/airflow-1.10.10/index.html
+++ b/blog/airflow-1.10.10/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 1.10.10 | Apache Airflow</title><meta property="og:title" content="Apache Airflow 1.10.10" />
-<meta property="og:description" content="We are happy to present Apache Airflow 1.10.10" />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-1.10.10/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2020-04-09T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 1.10.10 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-1.10.10/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 1.10.10">
+ <meta property="og:description" content="We are happy to present Apache Airflow 1.10.10">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2020-04-09T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 1.10.10">
-<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.10"><meta itemprop="datePublished" content="2020-04-09T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="1143"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="release," />
+ <meta itemprop="name" content="Apache Airflow 1.10.10">
+ <meta itemprop="description" content="We are happy to present Apache Airflow 1.10.10">
+ <meta itemprop="datePublished" content="2020-04-09T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="1143">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -457,7 +465,7 @@
<p>By default the Web UI will show times in UTC. It is possible to change the timezone shown by using the menu in the top
right (click on the clock to activate it):</p>
<p><strong>Screenshot</strong>:
-<img src="rbac-ui-timezone.gif" alt="Allow user to chose timezone to use in the RBAC UI"></p>
+<img alt="Allow user to chose timezone to use in the RBAC UI" src="/blog/airflow-1.10.10/rbac-ui-timezone.gif"></p>
<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/timezone.html#web-ui">https://airflow.apache.org/docs/1.10.10/timezone.html#web-ui</a></p>
<p><strong>Note</strong>: This feature is only available for the RBAC UI (enabled using <code>rbac=True</code> in <code>[webserver]</code> section in your <code>airflow.cfg</code>).</p>
<h3 id="add-production-docker-image-support">Add Production Docker image support</h3>
@@ -490,10 +498,10 @@ configure Secrets backend.</p>
</ul>
<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html">https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html</a></p>
<p>Example configuration to use Hashicorp Vault as the backend:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="k">[secrets]</span>
-<span class="na">backend</span> <span class="o">=</span> <span class="s">airflow.contrib.secrets.hashicorp_vault.VaultBackend</span>
-<span class="na">backend_kwargs</span> <span class="o">=</span> <span class="s">{"url": "http://127.0.0.1:8200", "connections_path": "connections", "variables_path": "variables", "mount_point": "airflow"}</span>
-</code></pre></div><h3 id="stateless-webserver-using-dag-serialization">Stateless Webserver using DAG Serialization</h3>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[secrets]</span>
+</span></span><span class="line"><span class="cl"><span class="na">backend</span> <span class="o">=</span> <span class="s">airflow.contrib.secrets.hashicorp_vault.VaultBackend</span>
+</span></span><span class="line"><span class="cl"><span class="na">backend_kwargs</span> <span class="o">=</span> <span class="s">{"url": "http://127.0.0.1:8200", "connections_path": "connections", "variables_path": "variables", "mount_point": "airflow"}</span>
+</span></span></code></pre></div><h3 id="stateless-webserver-using-dag-serialization">Stateless Webserver using DAG Serialization</h3>
<p>The Webserver can now run without access to DAG Files when DAG Serialization is turned on.
The 2 limitations we had in 1.10.7-1.10.9 (
<a href="https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations">https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations</a>)
@@ -516,7 +524,7 @@ Executor. This should significantly improve execution time and resource usage.</
<p>From Airflow 1.10.10, when a user clicks on Trigger Dag button, a new screen confirming the trigger request, and allowing the user to pass a JSON configuration
blob would be show.</p>
<p><strong>Screenshot</strong>:
-<img src="trigger-dag-conf.png" alt="Allow passing DagRun conf when triggering dags via UI"></p>
+<img alt="Allow passing DagRun conf when triggering dags via UI" src="/blog/airflow-1.10.10/trigger-dag-conf.png"></p>
<p>Details: <a href="https://github.com/apache/airflow/pull/7312">https://github.com/apache/airflow/pull/7312</a></p>
<h2 id="updating-guide">Updating Guide</h2>
<p>If you are updating Apache Airflow from a previous version to <code>1.10.10</code>, please take a note of the following:</p>
@@ -723,7 +731,7 @@ objc[1873]: +[__NSPlaceholderDate initialize] may have been in progress in anoth
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-1.10.10/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-1.10.10>
@@ -875,13 +883,13 @@ objc[1873]: +[__NSPlaceholderDate initialize] may have been in progress in anoth
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-1.10.12/index.html b/blog/airflow-1.10.12/index.html
index 7ac76dddc4..c4271ff35e 100644
--- a/blog/airflow-1.10.12/index.html
+++ b/blog/airflow-1.10.12/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 1.10.12 | Apache Airflow</title><meta property="og:title" content="Apache Airflow 1.10.12" />
-<meta property="og:description" content="We are happy to present Apache Airflow 1.10.12" />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-1.10.12/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2020-08-25T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 1.10.12 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-1.10.12/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 1.10.12">
+ <meta property="og:description" content="We are happy to present Apache Airflow 1.10.12">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2020-08-25T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 1.10.12">
-<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.12"><meta itemprop="datePublished" content="2020-08-25T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="911"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="release," />
+ <meta itemprop="name" content="Apache Airflow 1.10.12">
+ <meta itemprop="description" content="We are happy to present Apache Airflow 1.10.12">
+ <meta itemprop="datePublished" content="2020-08-25T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="911">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -475,9 +483,9 @@ than <code>XCom.MAX_XCOM_SIZE</code> (48 KB).</p>
<p>Further improving Airflow’s Secret Management story, from Airflow 1.10.12, users don’t need to hardcode
the <strong>sensitive</strong> config value in airflow.cfg nor then need to use an Environment variable to set this config.</p>
<p>For example, the metadata database connection string can either be set in airflow.cfg like this:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="k">[core]</span>
-<span class="na">sql_alchemy_conn_secret</span> <span class="o">=</span> <span class="s">sql_alchemy_conn</span>
-</code></pre></div><p>This will retrieve config option from the set Secret Backends.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[core]</span>
+</span></span><span class="line"><span class="cl"><span class="na">sql_alchemy_conn_secret</span> <span class="o">=</span> <span class="s">sql_alchemy_conn</span>
+</span></span></code></pre></div><p>This will retrieve config option from the set Secret Backends.</p>
<p>As you can see you just need to add a <code>_secret</code> suffix at the end of the actual config option
and the value needs to be the <strong>key</strong> which the Secrets backend will look for.</p>
<p>Similarly, <code>_secret</code> config options can also be set using a corresponding environment variable. For example:</p>
@@ -493,11 +501,11 @@ your check.</p>
beginner errors that may get past a code reviewer, rather than as technical
security controls.</p>
<p>For example, don’t run tasks without <code>airflow</code> owners:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">task_must_have_owners</span><span class="p">(</span><span class="n">task</span><span class="p">):</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">task</span><span class="o">.</span><span class="n">owner</span> <span class="ow">or</span> <span class="n">task</span><span class="o">.</span><span class="n">owner</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="n">conf</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'operators'</span> [...]
- <span class="k">raise</span> <span class="n">AirflowClusterPolicyViolation</span><span class="p">(</span>
- <span class="s1">'Task must have non-None non-default owner. Current value: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">task</span><span class="o">.</span><span class="n">owner</span><span class="p">))</span>
-</code></pre></div><p>More details: <a href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">task_must_have_owners</span><span class="p">(</span><span class="n">task</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="ow">not</span> <span class="n">task</span><span class="o">.</span><span class="n">owner</span> <span class="ow">or</span> <span class="n">task</span><span class="o">.</span><span class="n">owner</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="n">conf</span><span class="o">.</span><span class="n">get</span><span class="p">( [...]
+</span></span><span class="line"><span class="cl"> <span class="k">raise</span> <span class="n">AirflowClusterPolicyViolation</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="s1">'Task must have non-None non-default owner. Current value: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">task</span><span class="o">.</span><span class="n">owner</span><span class="p">))</span>
+</span></span></code></pre></div><p>More details: <a href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
<h3 id="launch-pods-via-yaml-files-when-using-kubernetesexecutor-and-kubernetespodoperator">Launch Pods via YAML files when using KubernetesExecutor and KubernetesPodOperator</h3>
<p>As of 1.10.12, users can launch pods via YAML files instead of passing various configurations.</p>
<p>To allow greater flexibility we have deprecated Airflow’s Pod class and instead now use classes and
@@ -701,7 +709,7 @@ William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314, retornam,
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-1.10.12/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-1.10.12>
@@ -853,13 +861,13 @@ William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314, retornam,
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-1.10.8-1.10.9/index.html b/blog/airflow-1.10.8-1.10.9/index.html
index a8cbb5ffd4..ba74917415 100644
--- a/blog/airflow-1.10.8-1.10.9/index.html
+++ b/blog/airflow-1.10.8-1.10.9/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 1.10.8 & 1.10.9 | Apache Airflow</title><meta property="og:title" content="Apache Airflow 1.10.8 & 1.10.9" />
-<meta property="og:description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-1.10.8-1.10.9/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2020-02-23T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 1.10.8 & 1.10.9 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-1.10.8-1.10.9/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 1.10.8 & 1.10.9">
+ <meta property="og:description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2020-02-23T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 1.10.8 & 1.10.9">
-<meta itemprop="description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow."><meta itemprop="datePublished" content="2020-02-23T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="437"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="release," />
+ <meta itemprop="name" content="Apache Airflow 1.10.8 & 1.10.9">
+ <meta itemprop="description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow.">
+ <meta itemprop="datePublished" content="2020-02-23T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="437">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -458,13 +466,13 @@
<p>In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button.</p>
<p>For example:</p>
<p>In your Dag file, pass a list of tags you want to add to DAG object:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
- <span class="n">dag_id</span><span class="o">=</span><span class="s1">'example_dag_tag'</span><span class="p">,</span>
- <span class="n">schedule_interval</span><span class="o">=</span><span class="s1">'0 0 * * *'</span><span class="p">,</span>
- <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s1">'example'</span><span class="p">]</span>
-<span class="p">)</span>
-</code></pre></div><p><strong>Screenshot</strong>:
-<img src="airflow-dag-tags.png" alt="Add filter by DAG tags"></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">dag_id</span><span class="o">=</span><span class="s1">'example_dag_tag'</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">schedule_interval</span><span class="o">=</span><span class="s1">'0 0 * * *'</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s1">'example'</span><span class="p">]</span>
+</span></span><span class="line"><span class="cl"><span class="p">)</span>
+</span></span></code></pre></div><p><strong>Screenshot</strong>:
+<img alt="Add filter by DAG tags" src="/blog/airflow-1.10.8-1.10.9/airflow-dag-tags.png"></p>
<p><strong>Note</strong>: This feature is only available for the RBAC UI (enabled using <code>rbac=True</code> in <code>[webserver]</code> section in your <code>airflow.cfg</code>).</p>
<h2 id="special-note--deprecations">Special Note / Deprecations</h2>
<h3 id="python-2">Python 2</h3>
@@ -635,7 +643,7 @@ for 1.10.* series but would no longer be available from Airflow 2.0</p>
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-1.10.8-1.10.9>
@@ -787,13 +795,13 @@ for 1.10.* series but would no longer be available from Airflow 2.0</p>
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.10.0/index.html b/blog/airflow-2.10.0/index.html
index 4598f2a0da..583bc274a8 100644
--- a/blog/airflow-2.10.0/index.html
+++ b/blog/airflow-2.10.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 2.10.0 is here | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.10.0 is here" />
-<meta property="og:description" content="Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!" />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.10.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2024-08-08T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 2.10.0 is here | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.10.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 2.10.0 is here">
+ <meta property="og:description" content="Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2024-08-08T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 2.10.0 is here">
-<meta itemprop="description" content="Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!"><meta itemprop="datePublished" content="2024-08-08T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="1040"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 2.10.0 is here">
+ <meta itemprop="description" content="Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!">
+ <meta itemprop="datePublished" content="2024-08-08T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="1040">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -452,94 +460,94 @@
<p>We are introducing a new feature that allows for the concurrent use of multiple executors within a single Airflow environment. This flexibility enables users to take advantage of the specific strengths of different executors for various tasks, improving overall efficiency and mitigating weaknesses. Users can set a default executor for the entire environment and, if necessary, assign particular executors to individual DAGs or tasks.</p>
<p>To configure multiple executors we can pass comma separated list in airflow configuration. The first executor in the list will be the default executor for the environment.</p>
<pre tabindex="0"><code>[core]
-executor = 'LocalExecutor,CeleryExecutor'
+executor = 'LocalExecutor,CeleryExecutor'
</code></pre><p>To make it easier for dag authors, we can also specify aliases for executors that can be specified in the executor configuration</p>
<pre tabindex="0"><code class="language-commandline" data-lang="commandline">[core]
-executor = 'LocalExecutor,KubernetesExecutor,my.custom.module.ExecutorClass:ShortName'
+executor = 'LocalExecutor,KubernetesExecutor,my.custom.module.ExecutorClass:ShortName'
</code></pre><p>DAG authors can specify executors to use at the task</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="n">BashOperator</span><span class="p">(</span>
- <span class="n">task_id</span><span class="o">=</span><span class="s2">"hello_world"</span><span class="p">,</span>
- <span class="n">executor</span><span class="o">=</span><span class="s2">"ShortName"</span><span class="p">,</span>
- <span class="n">bash_command</span><span class="o">=</span><span class="s2">"echo 'hello world!'"</span><span class="p">,</span>
-<span class="p">)</span>
-
-<span class="nd">@task</span><span class="p">(</span><span class="n">executor</span><span class="o">=</span><span class="s2">"KubernetesExecutor"</span><span class="p">)</span>
-<span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
- <span class="nb">print</span><span class="p">(</span><span class="s2">"hello world!"</span><span class="p">)</span>
-</code></pre></div><p>We can also specify executors on the DAG level</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
- <span class="nb">print</span><span class="p">(</span><span class="s2">"hello world!"</span><span class="p">)</span>
-
-<span class="k">def</span> <span class="nf">hello_world_again</span><span class="p">():</span>
- <span class="nb">print</span><span class="p">(</span><span class="s2">"hello world again!"</span><span class="p">)</span>
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
- <span class="n">dag_id</span><span class="o">=</span><span class="s2">"hello_worlds"</span><span class="p">,</span>
- <span class="n">default_args</span><span class="o">=</span><span class="p">{</span><span class="s2">"executor"</span><span class="p">:</span> <span class="s2">"ShortName"</span><span class="p">},</span> <span class="c1"># Applies to all tasks in the DAG</span>
-<span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
- <span class="c1"># All tasks will use the executor from default args automatically</span>
- <span class="n">hw</span> <span class="o">=</span> <span class="n">hello_world</span><span class="p">()</span>
- <span class="n">hw_again</span> <span class="o">=</span> <span class="n">hello_world_again</span><span class="p">()</span>
-</code></pre></div><h2 id="dynamic-dataset-scheduling-through-datasetalias">Dynamic Dataset scheduling through DatasetAlias</h2>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">BashOperator</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">task_id</span><span class="o">=</span><span class="s2">"hello_world"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">executor</span><span class="o">=</span><span class="s2">"ShortName"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">bash_command</span><span class="o">=</span><span class="s2">"echo 'hello world!'"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"><span class="p">)</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@task</span><span class="p">(</span><span class="n">executor</span><span class="o">=</span><span class="s2">"KubernetesExecutor"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s2">"hello world!"</span><span class="p">)</span>
+</span></span></code></pre></div><p>We can also specify executors on the DAG level</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s2">"hello world!"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">hello_world_again</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s2">"hello world again!"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">dag_id</span><span class="o">=</span><span class="s2">"hello_worlds"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">default_args</span><span class="o">=</span><span class="p">{</span><span class="s2">"executor"</span><span class="p">:</span> <span class="s2">"ShortName"</span><span class="p">},</span> <span class="c1"># Applies to all tasks in the DAG</span>
+</span></span><span class="line"><span class="cl"><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="c1"># All tasks will use the executor from default args automatically</span>
+</span></span><span class="line"><span class="cl"> <span class="n">hw</span> <span class="o">=</span> <span class="n">hello_world</span><span class="p">()</span>
+</span></span><span class="line"><span class="cl"> <span class="n">hw_again</span> <span class="o">=</span> <span class="n">hello_world_again</span><span class="p">()</span>
+</span></span></code></pre></div><h2 id="dynamic-dataset-scheduling-through-datasetalias">Dynamic Dataset scheduling through DatasetAlias</h2>
<p>Airflow 2.10 comes with <code>DatasetAlias</code> class which can be passed as a value in the <code>outlets</code>, <code>inlets</code> on a task, and <code>schedule</code> on a DAG. An instance of <code>DatasetAlias</code> is resolved dynamically to a real dataset. Downstream can depend on either the resolved dataset or on an alias itself.</p>
<p><code>DatasetAlias</code> has one argument <code>name</code> that uniquely identifies the dataset. The task must first declare the alias as an outlet, and use <code>outlet_events</code> or <code>yield Metadata</code> to add events to it.</p>
<h3 id="emit-a-dataset-event-during-task-execution-through-outlet_events">Emit a dataset event during task execution through outlet_events</h3>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">airflow.datasets</span> <span class="kn">import</span> <span class="n">DatasetAlias</span>
-
-<span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"my-task-outputs"</span><span class="p">)])</span>
-<span class="k">def</span> <span class="nf">my_task_with_outlet_events</span><span class="p">(</span><span class="o">*</span><span class="p">,</span> <span class="n">outlet_events</span><span class="p">):</span>
- <span class="n">outlet_events</span><span class="p">[</span><span class="s2">"my-task-outputs"</span><span class="p">]</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task"</span><span class="p">))</span>
-</code></pre></div><h3 id="emit-a-dataset-event-during-task-execution-by-yielding-metadata">Emit a dataset event during task execution by yielding Metadata</h3>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">airflow.datasets.metadata</span> <span class="kn">import</span> <span class="n">Metadata</span>
-
-<span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"my-task-outputs"</span><span class="p">)])</span>
-<span class="k">def</span> <span class="nf">my_task_with_metadata</span><span class="p">():</span>
- <span class="n">s3_dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task}"</span><span class="p">)</span>
- <span class="k">yield</span> <span class="n">Metadata</span><span class="p">(</span><span class="n">s3_dataset</span><span class="p">,</span> <span class="n">alias</span><span class="o">=</span><span class="s2">"my-task-outputs"</span><span class="p">)</span>
-</code></pre></div><p>There are two options for scheduling based on dataset aliases. Schedule based on <code>DatasetAlias</code> or real datasets.</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dataset-alias-producer"</span><span class="p">):</span>
- <span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"example-alias"</span><span class="p">)])</span>
- <span class="k">def</span> <span class="nf">produce_dataset_events</span><span class="p">(</span><span class="o">*</span><span class="p">,</span> <span class="n">outlet_events</span><span class="p">):</span>
- <span class="n">outlet_events</span><span class="p">[</span><span class="s2">"example-alias"</span><span class="p">]</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task"</span><span class="p">))</span>
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dataset-consumer"</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task"</span><span class="p">)):</span>
- <span class="o">...</span>
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dataset-alias-consumer"</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"example-alias"</span><span class="p">)):</span>
- <span class="o">...</span>
-</code></pre></div><h3 id="dataset-aliases-ui-enhancements">Dataset Aliases UI Enhancements</h3>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.datasets</span> <span class="kn">import</span> <span class="n">DatasetAlias</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"my-task-outputs"</span><span class="p">)])</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">my_task_with_outlet_events</span><span class="p">(</span><span class="o">*</span><span class="p">,</span> <span class="n">outlet_events</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="n">outlet_events</span><span class="p">[</span><span class="s2">"my-task-outputs"</span><span class="p">]</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task"</span><span class="p">))</span>
+</span></span></code></pre></div><h3 id="emit-a-dataset-event-during-task-execution-by-yielding-metadata">Emit a dataset event during task execution by yielding Metadata</h3>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.datasets.metadata</span> <span class="kn">import</span> <span class="n">Metadata</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"my-task-outputs"</span><span class="p">)])</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">my_task_with_metadata</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="n">s3_dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task}"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">Metadata</span><span class="p">(</span><span class="n">s3_dataset</span><span class="p">,</span> <span class="n">alias</span><span class="o">=</span><span class="s2">"my-task-outputs"</span><span class="p">)</span>
+</span></span></code></pre></div><p>There are two options for scheduling based on dataset aliases. Schedule based on <code>DatasetAlias</code> or real datasets.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dataset-alias-producer"</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"example-alias"</span><span class="p">)])</span>
+</span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">produce_dataset_events</span><span class="p">(</span><span class="o">*</span><span class="p">,</span> <span class="n">outlet_events</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="n">outlet_events</span><span class="p">[</span><span class="s2">"example-alias"</span><span class="p">]</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task"</span><span class="p">))</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dataset-consumer"</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="n">Dataset</span><span class="p">(</span><span class="s2">"s3://bucket/my-task"</span><span class="p">)):</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dataset-alias-consumer"</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="n">DatasetAlias</span><span class="p">(</span><span class="s2">"example-alias"</span><span class="p">)):</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><h3 id="dataset-aliases-ui-enhancements">Dataset Aliases UI Enhancements</h3>
<p>Now users can see Dataset Aliases in legend of each cross-dag dependency graph with a corresponded icon/color.</p>
-<p><img src="dag_dependencies_legend.png" alt="DAG Dependencies graph"></p>
+<p><img alt="DAG Dependencies graph" src="/blog/airflow-2.10.0/dag_dependencies_legend.png"></p>
<h2 id="dark-mode-for-airflow-ui">Dark Mode for Airflow UI</h2>
<p>Airflow 2.10 comes with new Dark Mode feature which is designed to enhance user experience by offering an alternative visual theme that is easier on the eyes, especially in low-light conditions. You can toggle the crescent icon on the right side of the navigation bar to switch between light and dark mode.</p>
-<p><img src="airflow_dark_mode.png" alt="Airflow Dark mode"></p>
-<p><img src="airflow_light_mode.png" alt="Airflow Light mode"></p>
+<p><img alt="Airflow Dark mode" src="/blog/airflow-2.10.0/airflow_dark_mode.png"></p>
+<p><img alt="Airflow Light mode" src="/blog/airflow-2.10.0/airflow_light_mode.png"></p>
<h2 id="task-instance-history">Task Instance History</h2>
<p>In Apache Airflow 2.10.0, when a task instance is retried or cleared, its execution history is maintained. You can view this history by clicking on the task instance in the Grid view, allowing you to access information about each attempt, such as logs, execution durations, and any failures. This feature improves transparency into the task’s execution process, making it easier to troubleshoot and analyze your DAGs.</p>
-<p><img src="task_instance_history.png" alt="Task instance history"></p>
+<p><img alt="Task instance history" src="/blog/airflow-2.10.0/task_instance_history.png"></p>
<p>The history displays the final values of the task instance attributes for each specific run. On the log page, you can also access the logs for each attempt of the task instance. This information is valuable for debugging purposes.</p>
-<p><img src="task_instance_history_log.png" alt="Task instance history"></p>
+<p><img alt="Task instance history" src="/blog/airflow-2.10.0/task_instance_history_log.png"></p>
<h2 id="dataset-ui-enhancements">Dataset UI Enhancements</h2>
<p>The dataset page has been revamped to include a focused dataset events section with additional details such as extras, consuming DAGs, and producing tasks.
-<img src="dataset_list.png" alt="Dataset list"></p>
+<img alt="Dataset list" src="/blog/airflow-2.10.0/dataset_list.png"></p>
<p>We now have separate dependency graph and dataset list pages in new tabs, enhancing the user experience.</p>
-<p><img src="dependency_graph.png" alt="Dataset dependency graph"></p>
+<p><img alt="Dataset dependency graph" src="/blog/airflow-2.10.0/dependency_graph.png"></p>
<p>Dataset events are now displayed in both the Details tab of each DAG run and within the DAG graph.</p>
-<p><img src="dataset_details.png" alt="Dataset list"></p>
+<p><img alt="Dataset list" src="/blog/airflow-2.10.0/dataset_details.png"></p>
<h3 id="toggle-datasets-in-graph">Toggle datasets in Graph</h3>
<p>We can now toggle the datasets in the DAG graph</p>
-<p><img src="dataset_toggle_on.png" alt="Dataset toggle button on">
-<img src="dataset_toggle_off.png" alt="Dataset toggle button off"></p>
+<p><img alt="Dataset toggle button on" src="/blog/airflow-2.10.0/dataset_toggle_on.png">
+<img alt="Dataset toggle button off" src="/blog/airflow-2.10.0/dataset_toggle_off.png"></p>
<h3 id="dataset-conditions-in-dag-graph-view">Dataset Conditions in DAG Graph view</h3>
<p>We now display the graph view with logical gates. Datasets with actual events are highlighted with a different border, making it easier to see what triggered the selected run.</p>
-<p><img src="render_dataset_conditions.png" alt="Render dataset conditions in graph view"></p>
+<p><img alt="Render dataset conditions in graph view" src="/blog/airflow-2.10.0/render_dataset_conditions.png"></p>
<h3 id="dataset-event-info-in-dag-graph">Dataset event info in DAG Graph</h3>
<p>For a DAG run, users can now view the dataset events connected to it directly in the graph view.</p>
-<p><img src="dataset_info.png" alt="Dataset event info"></p>
+<p><img alt="Dataset event info" src="/blog/airflow-2.10.0/dataset_info.png"></p>
<h2 id="on-demand-dag-re-parsing">On-demand DAG Re-parsing</h2>
<p>In 2.10 users can now reparse the DAGs on demand using below button on DAG list and DAG detail pages</p>
-<p><img src="DAG_reparsing_button_list.png" alt="DAG Reparsing button on DAG list page">
-<img src="DAG_reparse_button_detail.png" alt="DAG Reparsing button on DAG detail page"></p>
+<p><img alt="DAG Reparsing button on DAG list page" src="/blog/airflow-2.10.0/DAG_reparsing_button_list.png">
+<img alt="DAG Reparsing button on DAG detail page" src="/blog/airflow-2.10.0/DAG_reparse_button_detail.png"></p>
<h2 id="additional-new-features">Additional new features</h2>
<p>Here are just a few interesting new features since there are too many to list in full:</p>
<ul>
@@ -717,7 +725,7 @@ executor = 'LocalExecutor,KubernetesExecutor,my.custom.module.ExecutorClass:Shor
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.10.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.10.0>
@@ -869,13 +877,13 @@ executor = 'LocalExecutor,KubernetesExecutor,my.custom.module.ExecutorClass:Shor
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.2.0/index.html b/blog/airflow-2.2.0/index.html
index 41810b51e3..c238f2653f 100644
--- a/blog/airflow-2.2.0/index.html
+++ b/blog/airflow-2.2.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>What's new in Apache Airflow 2.2.0 | Apache Airflow</title><meta property="og:title" content="What's new in Apache Airflow 2.2.0" />
-<meta property="og:description" content="We're proud to announce that Apache Airflow 2.2.0 has been released." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.2.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2021-10-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>What's new in Apache Airflow 2.2.0 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.2.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="What's new in Apache Airflow 2.2.0">
+ <meta property="og:description" content="We're proud to announce that Apache Airflow 2.2.0 has been released.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2021-10-11T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="What's new in Apache Airflow 2.2.0">
-<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.2.0 has been released."><meta itemprop="datePublished" content="2021-10-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="658"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="What's new in Apache Airflow 2.2.0">
+ <meta itemprop="description" content="We're proud to announce that Apache Airflow 2.2.0 has been released.">
+ <meta itemprop="datePublished" content="2021-10-11T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="658">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -636,7 +644,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.2.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.2.0>
@@ -788,13 +796,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.3.0/index.html b/blog/airflow-2.3.0/index.html
index 8fd51b5c8a..cf2a1869a2 100644
--- a/blog/airflow-2.3.0/index.html
+++ b/blog/airflow-2.3.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 2.3.0 is here | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.3.0 is here" />
-<meta property="og:description" content="We're proud to announce that Apache Airflow 2.3.0 has been released." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.3.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2022-04-30T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 2.3.0 is here | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.3.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 2.3.0 is here">
+ <meta property="og:description" content="We're proud to announce that Apache Airflow 2.3.0 has been released.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2022-04-30T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 2.3.0 is here">
-<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.3.0 has been released."><meta itemprop="datePublished" content="2022-04-30T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="708"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 2.3.0 is here">
+ <meta itemprop="description" content="We're proud to announce that Apache Airflow 2.3.0 has been released.">
+ <meta itemprop="datePublished" content="2022-04-30T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="708">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -450,25 +458,25 @@
to create a list of tasks, here you can create the same tasks without having to know the exact number of tasks ahead of time.</p>
<p>You can have a <code>task</code> generate the list to iterate over, which is not possible with a <code>for</code> loop.</p>
<p>Here is an example:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="nd">@task</span>
-<span class="k">def</span> <span class="nf">make_list</span><span class="p">():</span>
- <span class="c1"># This can also be from an API call, checking a database, -- almost anything you like, as long as the</span>
- <span class="c1"># resulting list/dictionary can be stored in the current XCom backend.</span>
- <span class="k">return</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="p">{</span><span class="s2">"a"</span><span class="p">:</span> <span class="s2">"b"</span><span class="p">},</span> <span class="s2">"str"</span><span class="p">]</span>
-
-
-<span class="nd">@task</span>
-<span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
- <span class="nb">print</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">arg</span><span class="p">))</span>
-
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dynamic-map"</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2022</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span> <span class="k">as</span> [...]
- <span class="n">consumer</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="n">arg</span><span class="o">=</span><span class="n">make_list</span><span class="p">())</span>
-</code></pre></div><p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html">Dynamic Task Mapping</a></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="nd">@task</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">make_list</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="c1"># This can also be from an API call, checking a database, -- almost anything you like, as long as the</span>
+</span></span><span class="line"><span class="cl"> <span class="c1"># resulting list/dictionary can be stored in the current XCom backend.</span>
+</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="p">{</span><span class="s2">"a"</span><span class="p">:</span> <span class="s2">"b"</span><span class="p">},</span> <span class="s2">"str"</span><span class="p">]</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@task</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">arg</span><span class="p">))</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dynamic-map"</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2022</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">2</span>< [...]
+</span></span><span class="line"><span class="cl"> <span class="n">consumer</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="n">arg</span><span class="o">=</span><span class="n">make_list</span><span class="p">())</span>
+</span></span></code></pre></div><p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html">Dynamic Task Mapping</a></p>
<h2 id="grid-view-replaces-tree-view">Grid View replaces Tree View</h2>
<p>Grid view replaces tree view in Airflow 2.3.0.</p>
<p><strong>Screenshots</strong>:
-<img src="grid-view.png" alt="The new grid view"></p>
+<img alt="The new grid view" src="/blog/airflow-2.3.0/grid-view.png"></p>
<h2 id="purge-history-from-metadata-database">Purge history from metadata database</h2>
<p>Airflow 2.3.0 introduces a new <code>airflow db clean</code> command that can be used to purge old data from the metadata database.</p>
<p>You would want to use this command if you want to reduce the size of the metadata database.</p>
@@ -482,20 +490,20 @@ to create a list of tasks, here you can create the same tasks without having to
<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/cli-and-env-variables-ref.html#dag-processor">dag-processor CLI command</a></p>
<h2 id="json-serialization-for-connections">JSON serialization for connections</h2>
<p>You can now create connections using the <code>json</code> serialization format.</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">airflow connections add <span class="s1">'my_prod_db'</span> <span class="se">\
-</span><span class="se"></span> --conn-json <span class="s1">'{
-</span><span class="s1"> "conn_type": "my-conn-type",
-</span><span class="s1"> "login": "my-login",
-</span><span class="s1"> "password": "my-password",
-</span><span class="s1"> "host": "my-host",
-</span><span class="s1"> "port": 1234,
-</span><span class="s1"> "schema": "my-schema",
-</span><span class="s1"> "extra": {
-</span><span class="s1"> "param1": "val1",
-</span><span class="s1"> "param2": "val2"
-</span><span class="s1"> }
-</span><span class="s1"> }'</span>
-</code></pre></div><p>You can also use <code>json</code> serialization format when setting the connection in environment variables.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">airflow connections add <span class="s1">'my_prod_db'</span> <span class="se">\
+</span></span></span><span class="line"><span class="cl"><span class="se"></span> --conn-json <span class="s1">'{
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "conn_type": "my-conn-type",
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "login": "my-login",
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "password": "my-password",
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "host": "my-host",
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "port": 1234,
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "schema": "my-schema",
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "extra": {
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "param1": "val1",
+</span></span></span><span class="line"><span class="cl"><span class="s1"> "param2": "val2"
+</span></span></span><span class="line"><span class="cl"><span class="s1"> }
+</span></span></span><span class="line"><span class="cl"><span class="s1"> }'</span>
+</span></span></code></pre></div><p>You can also use <code>json</code> serialization format when setting the connection in environment variables.</p>
<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/howto/connection.html">JSON serialization for connections</a></p>
<h2 id="airflow-db-downgrade-and-offline-generation-of-sql-scripts">Airflow <code>db downgrade</code> and Offline generation of SQL scripts</h2>
<p>Airflow 2.3.0 introduced a new command <code>airflow db downgrade</code> that will downgrade the database to your chosen version.</p>
@@ -504,18 +512,18 @@ to create a list of tasks, here you can create the same tasks without having to
<h2 id="reuse-of-decorated-tasks">Reuse of decorated tasks</h2>
<p>You can now reuse decorated tasks across your dag files. A decorated task has an <code>override</code> method that allows you to override it’s arguments.</p>
<p>Here’s an example:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="nd">@task</span>
-<span class="k">def</span> <span class="nf">add_task</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
- <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Task args: x=</span><span class="si">{</span><span class="n">x</span><span class="si">}</span><span class="s2">, y=</span><span class="si">{</span><span class="n">y</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
-
-
-<span class="nd">@dag</span><span class="p">(</span><span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2022</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
-<span class="k">def</span> <span class="nf">mydag</span><span class="p">():</span>
- <span class="n">start</span> <span class="o">=</span> <span class="n">add_task</span><span class="o">.</span><span class="n">override</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s2">"start"</span><span class="p">)(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
- <span class="n">start</span> <span class="o">>></span> <span class="n">add_task</span><span class="o">.</span><span class="n">override</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="sa">f</span><span class="s2">"add_start_</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">)(</span><span class="n">start</span><span class="p">,</span> <span class="n [...]
-</code></pre></div><p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/tutorial_taskflow_api.html#reusing-a-decorated-task">Reuse of decorated DAGs</a></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="nd">@task</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">add_task</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Task args: x=</span><span class="si">{</span><span class="n">x</span><span class="si">}</span><span class="s2">, y=</span><span class="si">{</span><span class="n">y</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@dag</span><span class="p">(</span><span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2022</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">mydag</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="n">start</span> <span class="o">=</span> <span class="n">add_task</span><span class="o">.</span><span class="n">override</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s2">"start"</span><span class="p">)(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="n">start</span> <span class="o">>></span> <span class="n">add_task</span><span class="o">.</span><span class="n">override</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="sa">f</span><span class="s2">"add_start_</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">)(</span><span class="n">s [...]
+</span></span></code></pre></div><p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/tutorial_taskflow_api.html#reusing-a-decorated-task">Reuse of decorated DAGs</a></p>
<h2 id="other-small-features">Other small features</h2>
<p>This isn’t a comprehensive list, but some noteworthy or interesting small features include:</p>
<ul>
@@ -682,7 +690,7 @@ Sade’s “Smooth Operator”. Enjoy!</li>
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.3.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.3.0>
@@ -834,13 +842,13 @@ Sade’s “Smooth Operator”. Enjoy!</li>
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.4.0/index.html b/blog/airflow-2.4.0/index.html
index fb8ade93ed..a0d27e1aac 100644
--- a/blog/airflow-2.4.0/index.html
+++ b/blog/airflow-2.4.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 2.4.0: That Data Aware Release | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.4.0: That Data Aware Release" />
-<meta property="og:description" content="We're proud to announce that Apache Airflow 2.4.0 has been released with many exciting improvements." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.4.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2022-09-19T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 2.4.0: That Data Aware Release | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.4.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 2.4.0: That Data Aware Release">
+ <meta property="og:description" content="We're proud to announce that Apache Airflow 2.4.0 has been released with many exciting improvements.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2022-09-19T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 2.4.0: That Data Aware Release">
-<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.4.0 has been released with many exciting improvements."><meta itemprop="datePublished" content="2022-09-19T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="848"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 2.4.0: That Data Aware Release">
+ <meta itemprop="description" content="We're proud to announce that Apache Airflow 2.4.0 has been released with many exciting improvements.">
+ <meta itemprop="datePublished" content="2022-09-19T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="849">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -448,36 +456,36 @@
<p>This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.</p>
<p>What does this mean, exactly? This is a great new feature that lets DAG authors create smaller, more self-contained DAGs, which chain together into a larger data-based workflow. If you are currently using <code>ExternalTaskSensor</code> or <code>TriggerDagRunOperator</code> you should take a look at datasets – in most cases you can replace them with something that will speed up the scheduling!</p>
<p>But enough talking, lets have a short example. First lets write a simple DAG with a task called <code>my_task</code> that produces a dataset called <code>my-dataset</code>:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">Dataset</span>
-
-
-<span class="n">dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="p">(</span><span class="n">uri</span><span class="o">=</span><span class="s1">'my-dataset'</span><span class="p">)</span>
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s1">'producer'</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
- <span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">dataset</span><span class="p">])</span>
- <span class="k">def</span> <span class="nf">my_task</span><span class="p">():</span>
- <span class="o">...</span>
-</code></pre></div><p>Datasets are defined by a URI. Now, we can create a second DAG (<code>consumer</code>) that gets scheduled whenever this dataset changes:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">
-<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">Dataset</span>
-
-
-<span class="n">dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="p">(</span><span class="n">uri</span><span class="o">=</span><span class="s1">'my-dataset'</span><span class="p">)</span>
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s1">'dataset-consumer'</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="p">[</span><span class="n">dataset</span><span class="p">]):</span>
- <span class="o">...</span>
-</code></pre></div><p>With these two DAGs, the instant <code>my_task</code> finishes, Airflow will create the DAG run for the <code>dataset-consumer</code> workflow.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">Dataset</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="p">(</span><span class="n">uri</span><span class="o">=</span><span class="s1">'my-dataset'</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s1">'producer'</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="nd">@task</span><span class="p">(</span><span class="n">outlets</span><span class="o">=</span><span class="p">[</span><span class="n">dataset</span><span class="p">])</span>
+</span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">my_task</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><p>Datasets are defined by a URI. Now, we can create a second DAG (<code>consumer</code>) that gets scheduled whenever this dataset changes:</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">Dataset</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">dataset</span> <span class="o">=</span> <span class="n">Dataset</span><span class="p">(</span><span class="n">uri</span><span class="o">=</span><span class="s1">'my-dataset'</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s1">'dataset-consumer'</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="p">[</span><span class="n">dataset</span><span class="p">]):</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><p>With these two DAGs, the instant <code>my_task</code> finishes, Airflow will create the DAG run for the <code>dataset-consumer</code> workflow.</p>
<p>We know that what exists right now won’t fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation.</p>
<p>Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in the future - and it’s part of our goal to have smaller releases to get new features in your hands sooner!</p>
<p>For more information on datasets, see the <a href="https://airflow.apache.org/docs/apache-airflow/2.4.0/concepts/datasets.html">documentation on Data-aware scheduling</a>. That includes details on how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what a dataset is (hint: don’t include “date partitions” in a dataset, it’s higher level than that).</p>
<h2 id="easier-management-of-conflicting-python-dependencies-using-the-new-externalpythonoperator">Easier management of conflicting python dependencies using the new ExternalPythonOperator</h2>
<p>As much as we wish all python libraries could be used happily together that sadly isn’t the world we live in, and sometimes there are conflicts when trying to install multiple python libraries in an Airflow install – right now we hear this a lot with <code>dbt-core</code>.</p>
<p>To make this easier we have introduced <code>@task.external_python</code> (and the matching <code>ExternalPythonOperator</code>) that lets you run an python function as an Airflow task in a pre-configured virtual env, or even a whole different python version. For example:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="nd">@task</span><span class="o">.</span><span class="n">external_python</span><span class="p">(</span><span class="n">python</span><span class="o">=</span><span class="s1">'/opt/venvs/task_deps/bin/python'</span><span class="p">)</span>
-<span class="k">def</span> <span class="nf">my_task</span><span class="p">(</span><span class="n">data_interval_start</span><span class="p">,</span> <span class="n">data_interval_env</span><span class="p">)</span>
- <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Looking at data between </span><span class="si">{</span><span class="n">data_interval_start</span><span class="si">}</span><span class="s1"> and </span><span class="si">{</span><span class="n">data_interval_end</span><span class="si">}</span><span class="s1">'</span><span class="p">)</span>
- <span class="o">...</span>
-</code></pre></div><p>There are a few subtlties as to what you need installed in the virtual env depending on which context variables you access, so be sure to read the <a href="http://airflow.apache.org/docs/apache-airflow/2.4.0/howto/operator/python.html#externalpythonoperator">how-to on using the ExternalPythonOperator</a></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="nd">@task.external_python</span><span class="p">(</span><span class="n">python</span><span class="o">=</span><span class="s1">'/opt/venvs/task_deps/bin/python'</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">my_task</span><span class="p">(</span><span class="n">data_interval_start</span><span class="p">,</span> <span class="n">data_interval_env</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Looking at data between </span><span class="si">{</span><span class="n">data_interval_start</span><span class="si">}</span><span class="s1"> and </span><span class="si">{</span><span class="n">data_interval_end</span><span class="si">}</span><span class="s1">'</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><p>There are a few subtlties as to what you need installed in the virtual env depending on which context variables you access, so be sure to read the <a href="http://airflow.apache.org/docs/apache-airflow/2.4.0/howto/operator/python.html#externalpythonoperator">how-to on using the ExternalPythonOperator</a></p>
<h2 id="more-improvements-to-dynamic-task-mapping-aip-42">More improvements to Dynamic Task Mapping (AIP-42)</h2>
<p>You asked, we listened. Dynamic task mapping now includes support for:</p>
<ul>
@@ -488,34 +496,34 @@
<p>For more information on dynamic task mapping, see the new sections of the doc on <a href="https://airflow.apache.org/docs/apache-airflow/2.4.0/concepts/dynamic-task-mapping.html#transforming-mapped-data">Transforming Mapped Data</a>, <a href="https://airflow.apache.org/docs/apache-airflow/2.4.0/concepts/dynamic-task-mapping.html#combining-upstream-data-aka-zipping">Combining upstream data (aka “zipping”)</a>, and <a href="https://airflow.apache.org/docs/apache-airflow/2.4. [...]
<h2 id="auto-register-dags-used-in-a-context-manager-no-more-as-dag-needed">Auto-register DAGs used in a context manager (no more <code>as dag:</code> needed)</h2>
<p>This one is a small quality of life improvement, and I don’t want to admit how many times I forgot the <code>as dag:</code>, or worse, had <code>as dag:</code> repeated.</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"example"</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
- <span class="o">...</span>
-
-
-<span class="nd">@dag</span>
-<span class="k">def</span> <span class="nf">dag_maker</span><span class="p">():</span>
- <span class="o">...</span>
-
-
-<span class="n">dag2</span> <span class="o">=</span> <span class="n">dag_maker</span><span class="p">()</span>
-</code></pre></div><p>can become</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"example"</span><span class="p">):</span>
- <span class="o">...</span>
-
-
-<span class="nd">@dag</span>
-<span class="k">def</span> <span class="nf">my_dag</span><span class="p">():</span>
- <span class="o">...</span>
-
-
-<span class="n">my_dag</span><span class="p">()</span>
-</code></pre></div><p>If you want to disable the behaviour for any reason, set <code>auto_register=False</code> on the DAG:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="c1"># This dag will not be picked up by Airflow as it's not assigned to a variable</span>
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"example"</span><span class="p">,</span> <span class="n">auto_register</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="o">...</span>
-</code></pre></div><h2 id="additional-improvements">Additional improvements</h2>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"example"</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@dag</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">dag_maker</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">dag2</span> <span class="o">=</span> <span class="n">dag_maker</span><span class="p">()</span>
+</span></span></code></pre></div><p>can become</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"example"</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@dag</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">my_dag</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">my_dag</span><span class="p">()</span>
+</span></span></code></pre></div><p>If you want to disable the behaviour for any reason, set <code>auto_register=False</code> on the DAG:</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># This dag will not be picked up by Airflow as it's not assigned to a variable</span>
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"example"</span><span class="p">,</span> <span class="n">auto_register</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><h2 id="additional-improvements">Additional improvements</h2>
<p>With over 650 commits the <a href="https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html#airflow-2-4-0-2022-09-19">full list of features, fixes and changes</a> is too big to go in to here (check out the release notes for a full list), but some noteworthy or interesting small features include:</p>
<ul>
<li>Auto-refresh on the home page</li>
@@ -685,7 +693,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.4.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.4.0>
@@ -837,13 +845,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.5.0/index.html b/blog/airflow-2.5.0/index.html
index 2cd2497dda..1299cc6ef8 100644
--- a/blog/airflow-2.5.0/index.html
+++ b/blog/airflow-2.5.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 2.5.0: Tick-Tock | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.5.0: Tick-Tock" />
-<meta property="og:description" content="We're proud to announce that Apache Airflow 2.5.0 has been released with many quality of life changes." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.5.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2022-12-02T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 2.5.0: Tick-Tock | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.5.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 2.5.0: Tick-Tock">
+ <meta property="og:description" content="We're proud to announce that Apache Airflow 2.5.0 has been released with many quality of life changes.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2022-12-02T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 2.5.0: Tick-Tock">
-<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.5.0 has been released with many quality of life changes."><meta itemprop="datePublished" content="2022-12-02T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="392"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 2.5.0: Tick-Tock">
+ <meta itemprop="description" content="We're proud to announce that Apache Airflow 2.5.0 has been released with many quality of life changes.">
+ <meta itemprop="datePublished" content="2022-12-02T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="392">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -616,7 +624,7 @@ c. Everything runs in one process, so you can put a breakpoint in your IDE, and
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.5.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.5.0>
@@ -768,13 +776,13 @@ c. Everything runs in one process, so you can put a breakpoint in your IDE, and
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.6.0/index.html b/blog/airflow-2.6.0/index.html
index 9ce9676065..ec639d1f24 100644
--- a/blog/airflow-2.6.0/index.html
+++ b/blog/airflow-2.6.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>what's new in Apache Airflow 2.6.0 | Apache Airflow</title><meta property="og:title" content="what's new in Apache Airflow 2.6.0" />
-<meta property="og:description" content="Apache Airflow 2.6.0 has been released!" />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.6.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2023-04-30T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>what's new in Apache Airflow 2.6.0 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.6.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="what's new in Apache Airflow 2.6.0">
+ <meta property="og:description" content="Apache Airflow 2.6.0 has been released!">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2023-04-30T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="what's new in Apache Airflow 2.6.0">
-<meta itemprop="description" content="Apache Airflow 2.6.0 has been released!"><meta itemprop="datePublished" content="2023-04-30T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="535"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="what's new in Apache Airflow 2.6.0">
+ <meta itemprop="description" content="Apache Airflow 2.6.0 has been released!">
+ <meta itemprop="datePublished" content="2023-04-30T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="535">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -448,17 +456,17 @@
<p>As the changelog is quite large, the following are some notable new features that shipped in this release.</p>
<h2 id="trigger-logs-can-now-be-viewed-in-webserver">Trigger logs can now be viewed in webserver</h2>
<p>Trigger logs have now been added to task logs. They appear right alongside the rest of the logs from your task.</p>
-<p><img src="trigger_logging.png" alt="Trigger logs shown in task log"></p>
+<p><img alt="Trigger logs shown in task log" src="/blog/airflow-2.6.0/trigger_logging.png"></p>
<p>Adding this feature required changes across the entire Airflow logging stack, so be sure to update your providers if you are using remote logging.</p>
<h2 id="grid-view-improvements">Grid view improvements</h2>
<p>The grid view has received a number of minor improvements in this release.</p>
<p>Most notably, there is now a graph tab in the grid view. This offers a more integrated graph representation of the DAG, where choosing a task in either the grid or graph will highlight the same task in both views.</p>
-<p><img src="graph.png" alt="The new graph view"></p>
+<p><img alt="The new graph view" src="/blog/airflow-2.6.0/graph.png"></p>
<p>You can also filter upstream and downstream from a single task. For example, in the screenshot above, <code>describe_integrity</code> is the selected task. If you choose to filter downstream, this is the result:</p>
-<p><img src="filter_downstream.png" alt="The new graph view can be filtered to show downstream tasks only"></p>
+<p><img alt="The new graph view can be filtered to show downstream tasks only" src="/blog/airflow-2.6.0/filter_downstream.png"></p>
<h2 id="trigger-ui-based-on-dag-level-params">Trigger UI based on DAG level params</h2>
<p>A user-friendly form is now shown to users triggering runs for DAGs with DAG level params.</p>
-<p><img src="trigger_dag_form.png" alt="Form shown for params in UI when triggering a DAG"></p>
+<p><img alt="Form shown for params in UI when triggering a DAG" src="/blog/airflow-2.6.0/trigger_dag_form.png"></p>
<p>See the <a href="https://airflow.apache.org/docs/apache-airflow/2.6.0/core-concepts/params.html#use-params-to-provide-a-trigger-ui-form">Params docs</a> for more details.</p>
<h2 id="consolidation-of-handling-stuck-queued-tasks">Consolidation of handling stuck queued tasks</h2>
<p>Airflow now has a single configuration, <code>[scheduler] task_queued_timeout</code>, to handle tasks that get stuck in queued for too long. With a simpler implementation than the outgoing code handling these tasks, tasks stuck in queued will no longer slip through the cracks and stay stuck.</p>
@@ -468,18 +476,18 @@
<p>See the <a href="https://airflow.apache.org/docs/apache-airflow/2.6.0/administration-and-deployment/cluster-policies.html">cluster policy docs</a> for more details.</p>
<h2 id="notification-support-added">Notification support added</h2>
<p>The notifications framework allows you to send messages to external systems when a task instance/DAG run changes state. For example, you can easily post a message to Slack</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
- <span class="err">“</span><span class="n">slack_notifier_example</span><span class="err">”</span><span class="p">,</span>
- <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2023</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
- <span class="n">on_success_callback</span><span class="o">=</span><span class="p">[</span>
- <span class="n">send_slack_notification</span><span class="p">(</span>
- <span class="n">text</span><span class="o">=</span><span class="s2">"The DAG {{ dag.dag_id }} succeeded"</span><span class="p">,</span>
- <span class="n">channel</span><span class="o">=</span><span class="s2">"#general"</span><span class="p">,</span>
- <span class="n">username</span><span class="o">=</span><span class="s2">"Airflow"</span><span class="p">,</span>
- <span class="p">)</span>
- <span class="p">],</span>
-<span class="p">):</span>
-</code></pre></div><p>As of today, Slack is the only system supported out of the box. However, watch this space as more integrations will be added soon.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="err">“</span><span class="n">slack_notifier_example</span><span class="err">”</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2023</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
+</span></span><span class="line"><span class="cl"> <span class="n">on_success_callback</span><span class="o">=</span><span class="p">[</span>
+</span></span><span class="line"><span class="cl"> <span class="n">send_slack_notification</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">text</span><span class="o">=</span><span class="s2">"The DAG {{ dag.dag_id }} succeeded"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">channel</span><span class="o">=</span><span class="s2">"#general"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">username</span><span class="o">=</span><span class="s2">"Airflow"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="p">],</span>
+</span></span><span class="line"><span class="cl"><span class="p">):</span>
+</span></span></code></pre></div><p>As of today, Slack is the only system supported out of the box. However, watch this space as more integrations will be added soon.</p>
<p>You can also create notifiers for your own use, refer to the <a href="https://airflow.apache.org/docs/apache-airflow/2.6.0/howto/notifications.html">notifier how-to docs</a> for more details.</p>
<h2 id="thanks-to-the-contributors">Thanks to the contributors</h2>
<p>Thanks to everyone who contributed to this release, including Andrey Anshin, Ash Berlin-Taylor, Brent Bovenzi, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk, Jed Cunningham, Josh Fell, Michael Petro, Niko Oliveira, Pierre Jeambrun, Tzu-ping Chung, Victor Chiapaikeo, and over 120 others!</p>
@@ -640,7 +648,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.6.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.6.0>
@@ -792,13 +800,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.7.0/index.html b/blog/airflow-2.7.0/index.html
index a5492ac96e..c1b9be4010 100644
--- a/blog/airflow-2.7.0/index.html
+++ b/blog/airflow-2.7.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 2.7.0 is here | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.7.0 is here" />
-<meta property="og:description" content="Apache Airflow 2.7.0 has been released!" />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.7.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2023-08-18T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 2.7.0 is here | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.7.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 2.7.0 is here">
+ <meta property="og:description" content="Apache Airflow 2.7.0 has been released!">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2023-08-18T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 2.7.0 is here">
-<meta itemprop="description" content="Apache Airflow 2.7.0 has been released!"><meta itemprop="datePublished" content="2023-08-18T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="816"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 2.7.0 is here">
+ <meta itemprop="description" content="Apache Airflow 2.7.0 has been released!">
+ <meta itemprop="datePublished" content="2023-08-18T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="816">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -457,10 +465,10 @@
<p>You can read more about setup and teardown in the <a href="/blog/introducing_setup_teardown/">Introducing Setup and Teardown tasks blog post</a>, or in the <a href="https://airflow.apache.org/docs/apache-airflow/2.7.0/howto/setup-and-teardown.html">setup and teardown docs</a>.</p>
<h2 id="cluster-activity-ui">Cluster Activity UI</h2>
<p>There is a new top level page in Airflow, the Cluster Activity page. This gives an overview of the cluster, including component health, dag and task state counts, and more!</p>
-<p><img src="cluster_activity.png" alt="New cluster activity page"></p>
+<p><img alt="New cluster activity page" src="/blog/airflow-2.7.0/cluster_activity.png"></p>
<h2 id="graph-and-gantt-views-moved-into-the-grid-view-ui">Graph and gantt views moved into the Grid view UI</h2>
<p>The graph and gantt views have been rewritten and moved into the now familiar grid view. This makes it easier to jump between task details, logs, graph, and gantt views without losing your place in a complicated DAG.</p>
-<p><img src="graph_in_grid.png" alt="Graph in grid view"></p>
+<p><img alt="Graph in grid view" src="/blog/airflow-2.7.0/graph_in_grid.png"></p>
<h2 id="enable-deferrable-mode-for-all-deferable-tasks-with-1-config-setting">Enable deferrable mode for all deferable tasks with 1 config setting</h2>
<p>Airflow 2.7.0 comes with a new config option, <code>default_deferrable</code>, which allows admins to enable deferrable mode for all deferrable tasks without requiring any DAG modifications. Simply set it in your config and enjoy async tasks!</p>
<h2 id="openlineage-built-in-integration">OpenLineage built-in integration</h2>
@@ -643,7 +651,7 @@ The following providers have been moved and require certain minimum providers ve
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.7.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.7.0>
@@ -795,13 +803,13 @@ The following providers have been moved and require certain minimum providers ve
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.8.0/index.html b/blog/airflow-2.8.0/index.html
index 9561bb19ed..0909edf068 100644
--- a/blog/airflow-2.8.0/index.html
+++ b/blog/airflow-2.8.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 2.8.0 is here | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.8.0 is here" />
-<meta property="og:description" content="Introducing Apache Airflow 2.8.0: Enhanced with New Features and Significant Improvements" />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.8.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2023-12-15T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 2.8.0 is here | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.8.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 2.8.0 is here">
+ <meta property="og:description" content="Introducing Apache Airflow 2.8.0: Enhanced with New Features and Significant Improvements">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2023-12-15T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 2.8.0 is here">
-<meta itemprop="description" content="Introducing Apache Airflow 2.8.0: Enhanced with New Features and Significant Improvements"><meta itemprop="datePublished" content="2023-12-15T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="730"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 2.8.0 is here">
+ <meta itemprop="description" content="Introducing Apache Airflow 2.8.0: Enhanced with New Features and Significant Improvements">
+ <meta itemprop="datePublished" content="2023-12-15T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="730">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -449,24 +457,24 @@
<p>Airflow now offers a generic abstraction layer over various object stores like S3, GCS, and Azure Blob Storage, enabling the use of different storage systems in DAGs without code modification.</p>
<p>In addition, it allows you to use most of the standard Python modules, like shutil, that can work with file-like objects.</p>
<p>Here is an example of how to use the new feature to open a file:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">airflow.io.path</span> <span class="kn">import</span> <span class="n">ObjectStoragePath</span>
-
-<span class="n">base</span> <span class="o">=</span> <span class="n">ObjectStoragePath</span><span class="p">(</span><span class="s2">"s3://my-bucket/"</span><span class="p">,</span> <span class="n">conn_id</span><span class="o">=</span><span class="s2">"aws_default"</span><span class="p">)</span> <span class="c1"># conn_id is optional</span>
-
-<span class="nd">@task</span>
-<span class="k">def</span> <span class="nf">read_file</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="n">ObjectStoragePath</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
- <span class="k">with</span> <span class="n">path</span><span class="o">.</span><span class="n">open</span><span class="p">()</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
-</code></pre></div><p>The above example is just the tip of the iceberg. The new feature allows you to configure an alternative backend for a scheme or protocol.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.io.path</span> <span class="kn">import</span> <span class="n">ObjectStoragePath</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">base</span> <span class="o">=</span> <span class="n">ObjectStoragePath</span><span class="p">(</span><span class="s2">"s3://my-bucket/"</span><span class="p">,</span> <span class="n">conn_id</span><span class="o">=</span><span class="s2">"aws_default"</span><span class="p">)</span> <span class="c1"># conn_id is optional</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="nd">@task</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">read_file</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="n">ObjectStoragePath</span><span class="p">)</span> <span class="o">-></span> <span class="nb">str</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="k">with</span> <span class="n">path</span><span class="o">.</span><span class="n">open</span><span class="p">()</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
+</span></span></code></pre></div><p>The above example is just the tip of the iceberg. The new feature allows you to configure an alternative backend for a scheme or protocol.</p>
<p>Here is an example of how to configure a custom backend for the <code>dbfs</code> scheme:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">airflow.io.path</span> <span class="kn">import</span> <span class="n">ObjectStoragePath</span>
-<span class="kn">from</span> <span class="nn">airflow.io.store</span> <span class="kn">import</span> <span class="n">attach</span>
-
-<span class="kn">from</span> <span class="nn">fsspec.implementations.dbfs</span> <span class="kn">import</span> <span class="n">DBFSFileSystem</span>
-
-<span class="n">attach</span><span class="p">(</span><span class="n">protocol</span><span class="o">=</span><span class="s2">"dbfs"</span><span class="p">,</span> <span class="n">fs</span><span class="o">=</span><span class="n">DBFSFileSystem</span><span class="p">(</span><span class="n">instance</span><span class="o">=</span><span class="s2">"myinstance"</span><span class="p">,</span> <span class="n">token</span><span class="o">=</span><span class="s2">"mytoken"< [...]
-<span class="n">base</span> <span class="o">=</span> <span class="n">ObjectStoragePath</span><span class="p">(</span><span class="s2">"dbfs://my-location/"</span><span class="p">)</span>
-</code></pre></div><p>For more information: <a href="https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html">Airflow Object Storage</a></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.io.path</span> <span class="kn">import</span> <span class="n">ObjectStoragePath</span>
+</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.io.store</span> <span class="kn">import</span> <span class="n">attach</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fsspec.implementations.dbfs</span> <span class="kn">import</span> <span class="n">DBFSFileSystem</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">attach</span><span class="p">(</span><span class="n">protocol</span><span class="o">=</span><span class="s2">"dbfs"</span><span class="p">,</span> <span class="n">fs</span><span class="o">=</span><span class="n">DBFSFileSystem</span><span class="p">(</span><span class="n">instance</span><span class="o">=</span><span class="s2">"myinstance"</span><span class="p">,</span> <span class="n">token</span><span cla [...]
+</span></span><span class="line"><span class="cl"><span class="n">base</span> <span class="o">=</span> <span class="n">ObjectStoragePath</span><span class="p">(</span><span class="s2">"dbfs://my-location/"</span><span class="p">)</span>
+</span></span></code></pre></div><p>For more information: <a href="https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html">Airflow Object Storage</a></p>
<p>The support for a specific object storage system depends on the installed providers,
with out-of-the-box support for the file scheme.</p>
<h2 id="ship-logs-from-other-components-to-task-logs">Ship logs from other components to Task logs</h2>
@@ -489,10 +497,10 @@ can be achieved by setting the <code>venv_cache_path</code> to a file system fol
<p>There are a number of improvements to the Web UI in this release, including:</p>
<h2 id="add-multiselect-to-run-state-in-grid-view">Add multiselect to run state in grid view:</h2>
<p>The grid view now supports multiselect for run states. This allows you to select multiple states to filter the dag runs shown in the grid view.</p>
-<p><img src="multiselect-states.png" alt="Multiselect on the run state"></p>
+<p><img alt="Multiselect on the run state" src="/blog/airflow-2.8.0/multiselect-states.png"></p>
<h2 id="improved-visibility-of-task-status-in-the-graph-view">Improved visibility of task status in the Graph view</h2>
<p>You can now see the status of a task in the graph view through the border color of the task. This makes it easier to see the status of a task at a glance.</p>
-<p><img src="task_status_visibility.png" alt="Task status visibility"></p>
+<p><img alt="Task status visibility" src="/blog/airflow-2.8.0/task_status_visibility.png"></p>
<h2 id="raw-html-code-in-dag-docs-and-dag-params-descriptions-is-disabled-by-default">Raw HTML code in DAG docs and DAG params descriptions is disabled by default</h2>
<p>As part of our continuous quest to make airflow more secure by default, we have disabled raw HTML code in DAG docs and DAG params descriptions by default.
We care for your security, and “secure by default” is one of the things we follow strongly.</p>
@@ -661,7 +669,7 @@ We care for your security, and “secure by default” is one of the thi
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.8.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.8.0>
@@ -813,13 +821,13 @@ We care for your security, and “secure by default” is one of the thi
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-2.9.0/index.html b/blog/airflow-2.9.0/index.html
index e9b2d49b7c..b79a63bb44 100644
--- a/blog/airflow-2.9.0/index.html
+++ b/blog/airflow-2.9.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 2.9.0: Dataset and UI Improvements | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.9.0: Dataset and UI Improvements" />
-<meta property="og:description" content="Apache Airflow 2.9.0 is here! Lots of exciting new Dataset and UI features/improvements this time around." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.9.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2024-04-08T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 2.9.0: Dataset and UI Improvements | Apache Airflow</title><meta property="og:url" content="/blog/airflow-2.9.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 2.9.0: Dataset and UI Improvements">
+ <meta property="og:description" content="Apache Airflow 2.9.0 is here! Lots of exciting new Dataset and UI features/improvements this time around.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2024-04-08T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Apache Airflow 2.9.0: Dataset and UI Improvements">
-<meta itemprop="description" content="Apache Airflow 2.9.0 is here! Lots of exciting new Dataset and UI features/improvements this time around."><meta itemprop="datePublished" content="2024-04-08T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="1041"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 2.9.0: Dataset and UI Improvements">
+ <meta itemprop="description" content="Apache Airflow 2.9.0 is here! Lots of exciting new Dataset and UI features/improvements this time around.">
+ <meta itemprop="datePublished" content="2024-04-08T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="1042">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -450,46 +458,45 @@
<h3 id="logical-operators-and-conditional-expressions-for-dag-scheduling">Logical operators and conditional expressions for DAG scheduling</h3>
<p>When Datasets were added in Airflow 2.4, DAGs only had scheduling support for logical AND combinations of Datasets. Simply, you could schedule against more than one Dataset, but a DAG run would only be created once all the Datasets were updated after the last run. Now in Airflow 2.9, we support logical OR and even arbitrary combinations of AND and OR.</p>
<p>As an example, you can schedule a DAG whenever <code>dataset_1</code> or <code>dataset_2</code> are updated :</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">schedule</span><span class="o">=</span><span class="p">(</span><span class="n">dataset_1</span> <span class="o">|</span> <span class="n">dataset_2</span><span class="p">),</span> <span class="o">...</span><span class="p">):</span>
- <span class="o">...</span>
-</code></pre></div><p>You can have arbitrary combinations:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">schedule</span><span class="o">=</span><span class="p">((</span><span class="n">dataset_1</span> <span class="o">|</span> <span class="n">dataset_2</span><span class="p">)</span> <span class="o">&</span> <span class="n">dataset_3</span><span class="p">),</span> <span class="o">...</span>< [...]
- <span class="o">...</span>
-</code></pre></div><p>You can read more about this new functionality in the <a href="https://airflow.apache.org/docs/apache-airflow/2.9.0/authoring-and-scheduling/datasets.html#advanced-dataset-scheduling-with-conditional-expressions">data-aware scheduling docs</a>.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">schedule</span><span class="o">=</span><span class="p">(</span><span class="n">dataset_1</span> <span class="o">|</span> <span class="n">dataset_2</span><span class="p">),</span> <span class="o">...</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><p>You can have arbitrary combinations:</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">schedule</span><span class="o">=</span><span class="p">((</span><span class="n">dataset_1</span> <span class="o">|</span> <span class="n">dataset_2</span><span class="p">)</span> <span class="o">&</span> <span class="n">dataset_3</span><span class="p">) [...]
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><p>You can read more about this new functionality in the <a href="https://airflow.apache.org/docs/apache-airflow/2.9.0/authoring-and-scheduling/datasets.html#advanced-dataset-scheduling-with-conditional-expressions">data-aware scheduling docs</a>.</p>
<h3 id="combining-dataset-and-time-based-schedules">Combining Dataset and Time-Based Schedules</h3>
<p>Airflow 2.9 comes with a new timetable, <code>DatasetOrTimeSchedule</code>, that allows you to schedule DAGs based on both dataset events and a timetable. Now you have the best of both worlds.</p>
<p>For example, to run whenever <code>dataset_1</code> updates and at midnight UTC:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
- <span class="n">schedule</span><span class="o">=</span><span class="n">DatasetOrTimeSchedule</span><span class="p">(</span>
- <span class="n">timetable</span><span class="o">=</span><span class="n">CronTriggerTimetable</span><span class="p">(</span><span class="s2">"0 0 * * *"</span><span class="p">,</span> <span class="n">timezone</span><span class="o">=</span><span class="s2">"UTC"</span><span class="p">),</span>
- <span class="n">datasets</span><span class="o">=</span><span class="p">[</span><span class="n">dag1_dataset</span><span class="p">],</span>
- <span class="p">),</span>
- <span class="o">...</span>
-<span class="p">):</span>
- <span class="o">...</span>
-
-</code></pre></div><h3 id="dataset-event-rest-api-endpoints">Dataset Event REST API endpoints</h3>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">schedule</span><span class="o">=</span><span class="n">DatasetOrTimeSchedule</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">timetable</span><span class="o">=</span><span class="n">CronTriggerTimetable</span><span class="p">(</span><span class="s2">"0 0 * * *"</span><span class="p">,</span> <span class="n">timezone</span><span class="o">=</span><span class="s2">"UTC"</span><span class="p">),</span>
+</span></span><span class="line"><span class="cl"> <span class="n">datasets</span><span class="o">=</span><span class="p">[</span><span class="n">dag1_dataset</span><span class="p">],</span>
+</span></span><span class="line"><span class="cl"> <span class="p">),</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span><span class="line"><span class="cl"><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="o">...</span>
+</span></span></code></pre></div><h3 id="dataset-event-rest-api-endpoints">Dataset Event REST API endpoints</h3>
<p>New REST API endpoints have been introduced for creating, listing, and deleting dataset events. This makes it possible for external systems to notify Airflow about dataset updates and unlocks management of event queues for more sophisticated use cases.</p>
<p>See the <a href="https://airflow.apache.org/docs/apache-airflow/2.9.0/stable-rest-api-ref.html#tag/Dataset">Dataset API docs</a> for more details.</p>
<h3 id="dataset-ui-enhancements">Dataset UI Enhancements</h3>
<p>The DAG’s graph view has been enhanced to display both the datasets it is scheduled on and those in the task outlets, providing a comprehensive overview of the datasets consumed and produced by the DAG.</p>
-<p><img src="datasets-in-graph.png" alt="Datasets in the graph view"></p>
+<p><img alt="Datasets in the graph view" src="/blog/airflow-2.9.0/datasets-in-graph.png"></p>
<p>The main datasets view now allows you to filter for both DAGs and datasets:</p>
-<p><img src="dataset-view-filtering.png" alt="Dataset view filtering"></p>
+<p><img alt="Dataset view filtering" src="/blog/airflow-2.9.0/dataset-view-filtering.png"></p>
<p>When viewing a Dataset, you can now create a manual dataset event through the UI by clicking the play button shown in the top right here:</p>
-<p><img src="create-manual-dataset-event.png" alt="Creating manual Dataset event"></p>
+<p><img alt="Creating manual Dataset event" src="/blog/airflow-2.9.0/create-manual-dataset-event.png"></p>
<h2 id="custom-names-for-dynamic-task-mapping">Custom names for Dynamic Task Mapping</h2>
<p>Gone are the days of clicking into index numbers and hunting for the dynamically mapped task you wanted to see! This has been a requested feature ever since task mapping was added in Airflow 2.3, and we are happy it’s finally here.</p>
<p>You can provide a <code>map_index_template</code> to mapped operators:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="n">BashOperator</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span>
- <span class="n">task_id</span><span class="o">=</span><span class="s2">"hello"</span><span class="p">,</span>
- <span class="n">bash_command</span><span class="o">=</span><span class="s2">"echo Hello $NAME"</span><span class="p">,</span>
- <span class="n">map_index_template</span><span class="o">=</span><span class="s2">"{{ task.env['NAME'] }}"</span><span class="p">,</span>
-<span class="p">)</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span>
- <span class="n">env</span><span class="o">=</span><span class="p">[{</span><span class="s2">"NAME"</span><span class="p">:</span> <span class="s2">"John"</span><span class="p">},</span> <span class="p">{</span><span class="s2">"NAME"</span><span class="p">:</span> <span class="s2">"Bob"</span><span class="p">},</span> <span class="p">{</span><span class="s2">"NAME"</span><span class="p">:</span> <span class="s2">"Fred"</span><span class [...]
-<span class="p">)</span>
-</code></pre></div><p>That template will be rendered after each task finishes running and will populate the name in the UI:</p>
-<p><img src="dynamic-task-mapping-custom-names.png" alt="Dynamic Task Mapping custom names"></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">BashOperator</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">task_id</span><span class="o">=</span><span class="s2">"hello"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">bash_command</span><span class="o">=</span><span class="s2">"echo Hello $NAME"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">map_index_template</span><span class="o">=</span><span class="s2">"{{ task.env['NAME'] }}"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"><span class="p">)</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">env</span><span class="o">=</span><span class="p">[{</span><span class="s2">"NAME"</span><span class="p">:</span> <span class="s2">"John"</span><span class="p">},</span> <span class="p">{</span><span class="s2">"NAME"</span><span class="p">:</span> <span class="s2">"Bob"</span><span class="p">},</span> <span class="p">{</span><span class="s2">"NAME"</span><span class="p">:</span> [...]
+</span></span><span class="line"><span class="cl"><span class="p">)</span>
+</span></span></code></pre></div><p>That template will be rendered after each task finishes running and will populate the name in the UI:</p>
+<p><img alt="Dynamic Task Mapping custom names" src="/blog/airflow-2.9.0/dynamic-task-mapping-custom-names.png"></p>
<p>More details on this, including a taskflow example, is available in the <a href="https://airflow.apache.org/docs/apache-airflow/2.9.0/authoring-and-scheduling/dynamic-task-mapping.html#named-mapping">dynamic task mapping docs</a>.</p>
<h2 id="object-storage-as-xcom-backend">Object Storage as XCom Backend</h2>
<p>You can now configure Object Storage to be used as an XCom backend, making it much easier to get XCom results into an object store. Deployment managers can configure the object store of their choice, a size threshold to route some results to the Airflow metadata database and some to the object store, and even a compression method to apply before the data is stored.</p>
@@ -505,34 +512,34 @@ xcom_objectstorage_compression = gzip
<h2 id="display-names-for-dags-and-tasks">Display names for DAGs and Tasks</h2>
<p>Get your emojis ready! You can now set a display name for dags and tasks, separate from the <code>dag_id</code> and <code>task_id</code>. This allows you to have localized display names in the UI, or just use a bunch of emojis.</p>
<p>Using <code>dag_display_name</code> and <code>task_display_name</code>, you can break away from the ascii handcuffs:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s2">"not_a_fun_dag_id"</span><span class="p">,</span> <span class="n">dag_display_name</span><span class="o">=</span><span class="s2">"📣 Best DAG ever 🎉"</span><span class="p">,</span> <span class="o">...</span><span class="p">):</span>
- <span class="n">BashOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s2">"some_task"</span><span class="p">,</span> <span class="n">task_display_name</span><span class="o">=</span><span class="s2">"🥳 Fun task!"</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
-</code></pre></div><p><img src="display-names.png" alt="Display names for DAGs and tasks"></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s2">"not_a_fun_dag_id"</span><span class="p">,</span> <span class="n">dag_display_name</span><span class="o">=</span><span class="s2">"📣 Best DAG ever 🎉"</span><span class="p">,</span> <span class="o">...</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="n">BashOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s2">"some_task"</span><span class="p">,</span> <span class="n">task_display_name</span><span class="o">=</span><span class="s2">"🥳 Fun task!"</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
+</span></span></code></pre></div><p><img alt="Display names for DAGs and tasks" src="/blog/airflow-2.9.0/display-names.png"></p>
<h2 id="task-log-grouping">Task log grouping</h2>
<p>Airflow now has support for arbitrary grouping of task logs.</p>
<p>By default, pre-execute and post-execute logs are grouped and collapsed, making it easier to see your task logs:</p>
-<p><img src="pre-post-logs-grouped.png" alt="Pre and post execute logs are grouped"></p>
+<p><img alt="Pre and post execute logs are grouped" src="/blog/airflow-2.9.0/pre-post-logs-grouped.png"></p>
<p>You can also use this feature in your task code to make your logs easier to follow:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="nd">@task</span>
-<span class="k">def</span> <span class="nf">big_hello</span><span class="p">():</span>
- <span class="nb">print</span><span class="p">(</span><span class="s2">"::group::Setup our big Hello"</span><span class="p">)</span>
- <span class="n">greeting</span> <span class="o">=</span> <span class="s2">""</span>
- <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="s2">"Hello Airflow 2.9"</span><span class="p">:</span>
- <span class="n">greeting</span> <span class="o">+=</span> <span class="n">c</span>
- <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Adding </span><span class="si">{</span><span class="n">c</span><span class="si">}</span><span class="s2"> to our greeting. Current greeting: </span><span class="si">{</span><span class="n">greeting</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
- <span class="nb">print</span><span class="p">(</span><span class="s2">"::endgroup::"</span><span class="p">)</span>
- <span class="nb">print</span><span class="p">(</span><span class="n">greeting</span><span class="p">)</span>
-</code></pre></div><p>That custom group is collapsed by default:</p>
-<p><img src="custom-log-grouping.png" alt="Custom log grouping collapsed by default"></p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="nd">@task</span>
+</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">big_hello</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s2">"::group::Setup our big Hello"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="n">greeting</span> <span class="o">=</span> <span class="s2">""</span>
+</span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="s2">"Hello Airflow 2.9"</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="n">greeting</span> <span class="o">+=</span> <span class="n">c</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Adding </span><span class="si">{</span><span class="n">c</span><span class="si">}</span><span class="s2"> to our greeting. Current greeting: </span><span class="si">{</span><span class="n">greeting</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s2">"::endgroup::"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="n">greeting</span><span class="p">)</span>
+</span></span></code></pre></div><p>That custom group is collapsed by default:</p>
+<p><img alt="Custom log grouping collapsed by default" src="/blog/airflow-2.9.0/custom-log-grouping.png"></p>
<p>And it can be expanded if you want to dig into the details:</p>
-<p><img src="custom-log-grouping-expanded.png" alt="Custom log grouping expanded"></p>
+<p><img alt="Custom log grouping expanded" src="/blog/airflow-2.9.0/custom-log-grouping-expanded.png"></p>
<h2 id="ui-modernization">UI Modernization</h2>
<p>In addition to all the UI improvements mentioned above, we have a bunch more improvements in Airflow 2.9!</p>
<p>The rest of the DAG level views have been moved into React and the grid view interface, allowing for a more cohesive experience. This includes the calendar, task duration, run duration (which replaces landing times), and the audit log. These weren’t just “moved”, they each were improved upon as well.</p>
<p>Here is the new run duration view, which replaces landing times. Users can toggle between landing times and simple run duration:</p>
-<p><img src="run-duration.png" alt="Run duration"></p>
+<p><img alt="Run duration" src="/blog/airflow-2.9.0/run-duration.png"></p>
<p>And the new task duration view. Users can toggle queued time on/off and see the median value across the displayed runs:</p>
-<p><img src="task-duration.png" alt="Task duration"></p>
+<p><img alt="Task duration" src="/blog/airflow-2.9.0/task-duration.png"></p>
<h2 id="additional-new-features">Additional new features</h2>
<p>Here are just a few interesting new features since there are too many to list in full:</p>
<ul>
@@ -703,7 +710,7 @@ xcom_objectstorage_compression = gzip
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-2.9.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-2.9.0>
@@ -855,13 +862,13 @@ xcom_objectstorage_compression = gzip
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-3.1.0/index.html b/blog/airflow-3.1.0/index.html
index 263cf3f302..849d1d2e14 100644
--- a/blog/airflow-3.1.0/index.html
+++ b/blog/airflow-3.1.0/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,34 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 3.1.0: Human-Centered Workflows | Apache Airflow</title><meta property="og:title" content="Apache Airflow 3.1.0: Human-Centered Workflows" />
-<meta property="og:description" content="Apache Airflow 3.1.0 introduces Human-in-the-Loop workflows, 17-language internationalization, deadline alerts, and React plugin system for data orchestration teams." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-3.1.0/" /><meta property="og:image" content="/blog/airflow-3.1.0/images/3.1.0.jpg" /><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2025-09-25T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Apache Airflow 3.1.0: Human-Centered Workflows | Apache Airflow</title><meta property="og:url" content="/blog/airflow-3.1.0/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Apache Airflow 3.1.0: Human-Centered Workflows">
+ <meta property="og:description" content="Apache Airflow 3.1.0 introduces Human-in-the-Loop workflows, 17-language internationalization, deadline alerts, and React plugin system for data orchestration teams.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2025-09-25T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Release">
+ <meta property="og:image" content="/blog/airflow-3.1.0/images/3.1.0.jpg">
-<meta itemprop="name" content="Apache Airflow 3.1.0: Human-Centered Workflows">
-<meta itemprop="description" content="Apache Airflow 3.1.0 introduces Human-in-the-Loop workflows, 17-language internationalization, deadline alerts, and React plugin system for data orchestration teams."><meta itemprop="datePublished" content="2025-09-25T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="1584"><meta itemprop="image" content="/blog/airflow-3.1.0/images/3.1.0.jpg">
-<meta itemprop="keywords" content="Release," />
+ <meta itemprop="name" content="Apache Airflow 3.1.0: Human-Centered Workflows">
+ <meta itemprop="description" content="Apache Airflow 3.1.0 introduces Human-in-the-Loop workflows, 17-language internationalization, deadline alerts, and React plugin system for data orchestration teams.">
+ <meta itemprop="datePublished" content="2025-09-25T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="1584">
+ <meta itemprop="image" content="/blog/airflow-3.1.0/images/3.1.0.jpg">
+ <meta itemprop="keywords" content="Release">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +94,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -449,7 +457,7 @@ processes, comprehensive internationalization support, and significant developer
🚏 Constraints: <a href="https://github.com/apache/airflow/tree/constraints-3.1.0">https://github.com/apache/airflow/tree/constraints-3.1.0</a></p>
<h2 id="-human-in-the-loop-hitl-when-automation-meets-human-judgment">🤝 Human-in-the-Loop (HITL): When Automation Meets Human Judgment</h2>
<p>This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for:</p>
-<p><img src="images/hitl.gif" alt="Human-in-the-Loop HITL"></p>
+<p><img alt="Human-in-the-Loop HITL" src="/blog/airflow-3.1.0/images/hitl.gif"></p>
<ul>
<li><strong>AI/ML Model Validation</strong>: Pause inference pipelines for human review of model outputs</li>
<li><strong>Content Moderation</strong>: Route content through human reviewers before publication</li>
@@ -458,25 +466,25 @@ processes, comprehensive internationalization support, and significant developer
</ul>
<p><strong>HITL</strong> tasks pause in a deferred state while presenting intuitive web forms in the Airflow UI. Users with appropriate roles can review context data, DAG parameters, and XCom values before making informed decisions.</p>
<h2 id="example-code">Example Code:</h2>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="kn">from</span> <span class="nn">airflow.sdk</span> <span class="kn">import</span> <span class="n">DAG</span>
-<span class="kn">from</span> <span class="nn">airflow.providers.standard.operators.hitl</span> <span class="kn">import</span> <span class="n">HITLOperator</span>
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s2">"content_moderation"</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="s2">"@daily"</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
- <span class="n">moderate_content</span> <span class="o">=</span> <span class="n">HITLOperator</span><span class="p">(</span>
- <span class="n">task_id</span><span class="o">=</span><span class="s2">"review_content"</span><span class="p">,</span>
- <span class="n">message</span><span class="o">=</span><span class="s2">"Please review this content for publication"</span><span class="p">,</span>
- <span class="n">data_key</span><span class="o">=</span><span class="s2">"content_to_review"</span>
- <span class="p">)</span>
-</code></pre></div><h1 id="-ui-enhancements--performance">📊 UI Enhancements & Performance</h1>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.sdk</span> <span class="kn">import</span> <span class="n">DAG</span>
+</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.providers.standard.operators.hitl</span> <span class="kn">import</span> <span class="n">HITLOperator</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s2">"content_moderation"</span><span class="p">,</span> <span class="n">schedule</span><span class="o">=</span><span class="s2">"@daily"</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="n">moderate_content</span> <span class="o">=</span> <span class="n">HITLOperator</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">task_id</span><span class="o">=</span><span class="s2">"review_content"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">message</span><span class="o">=</span><span class="s2">"Please review this content for publication"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">data_key</span><span class="o">=</span><span class="s2">"content_to_review"</span>
+</span></span><span class="line"><span class="cl"> <span class="p">)</span>
+</span></span></code></pre></div><h1 id="-ui-enhancements--performance">📊 UI Enhancements & Performance</h1>
<h2 id="calendar-and-gantt-views-make-their-comeback">Calendar and Gantt Views Make Their Comeback</h2>
<p>Remember those beloved Calendar and Gantt chart views from Airflow 2.x? They’re back, completely rebuilt for the
modern React UI after being omitted from the 3.0 release.</p>
<p>The new Calendar view is genuinely interactive with filtering capabilities that make it easy to drill down
into specific time periods and dag states.</p>
-<p><img src="images/calendar.gif" alt="Calendar View"></p>
+<p><img alt="Calendar View" src="/blog/airflow-3.1.0/images/calendar.gif"></p>
<p>The Gantt chart is now integrated directly into the grid view and renders much faster than the old
version, giving you that timeline perspective without the performance headaches.</p>
-<p><img src="images/gantt.png" alt="Gantt View"></p>
+<p><img alt="Gantt View" src="/blog/airflow-3.1.0/images/gantt.png"></p>
<h2 id="theme-updates-that-actually-matter">Theme Updates That Actually Matter</h2>
<p>We’ve refreshed the color palette using modern design principles, making the UI more consistent, professional
and most of all taken a careful look at contrast ratios so the UI should be more accessible.</p>
@@ -484,7 +492,7 @@ and most of all taken a careful look at contrast ratios so the UI should be more
<p>We’ve added a lot more filtering options across the pages!</p>
<p>Plus, you can now pin your <strong>favorite DAGs</strong> to keep them at the top of your list or to filter for them easily. It’s
one of those small features that makes a huge difference when dealing with 100s of workflows.</p>
-<p><img src="images/favorite.png" alt="Favorite Dags in the UI">
+<p><img alt="Favorite Dags in the UI" src="/blog/airflow-3.1.0/images/favorite.png">
Credited to Volker Janz.</p>
<blockquote>
<p>📊 <strong>UI Development Milestone</strong>: Airflow 3.1.0 features <strong>5x more UI pull requests</strong> than the 2.10 release and <strong>50% more</strong> than Airflow 3.0, demonstrating the community’s commitment to user experience excellence.</p>
@@ -499,28 +507,28 @@ exceed time thresholds, helping ensure SLA compliance and timely completion of c
<li><strong>Callback</strong>: Notifications via Airflow Notifiers or custom functions</li>
</ul>
<h2 id="example-code-1">Example Code:</h2>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
-<span class="kn">from</span> <span class="nn">airflow.sdk.definitions.deadline</span> <span class="kn">import</span> <span class="n">DeadlineAlert</span><span class="p">,</span> <span class="n">DeadlineReference</span><span class="p">,</span> <span class="n">AsyncCallback</span>
-<span class="kn">from</span> <span class="nn">airflow.providers.slack.notifications.slack_webhook</span> <span class="kn">import</span> <span class="n">SlackWebhookNotifier</span>
-
-<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
- <span class="s2">"critical_etl"</span><span class="p">,</span>
- <span class="n">deadline</span><span class="o">=</span><span class="n">DeadlineAlert</span><span class="p">(</span>
- <span class="n">reference</span><span class="o">=</span><span class="n">DeadlineReference</span><span class="o">.</span><span class="n">DAGRUN_QUEUED_AT</span><span class="p">,</span>
- <span class="n">interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">2</span><span class="p">),</span>
- <span class="n">callback</span><span class="o">=</span><span class="n">AsyncCallback</span><span class="p">(</span>
- <span class="n">SlackWebhookNotifier</span><span class="p">,</span>
- <span class="n">kwargs</span><span class="o">=</span><span class="p">{</span><span class="s2">"text"</span><span class="p">:</span> <span class="s2">"🚨 Critical ETL missed deadline!"</span><span class="p">}</span>
- <span class="p">)</span>
- <span class="p">)</span>
-<span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
- <span class="c1"># Your tasks here</span>
-</code></pre></div><p>Perfect for monitoring daily ETLs, alerting before critical deadlines, or escalating resource-constrained workflows.</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
+</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.sdk.definitions.deadline</span> <span class="kn">import</span> <span class="n">DeadlineAlert</span><span class="p">,</span> <span class="n">DeadlineReference</span><span class="p">,</span> <span class="n">AsyncCallback</span>
+</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">airflow.providers.slack.notifications.slack_webhook</span> <span class="kn">import</span> <span class="n">SlackWebhookNotifier</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="s2">"critical_etl"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">deadline</span><span class="o">=</span><span class="n">DeadlineAlert</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">reference</span><span class="o">=</span><span class="n">DeadlineReference</span><span class="o">.</span><span class="n">DAGRUN_QUEUED_AT</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">2</span><span class="p">),</span>
+</span></span><span class="line"><span class="cl"> <span class="n">callback</span><span class="o">=</span><span class="n">AsyncCallback</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="n">SlackWebhookNotifier</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="n">kwargs</span><span class="o">=</span><span class="p">{</span><span class="s2">"text"</span><span class="p">:</span> <span class="s2">"🚨 Critical ETL missed deadline!"</span><span class="p">}</span>
+</span></span><span class="line"><span class="cl"> <span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="p">)</span>
+</span></span><span class="line"><span class="cl"><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="c1"># Your tasks here</span>
+</span></span></code></pre></div><p>Perfect for monitoring daily ETLs, alerting before critical deadlines, or escalating resource-constrained workflows.</p>
<h1 id="-going-global-with-17-languages">🌍 Going Global with 17 Languages</h1>
<p>Airflow now speaks your team’s language. Literally. We have added comprehensive internationalization support
for <strong>17 languages</strong>, including Arabic, Chinese, French, German, Spanish and more. The interface detects your
browser preferences automatically, but you can switch languages on the fly without refreshing the page.</p>
-<p><img src="images/i18n-demo.gif" alt="Internationalization Demo"></p>
+<p><img alt="Internationalization Demo" src="/blog/airflow-3.1.0/images/i18n-demo.gif"></p>
<p>For our Arabic and Hebrew users, we’ve built in <strong>proper right-to-left (RTL) support</strong></p>
<p>The best part? We have made it straightforward for the community to contribute additional languages with clear
contribution guidelines, so this is just the beginning of Airflow’s global reach.</p>
@@ -540,7 +548,7 @@ FastAPI sub applications and middlewares that fit your specific processes.</p>
</ul>
<p>We’ve already seen teams integrate everything from Wikipedia searches to data lineage
visualizations to yes, someone building a snake game to play while waiting on dag runs!</p>
-<p><img src="images/snake.gif" alt="Snake Game Plugin">
+<p><img alt="Snake Game Plugin" src="/blog/airflow-3.1.0/images/snake.gif">
Credited to Tamara Fingerlin.</p>
<h1 id="-enhanced-developer-and-authoring-experience">🔧 Enhanced Developer and Authoring Experience</h1>
<h2 id="task-sdk-evolution">Task SDK Evolution</h2>
@@ -558,31 +566,31 @@ complete separation arrives in 3.2.0, the foundation enables:</p>
until completion, enabling responsive integration patterns for real-time workflows.</p>
<p>The below example use <a href="https://www.python-httpx.org/async/"><code>httpx</code></a> to trigger a dag run, and emits the final dag run
state after it finishes:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="kn">import</span> <span class="nn">asyncio</span>
-<span class="kn">import</span> <span class="nn">json</span>
-
-<span class="kn">import</span> <span class="nn">httpx</span>
-
-<span class="n">dag_id</span> <span class="o">=</span> <span class="s2">"my-dag"</span>
-
-<span class="k">async</span> <span class="k">def</span> <span class="nf">create_and_wait</span><span class="p">(</span><span class="n">client</span><span class="p">):</span>
- <span class="c1"># Create a dag run...</span>
- <span class="n">r</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="sa">f</span><span class="s2">"https://my-airflow.example.com/api/v2/dags/</span><span class="si">{</span><span class="n">dag_id</span><span class="si">}</span><span class="s2">/dagRuns"</span><span class="p">)</span>
- <span class="n">run_id</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s2">"dag_run_id"</span><span class="p">]</span>
- <span class="k">async</span> <span class="k">with</span> <span class="n">client</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span>
- <span class="s2">"GET"</span><span class="p">,</span>
- <span class="sa">f</span><span class="s2">"https://my-airflow.example.com/api/v2/dags/</span><span class="si">{</span><span class="n">dag_id</span><span class="si">}</span><span class="s2">/dagRuns/</span><span class="si">{</span><span class="n">run_id</span><span class="si">}</span><span class="s2">/wait"</span><span class="p">,</span>
- <span class="p">)</span> <span class="k">as</span> <span class="n">r</span><span class="p">:</span>
- <span class="k">async</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">aiter_lines</span><span class="p">():</span>
- <span class="k">pass</span> <span class="c1"># You can do progress report here instead.</span>
- <span class="nb">print</span><span class="p">(</span><span class="s2">"Dag run state:"</span><span class="p">,</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())[</span><span class="s2">"state"</span><span class="p">])</span>
-
-<span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
- <span class="k">async</span> <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">()</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
- <span class="k">await</span> <span class="n">create_and_wait</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
-
-<span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>
-</code></pre></div><h1 id="-amazing-community">🙏 Amazing Community</h1>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
+</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">dag_id</span> <span class="o">=</span> <span class="s2">"my-dag"</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">create_and_wait</span><span class="p">(</span><span class="n">client</span><span class="p">):</span>
+</span></span><span class="line"><span class="cl"> <span class="c1"># Create a dag run...</span>
+</span></span><span class="line"><span class="cl"> <span class="n">r</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="sa">f</span><span class="s2">"https://my-airflow.example.com/api/v2/dags/</span><span class="si">{</span><span class="n">dag_id</span><span class="si">}</span><span class="s2">/dagRuns"</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl"> <span class="n">run_id</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s2">"dag_run_id"</span><span class="p">]</span>
+</span></span><span class="line"><span class="cl"> <span class="k">async</span> <span class="k">with</span> <span class="n">client</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span>
+</span></span><span class="line"><span class="cl"> <span class="s2">"GET"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="sa">f</span><span class="s2">"https://my-airflow.example.com/api/v2/dags/</span><span class="si">{</span><span class="n">dag_id</span><span class="si">}</span><span class="s2">/dagRuns/</span><span class="si">{</span><span class="n">run_id</span><span class="si">}</span><span class="s2">/wait"</span><span class="p">,</span>
+</span></span><span class="line"><span class="cl"> <span class="p">)</span> <span class="k">as</span> <span class="n">r</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="k">async</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">aiter_lines</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="k">pass</span> <span class="c1"># You can do progress report here instead.</span>
+</span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="s2">"Dag run state:"</span><span class="p">,</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())[</span><span class="s2">"state"</span><span class="p">])</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
+</span></span><span class="line"><span class="cl"> <span class="k">async</span> <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">()</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
+</span></span><span class="line"><span class="cl"> <span class="k">await</span> <span class="n">create_and_wait</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
+</span></span><span class="line"><span class="cl">
+</span></span><span class="line"><span class="cl"><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>
+</span></span></code></pre></div><h1 id="-amazing-community">🙏 Amazing Community</h1>
<p>Apache Airflow 3.1.0 represents an extraordinary community effort, showcasing the vibrant ecosystem that drives this project forward with <strong>163 contributors</strong> making this release possible across <strong>1,400+ commits</strong>.</p>
<h2 id="leading-contributors">Leading Contributors</h2>
<p>Special thanks to our top 20 contributors who drove this release forward: <strong>Amogh Desai</strong>, <strong>Ash Berlin-Taylor</strong>, <strong>Brent Bovenzi</strong>, <strong>Bugra Ozturk</strong>, <strong>Daniel Standish</strong>, <strong>Elad Kalif</strong>, <strong>Ephraim Anierobi</strong>, <strong>GPK</strong>, <strong>Guan Ming (Wesley) Chiu</strong>, <strong>Jarek Potiuk</strong>, <strong>Jens Scheffler</strong>, <strong>Karthikeyan Singaravelan</strong>, <strong>Kaxil Nai [...]
@@ -779,7 +787,7 @@ human-centered. We can’t wait to see what you build with it!</p>
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-3.1.0/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-3.1.0>
@@ -931,13 +939,13 @@ human-centered. We can’t wait to see what you build with it!</p>
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-survey-2020/index.html b/blog/airflow-survey-2020/index.html
index 13753791ed..b6ee89c63e 100644
--- a/blog/airflow-survey-2020/index.html
+++ b/blog/airflow-survey-2020/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,36 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Airflow Survey 2020 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2020" />
-<meta property="og:description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey-2020/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2021-03-09T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Airflow Survey 2020 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-survey-2020/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Airflow Survey 2020">
+ <meta property="og:description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2021-03-09T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Community">
+ <meta property="article:tag" content="Survey">
+ <meta property="article:tag" content="Users">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Airflow Survey 2020">
-<meta itemprop="description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance."><meta itemprop="datePublished" content="2021-03-09T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="2136"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="community,survey,users," />
+ <meta itemprop="name" content="Airflow Survey 2020">
+ <meta itemprop="description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.">
+ <meta itemprop="datePublished" content="2021-03-09T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="2136">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Community,Survey,Users">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +96,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -453,58 +463,58 @@ an amount of active contributors. So listening and understanding our community i
possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
is true, we will learn next year!</p>
<h2 id="overview-of-the-user">Overview of the user</h2>
-<p><img src="What_best_describes_your_current_occupation.png" alt=""></p>
+<p><img src="/blog/airflow-survey-2020/What_best_describes_your_current_occupation.png"></p>
<p><strong>What best describes your current occupation? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Data Engineer</td>
-<td>115</td>
-<td>56.65</td>
-</tr>
-<tr>
-<td>Developer</td>
-<td>28</td>
-<td>13.79</td>
-</tr>
-<tr>
-<td>DevOps</td>
-<td>17</td>
-<td>8.37</td>
-</tr>
-<tr>
-<td>Solutions Architect</td>
-<td>14</td>
-<td>6.9</td>
-</tr>
-<tr>
-<td>Data Scientist</td>
-<td>12</td>
-<td>5.91</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>10</td>
-<td>4.93</td>
-</tr>
-<tr>
-<td>Data Analyst</td>
-<td>4</td>
-<td>1.97</td>
-</tr>
-<tr>
-<td>Support Engineer</td>
-<td>3</td>
-<td>1.48</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Data Engineer</td>
+ <td style="text-align: left">115</td>
+ <td style="text-align: left">56.65</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Developer</td>
+ <td style="text-align: left">28</td>
+ <td style="text-align: left">13.79</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DevOps</td>
+ <td style="text-align: left">17</td>
+ <td style="text-align: left">8.37</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Solutions Architect</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">6.9</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Data Scientist</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">5.91</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">10</td>
+ <td style="text-align: left">4.93</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Data Analyst</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">1.97</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Support Engineer</td>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">1.48</td>
+ </tr>
+ </tbody>
</table>
<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/">those from 2019</a> with
@@ -512,91 +522,91 @@ visible slight increase in ML use cases.</p>
<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
<p><strong>How many people work in your company? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>200+</td>
-<td>107</td>
-<td>52.71</td>
-</tr>
-<tr>
-<td>51-200</td>
-<td>44</td>
-<td>21.67</td>
-</tr>
-<tr>
-<td>11-50</td>
-<td>37</td>
-<td>18.23</td>
-</tr>
-<tr>
-<td>1-10</td>
-<td>15</td>
-<td>7.39</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">200+</td>
+ <td style="text-align: left">107</td>
+ <td style="text-align: left">52.71</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">51-200</td>
+ <td style="text-align: left">44</td>
+ <td style="text-align: left">21.67</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">11-50</td>
+ <td style="text-align: left">37</td>
+ <td style="text-align: left">18.23</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1-10</td>
+ <td style="text-align: left">15</td>
+ <td style="text-align: left">7.39</td>
+ </tr>
+ </tbody>
</table>
<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>1-5</td>
-<td>84</td>
-<td>41.38</td>
-</tr>
-<tr>
-<td>6-20</td>
-<td>75</td>
-<td>36.95</td>
-</tr>
-<tr>
-<td>21-50</td>
-<td>23</td>
-<td>11.33</td>
-</tr>
-<tr>
-<td>50+</td>
-<td>21</td>
-<td>10.34</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">1-5</td>
+ <td style="text-align: left">84</td>
+ <td style="text-align: left">41.38</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">6-20</td>
+ <td style="text-align: left">75</td>
+ <td style="text-align: left">36.95</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">21-50</td>
+ <td style="text-align: left">23</td>
+ <td style="text-align: left">11.33</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">50+</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">10.34</td>
+ </tr>
+ </tbody>
</table>
<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>No, we are happy with Airflow</td>
-<td>174</td>
-<td>85.71</td>
-</tr>
-<tr>
-<td>Yes</td>
-<td>29</td>
-<td>14.29</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">No, we are happy with Airflow</td>
+ <td style="text-align: left">174</td>
+ <td style="text-align: left">85.71</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes</td>
+ <td style="text-align: left">29</td>
+ <td style="text-align: left">14.29</td>
+ </tr>
+ </tbody>
</table>
<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
@@ -607,30 +617,30 @@ in the new major version. And the early feedback from 2.0 users seems to be conf
Luigi, Kubeflow or custom solutions.</p>
<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>I wish we could</td>
-<td>99</td>
-<td>48.77</td>
-</tr>
-<tr>
-<td>No</td>
-<td>59</td>
-<td>29.06</td>
-</tr>
-<tr>
-<td>Yes</td>
-<td>45</td>
-<td>22.17</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">I wish we could</td>
+ <td style="text-align: left">99</td>
+ <td style="text-align: left">48.77</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No</td>
+ <td style="text-align: left">59</td>
+ <td style="text-align: left">29.06</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes</td>
+ <td style="text-align: left">45</td>
+ <td style="text-align: left">22.17</td>
+ </tr>
+ </tbody>
</table>
<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
But it would be good to learn if there’s something else than time that is stopping people who wish to contribute
@@ -639,585 +649,585 @@ That said - if you know something we can improve please reach out via Slack, dev
discussions.</p>
<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>2020 %</th>
-<th>2019 %</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Very Likely</td>
-<td>125</td>
-<td>61.58</td>
-<td>45.45%</td>
-</tr>
-<tr>
-<td>Likely</td>
-<td>62</td>
-<td>30.54</td>
-<td>40.26%</td>
-</tr>
-<tr>
-<td>Neutral</td>
-<td>11</td>
-<td>5.42</td>
-<td>10.71%</td>
-</tr>
-<tr>
-<td>Unlikely</td>
-<td>3</td>
-<td>1.48</td>
-<td>2.60%</td>
-</tr>
-<tr>
-<td>Very unlikely</td>
-<td>2</td>
-<td>0.99</td>
-<td>0.97%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">2020 %</th>
+ <th style="text-align: left">2019 %</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Very Likely</td>
+ <td style="text-align: left">125</td>
+ <td style="text-align: left">61.58</td>
+ <td style="text-align: left">45.45%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Likely</td>
+ <td style="text-align: left">62</td>
+ <td style="text-align: left">30.54</td>
+ <td style="text-align: left">40.26%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Neutral</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">5.42</td>
+ <td style="text-align: left">10.71%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Unlikely</td>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">1.48</td>
+ <td style="text-align: left">2.60%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Very unlikely</td>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">0.99</td>
+ <td style="text-align: left">0.97%</td>
+ </tr>
+ </tbody>
</table>
<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Documentation</td>
-<td>154</td>
-<td>75.86</td>
-</tr>
-<tr>
-<td>Airflow website</td>
-<td>139</td>
-<td>68.47</td>
-</tr>
-<tr>
-<td>Slack</td>
-<td>128</td>
-<td>63.05</td>
-</tr>
-<tr>
-<td>Github</td>
-<td>127</td>
-<td>62.56</td>
-</tr>
-<tr>
-<td>Stack Overflow</td>
-<td>72</td>
-<td>35.47</td>
-</tr>
-<tr>
-<td>Airflow Summit Videos</td>
-<td>44</td>
-<td>21.67</td>
-</tr>
-<tr>
-<td>The dev mailing list</td>
-<td>33</td>
-<td>16.26</td>
-</tr>
-<tr>
-<td>Awesome Apache Airflow repository</td>
-<td>21</td>
-<td>10.34</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>15</td>
-<td>7.39</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Documentation</td>
+ <td style="text-align: left">154</td>
+ <td style="text-align: left">75.86</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Airflow website</td>
+ <td style="text-align: left">139</td>
+ <td style="text-align: left">68.47</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Slack</td>
+ <td style="text-align: left">128</td>
+ <td style="text-align: left">63.05</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Github</td>
+ <td style="text-align: left">127</td>
+ <td style="text-align: left">62.56</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Stack Overflow</td>
+ <td style="text-align: left">72</td>
+ <td style="text-align: left">35.47</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Airflow Summit Videos</td>
+ <td style="text-align: left">44</td>
+ <td style="text-align: left">21.67</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">The dev mailing list</td>
+ <td style="text-align: left">33</td>
+ <td style="text-align: left">16.26</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Awesome Apache Airflow repository</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">10.34</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">15</td>
+ <td style="text-align: left">7.39</td>
+ </tr>
+ </tbody>
</table>
<p>Here we see that Airflow documentation is the crucial source of information. What’s interesting is that more
than 60% of users are getting information from Github and Slack channels.</p>
-<p><img src="Where_are_you_based.png" alt=""></p>
+<p><img src="/blog/airflow-survey-2020/Where_are_you_based.png"></p>
<h2 id="airflow-uses-cases">Airflow uses cases</h2>
<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>No, we use vanilla Airflow</td>
-<td>154</td>
-<td>75.86</td>
-</tr>
-<tr>
-<td>Yes, we have small patches (no fork)</td>
-<td>34</td>
-<td>16.75</td>
-</tr>
-<tr>
-<td>Yes, we have separate fork</td>
-<td>15</td>
-<td>7.39</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">No, we use vanilla Airflow</td>
+ <td style="text-align: left">154</td>
+ <td style="text-align: left">75.86</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, we have small patches (no fork)</td>
+ <td style="text-align: left">34</td>
+ <td style="text-align: left">16.75</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, we have separate fork</td>
+ <td style="text-align: left">15</td>
+ <td style="text-align: left">7.39</td>
+ </tr>
+ </tbody>
</table>
<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>No guide on best practises on developing DAGs</td>
-<td>102</td>
-<td>50.25</td>
-</tr>
-<tr>
-<td>There is no easy option to launch Airflow</td>
-<td>64</td>
-<td>31.53</td>
-</tr>
-<tr>
-<td>Small number of tutorials on different aspects of using Airflow</td>
-<td>57</td>
-<td>28.08</td>
-</tr>
-<tr>
-<td>Documentation is not clear enough</td>
-<td>53</td>
-<td>26.11</td>
-</tr>
-<tr>
-<td>There is no easy option to deploy DAGs to an Airflow instance</td>
-<td>52</td>
-<td>25.62</td>
-</tr>
-<tr>
-<td>No problems</td>
-<td>34</td>
-<td>16.75</td>
-</tr>
-<tr>
-<td>Small number of blogs regarding Airflow</td>
-<td>30</td>
-<td>14.78</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">No guide on best practises on developing DAGs</td>
+ <td style="text-align: left">102</td>
+ <td style="text-align: left">50.25</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">There is no easy option to launch Airflow</td>
+ <td style="text-align: left">64</td>
+ <td style="text-align: left">31.53</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Small number of tutorials on different aspects of using Airflow</td>
+ <td style="text-align: left">57</td>
+ <td style="text-align: left">28.08</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Documentation is not clear enough</td>
+ <td style="text-align: left">53</td>
+ <td style="text-align: left">26.11</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">There is no easy option to deploy DAGs to an Airflow instance</td>
+ <td style="text-align: left">52</td>
+ <td style="text-align: left">25.62</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No problems</td>
+ <td style="text-align: left">34</td>
+ <td style="text-align: left">16.75</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Small number of blogs regarding Airflow</td>
+ <td style="text-align: left">30</td>
+ <td style="text-align: left">14.78</td>
+ </tr>
+ </tbody>
</table>
<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Original Airflow Graphical User Interface</td>
-<td>199</td>
-<td>98.03</td>
-</tr>
-<tr>
-<td>CLI</td>
-<td>88</td>
-<td>43.35</td>
-</tr>
-<tr>
-<td>API</td>
-<td>48</td>
-<td>23.65</td>
-</tr>
-<tr>
-<td>Custom (own created) Airflow Graphical User Interface</td>
-<td>12</td>
-<td>5.91</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>3</td>
-<td>1.48</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Original Airflow Graphical User Interface</td>
+ <td style="text-align: left">199</td>
+ <td style="text-align: left">98.03</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">CLI</td>
+ <td style="text-align: left">88</td>
+ <td style="text-align: left">43.35</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">API</td>
+ <td style="text-align: left">48</td>
+ <td style="text-align: left">23.65</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Custom (own created) Airflow Graphical User Interface</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">5.91</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">1.48</td>
+ </tr>
+ </tbody>
</table>
<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Yes, by triggering another DAG</td>
-<td>87</td>
-<td>42.86</td>
-</tr>
-<tr>
-<td>No, I don’t combine multiple DAGs</td>
-<td>79</td>
-<td>38.92</td>
-</tr>
-<tr>
-<td>Yes, through SubDAG</td>
-<td>40</td>
-<td>19.7</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>18</td>
-<td>8.87</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Yes, by triggering another DAG</td>
+ <td style="text-align: left">87</td>
+ <td style="text-align: left">42.86</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No, I don’t combine multiple DAGs</td>
+ <td style="text-align: left">79</td>
+ <td style="text-align: left">38.92</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, through SubDAG</td>
+ <td style="text-align: left">40</td>
+ <td style="text-align: left">19.7</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">18</td>
+ <td style="text-align: left">8.87</td>
+ </tr>
+ </tbody>
</table>
<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Using existing dedicated operators / hooks</td>
-<td>147</td>
-<td>72.41</td>
-</tr>
-<tr>
-<td>Using Bash / Python operator</td>
-<td>140</td>
-<td>68.97</td>
-</tr>
-<tr>
-<td>Using own custom operators / hooks</td>
-<td>138</td>
-<td>67.98</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>12</td>
-<td>5.91</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Using existing dedicated operators / hooks</td>
+ <td style="text-align: left">147</td>
+ <td style="text-align: left">72.41</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using Bash / Python operator</td>
+ <td style="text-align: left">140</td>
+ <td style="text-align: left">68.97</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using own custom operators / hooks</td>
+ <td style="text-align: left">138</td>
+ <td style="text-align: left">67.98</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">5.91</td>
+ </tr>
+ </tbody>
</table>
<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Amazon Web Services</td>
-<td>121</td>
-<td>59.61</td>
-</tr>
-<tr>
-<td>Internal company systems</td>
-<td>113</td>
-<td>55.67</td>
-</tr>
-<tr>
-<td>Google Cloud Platform / Google APIs</td>
-<td>97</td>
-<td>47.78</td>
-</tr>
-<tr>
-<td>Hadoop / Spark / Flink / Other Apache software</td>
-<td>72</td>
-<td>35.47</td>
-</tr>
-<tr>
-<td>Microsoft Azure</td>
-<td>21</td>
-<td>10.34</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>19</td>
-<td>9.36</td>
-</tr>
-<tr>
-<td>I do not use external services in my Airflow DAGs</td>
-<td>5</td>
-<td>2.46</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Amazon Web Services</td>
+ <td style="text-align: left">121</td>
+ <td style="text-align: left">59.61</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Internal company systems</td>
+ <td style="text-align: left">113</td>
+ <td style="text-align: left">55.67</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Google Cloud Platform / Google APIs</td>
+ <td style="text-align: left">97</td>
+ <td style="text-align: left">47.78</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Hadoop / Spark / Flink / Other Apache software</td>
+ <td style="text-align: left">72</td>
+ <td style="text-align: left">35.47</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Microsoft Azure</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">10.34</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">19</td>
+ <td style="text-align: left">9.36</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I do not use external services in my Airflow DAGs</td>
+ <td style="text-align: left">5</td>
+ <td style="text-align: left">2.46</td>
+ </tr>
+ </tbody>
</table>
-<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt=""></p>
+<p><img src="/blog/airflow-survey-2020/What_external_services_do_you_use_in_your_Airflow_DAGs.png"></p>
<p><strong>Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Adding new operators/sensors and hooks</td>
-<td>119</td>
-<td>58.62</td>
-</tr>
-<tr>
-<td>I don’t use Airflow plugins</td>
-<td>69</td>
-<td>33.99</td>
-</tr>
-<tr>
-<td>Adding AppBuilder views & menu items</td>
-<td>27</td>
-<td>13.3</td>
-</tr>
-<tr>
-<td>Adding new executors</td>
-<td>17</td>
-<td>8.37</td>
-</tr>
-<tr>
-<td>Adding OperatorExtraLinks</td>
-<td>13</td>
-<td>6.4</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Adding new operators/sensors and hooks</td>
+ <td style="text-align: left">119</td>
+ <td style="text-align: left">58.62</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t use Airflow plugins</td>
+ <td style="text-align: left">69</td>
+ <td style="text-align: left">33.99</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Adding AppBuilder views & menu items</td>
+ <td style="text-align: left">27</td>
+ <td style="text-align: left">13.3</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Adding new executors</td>
+ <td style="text-align: left">17</td>
+ <td style="text-align: left">8.37</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Adding OperatorExtraLinks</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6.4</td>
+ </tr>
+ </tbody>
</table>
<p>| Other</p>
<p><strong>Do you use Airflow’s data lineage feature? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>No, I will use such feature if fully supported in Airflow</td>
-<td>105</td>
-<td>51.72</td>
-</tr>
-<tr>
-<td>No, data lineage isn’t a concern for my usage.</td>
-<td>68</td>
-<td>33.5</td>
-</tr>
-<tr>
-<td>Yes, I use another data lineage product</td>
-<td>24</td>
-<td>11.82</td>
-</tr>
-<tr>
-<td>Yes, I use custom implementation</td>
-<td>5</td>
-<td>2.46</td>
-</tr>
-<tr>
-<td>Yes, I use Airflow’s experimental data lineage feature</td>
-<td>1</td>
-<td>0.49</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">No, I will use such feature if fully supported in Airflow</td>
+ <td style="text-align: left">105</td>
+ <td style="text-align: left">51.72</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No, data lineage isn’t a concern for my usage.</td>
+ <td style="text-align: left">68</td>
+ <td style="text-align: left">33.5</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, I use another data lineage product</td>
+ <td style="text-align: left">24</td>
+ <td style="text-align: left">11.82</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, I use custom implementation</td>
+ <td style="text-align: left">5</td>
+ <td style="text-align: left">2.46</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, I use Airflow’s experimental data lineage feature</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.49</td>
+ </tr>
+ </tbody>
</table>
<p>When asked what lineage product users use, the answers were varying from custom tools
to known product like Amundsen, Atlas or dbt.</p>
<h2 id="deployment">Deployment</h2>
<p><strong>How many active DAGs do you have in your largest Airflow instance? (open question)</strong></p>
<table>
-<thead>
-<tr>
-<th>Number of DAGs</th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>< 20</td>
-<td>64</td>
-<td>32</td>
-</tr>
-<tr>
-<td>21-40</td>
-<td>33</td>
-<td>16</td>
-</tr>
-<tr>
-<td>41-60</td>
-<td>13</td>
-<td>6</td>
-</tr>
-<tr>
-<td>61-100</td>
-<td>32</td>
-<td>16</td>
-</tr>
-<tr>
-<td>101-200</td>
-<td>31</td>
-<td>15</td>
-</tr>
-<tr>
-<td>201-300</td>
-<td>8</td>
-<td>4</td>
-</tr>
-<tr>
-<td>301-999</td>
-<td>12</td>
-<td>6</td>
-</tr>
-<tr>
-<td>1000+</td>
-<td>10</td>
-<td>5</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left">Number of DAGs</th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">< 20</td>
+ <td style="text-align: left">64</td>
+ <td style="text-align: left">32</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">21-40</td>
+ <td style="text-align: left">33</td>
+ <td style="text-align: left">16</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">41-60</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">61-100</td>
+ <td style="text-align: left">32</td>
+ <td style="text-align: left">16</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">101-200</td>
+ <td style="text-align: left">31</td>
+ <td style="text-align: left">15</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">201-300</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">4</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">301-999</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">6</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1000+</td>
+ <td style="text-align: left">10</td>
+ <td style="text-align: left">5</td>
+ </tr>
+ </tbody>
</table>
<p><strong>What is the maximum number of tasks that you have used in one DAG? (open question)</strong></p>
<table>
-<thead>
-<tr>
-<th>Number of DAGs</th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>< 10</td>
-<td>42</td>
-<td>21</td>
-</tr>
-<tr>
-<td>11-20</td>
-<td>31</td>
-<td>15</td>
-</tr>
-<tr>
-<td>21-30</td>
-<td>15</td>
-<td>7</td>
-</tr>
-<tr>
-<td>31-40</td>
-<td>11</td>
-<td>5</td>
-</tr>
-<tr>
-<td>41-50</td>
-<td>22</td>
-<td>11</td>
-</tr>
-<tr>
-<td>51-100</td>
-<td>39</td>
-<td>19</td>
-</tr>
-<tr>
-<td>101-200</td>
-<td>16</td>
-<td>8</td>
-</tr>
-<tr>
-<td>201-500</td>
-<td>16</td>
-<td>8</td>
-</tr>
-<tr>
-<td>501+</td>
-<td>11</td>
-<td>5</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left">Number of DAGs</th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">< 10</td>
+ <td style="text-align: left">42</td>
+ <td style="text-align: left">21</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">11-20</td>
+ <td style="text-align: left">31</td>
+ <td style="text-align: left">15</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">21-30</td>
+ <td style="text-align: left">15</td>
+ <td style="text-align: left">7</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">31-40</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">5</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">41-50</td>
+ <td style="text-align: left">22</td>
+ <td style="text-align: left">11</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">51-100</td>
+ <td style="text-align: left">39</td>
+ <td style="text-align: left">19</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">101-200</td>
+ <td style="text-align: left">16</td>
+ <td style="text-align: left">8</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">201-500</td>
+ <td style="text-align: left">16</td>
+ <td style="text-align: left">8</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">501+</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">5</td>
+ </tr>
+ </tbody>
</table>
<p><strong>Which version of Airflow do you use currently? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>1.10.14</td>
-<td>55</td>
-<td>27.09</td>
-</tr>
-<tr>
-<td>2.0.0+</td>
-<td>45</td>
-<td>22.17</td>
-</tr>
-<tr>
-<td>1.10.12</td>
-<td>27</td>
-<td>13.3</td>
-</tr>
-<tr>
-<td>1.10.10</td>
-<td>26</td>
-<td>12.81</td>
-</tr>
-<tr>
-<td>1.10.11</td>
-<td>14</td>
-<td>6.9</td>
-</tr>
-<tr>
-<td>1.10.5 or older</td>
-<td>10</td>
-<td>4.93</td>
-</tr>
-<tr>
-<td>1.10.9</td>
-<td>8</td>
-<td>3.94</td>
-</tr>
-<tr>
-<td>1.10.13</td>
-<td>7</td>
-<td>3.45</td>
-</tr>
-<tr>
-<td>1.10.6</td>
-<td>4</td>
-<td>1.97</td>
-</tr>
-<tr>
-<td>1.10.7</td>
-<td>4</td>
-<td>1.97</td>
-</tr>
-<tr>
-<td>1.10.8</td>
-<td>3</td>
-<td>1.48</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">1.10.14</td>
+ <td style="text-align: left">55</td>
+ <td style="text-align: left">27.09</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2.0.0+</td>
+ <td style="text-align: left">45</td>
+ <td style="text-align: left">22.17</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.12</td>
+ <td style="text-align: left">27</td>
+ <td style="text-align: left">13.3</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.10</td>
+ <td style="text-align: left">26</td>
+ <td style="text-align: left">12.81</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.11</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">6.9</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.5 or older</td>
+ <td style="text-align: left">10</td>
+ <td style="text-align: left">4.93</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.9</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">3.94</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.13</td>
+ <td style="text-align: left">7</td>
+ <td style="text-align: left">3.45</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.6</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">1.97</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.7</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">1.97</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.8</td>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">1.48</td>
+ </tr>
+ </tbody>
</table>
<p>This was probably one of the most important questions in the survey. While it’s good to see
that more than 60% of users use one of three latest Airflow versions, it’s worrying that the rest
@@ -1225,529 +1235,529 @@ are using versions that are old or have known security vulnerabilities.</p>
<p>Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.</p>
<p><strong>What meta-database do you use? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Postgres 12</td>
-<td>36</td>
-<td>17.73</td>
-</tr>
-<tr>
-<td>Postgres 9.6</td>
-<td>33</td>
-<td>16.26</td>
-</tr>
-<tr>
-<td>Postgres 11</td>
-<td>31</td>
-<td>15.27</td>
-</tr>
-<tr>
-<td>MySQL 5.7</td>
-<td>27</td>
-<td>13.3</td>
-</tr>
-<tr>
-<td>MySQL 8.0</td>
-<td>20</td>
-<td>9.85</td>
-</tr>
-<tr>
-<td>Postgres 10</td>
-<td>20</td>
-<td>9.85</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>19</td>
-<td>9.36</td>
-</tr>
-<tr>
-<td>Postgres 13</td>
-<td>18</td>
-<td>8.87</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Postgres 12</td>
+ <td style="text-align: left">36</td>
+ <td style="text-align: left">17.73</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Postgres 9.6</td>
+ <td style="text-align: left">33</td>
+ <td style="text-align: left">16.26</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Postgres 11</td>
+ <td style="text-align: left">31</td>
+ <td style="text-align: left">15.27</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">MySQL 5.7</td>
+ <td style="text-align: left">27</td>
+ <td style="text-align: left">13.3</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">MySQL 8.0</td>
+ <td style="text-align: left">20</td>
+ <td style="text-align: left">9.85</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Postgres 10</td>
+ <td style="text-align: left">20</td>
+ <td style="text-align: left">9.85</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">19</td>
+ <td style="text-align: left">9.36</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Postgres 13</td>
+ <td style="text-align: left">18</td>
+ <td style="text-align: left">8.87</td>
+ </tr>
+ </tbody>
</table>
<p>This means that more about 69% of users decide to use Postgres as their meta-database.
MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.</p>
<p>It’s good to know that users rather avoid using SQLite in production deployments!</p>
<p><strong>What executor type do you use? (single choice)</strong></p>
-<p><img src="What_executor_type_do_you_use.png" alt=""></p>
+<p><img src="/blog/airflow-survey-2020/What_executor_type_do_you_use.png"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>2020</th>
-<th>2019</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Celery</td>
-<td>100</td>
-<td>49.26%</td>
-<td>44.81%</td>
-</tr>
-<tr>
-<td>Kubernetes</td>
-<td>48</td>
-<td>23.65%</td>
-<td>16.88%</td>
-</tr>
-<tr>
-<td>Local</td>
-<td>40</td>
-<td>19.7%</td>
-<td>27.60%</td>
-</tr>
-<tr>
-<td>Sequential</td>
-<td>10</td>
-<td>4.93%</td>
-<td>7.14%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>5</td>
-<td>2.46%</td>
-<td>3.57</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">2020</th>
+ <th style="text-align: left">2019</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Celery</td>
+ <td style="text-align: left">100</td>
+ <td style="text-align: left">49.26%</td>
+ <td style="text-align: left">44.81%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Kubernetes</td>
+ <td style="text-align: left">48</td>
+ <td style="text-align: left">23.65%</td>
+ <td style="text-align: left">16.88%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Local</td>
+ <td style="text-align: left">40</td>
+ <td style="text-align: left">19.7%</td>
+ <td style="text-align: left">27.60%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Sequential</td>
+ <td style="text-align: left">10</td>
+ <td style="text-align: left">4.93%</td>
+ <td style="text-align: left">7.14%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">5</td>
+ <td style="text-align: left">2.46%</td>
+ <td style="text-align: left">3.57</td>
+ </tr>
+ </tbody>
</table>
<p>In comparison to previous year it seems that more users use currently Celery and
Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
-suggest that users' deployments are growing, and they need more scalable solutions.</p>
+suggest that users’ deployments are growing, and they need more scalable solutions.</p>
<p>Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
is using other brokers or is not sure what is used in their deployments.</p>
<p><strong>What metrics do you use to monitor Airflow? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>I do not use monitoring</td>
-<td>65</td>
-<td>32.02</td>
-</tr>
-<tr>
-<td>External monitoring service</td>
-<td>60</td>
-<td>29.56</td>
-</tr>
-<tr>
-<td>Information from metadatabase</td>
-<td>51</td>
-<td>25.12</td>
-</tr>
-<tr>
-<td>Statsd</td>
-<td>49</td>
-<td>24.14</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>31</td>
-<td>15.27</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">I do not use monitoring</td>
+ <td style="text-align: left">65</td>
+ <td style="text-align: left">32.02</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">External monitoring service</td>
+ <td style="text-align: left">60</td>
+ <td style="text-align: left">29.56</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Information from metadatabase</td>
+ <td style="text-align: left">51</td>
+ <td style="text-align: left">25.12</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Statsd</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">24.14</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">31</td>
+ <td style="text-align: left">15.27</td>
+ </tr>
+ </tbody>
</table>
<p>The other responses included mostly information about tools used by users
including DataDog and Prometheus exporter.</p>
<p><strong>How do you deploy Airflow? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>On virtual machines (for example using AWS EC2)</td>
-<td>64</td>
-<td>31.53</td>
-</tr>
-<tr>
-<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
-<td>35</td>
-<td>17.24</td>
-</tr>
-<tr>
-<td>On Kubernetes (using custom deployments)</td>
-<td>29</td>
-<td>14.29</td>
-</tr>
-<tr>
-<td>On premises</td>
-<td>28</td>
-<td>13.79</td>
-</tr>
-<tr>
-<td>On Kubernetes (using another helm chart)</td>
-<td>20</td>
-<td>9.85</td>
-</tr>
-<tr>
-<td>On Kubernetes (using Apache Airflow’s helm chart)</td>
-<td>17</td>
-<td>8.37</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>12</td>
-<td>5.91</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">On virtual machines (for example using AWS EC2)</td>
+ <td style="text-align: left">64</td>
+ <td style="text-align: left">31.53</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
+ <td style="text-align: left">35</td>
+ <td style="text-align: left">17.24</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On Kubernetes (using custom deployments)</td>
+ <td style="text-align: left">29</td>
+ <td style="text-align: left">14.29</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On premises</td>
+ <td style="text-align: left">28</td>
+ <td style="text-align: left">13.79</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On Kubernetes (using another helm chart)</td>
+ <td style="text-align: left">20</td>
+ <td style="text-align: left">9.85</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On Kubernetes (using Apache Airflow’s helm chart)</td>
+ <td style="text-align: left">17</td>
+ <td style="text-align: left">8.37</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">5.91</td>
+ </tr>
+ </tbody>
</table>
<p>Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
10 percent more than in 2019. There’s slightly increase in usage of Airflow via
managed services (14.61% in 2019).</p>
<p><strong>Do you use containerisation for deployment? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Yes, using helm chart / kubernetes</td>
-<td>58</td>
-<td>28.57</td>
-</tr>
-<tr>
-<td>No, I don’t use containerisation</td>
-<td>57</td>
-<td>28.08</td>
-</tr>
-<tr>
-<td>Yes, single docker image</td>
-<td>49</td>
-<td>24.14</td>
-</tr>
-<tr>
-<td>Yes, using docker compose</td>
-<td>39</td>
-<td>19.21</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Yes, using helm chart / kubernetes</td>
+ <td style="text-align: left">58</td>
+ <td style="text-align: left">28.57</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No, I don’t use containerisation</td>
+ <td style="text-align: left">57</td>
+ <td style="text-align: left">28.08</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, single docker image</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">24.14</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, using docker compose</td>
+ <td style="text-align: left">39</td>
+ <td style="text-align: left">19.21</td>
+ </tr>
+ </tbody>
</table>
<p>Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
42% of those users use docker-compose for deployments.</p>
<p><strong>How do you distribute your DAGs? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
-<td>79</td>
-<td>38.92</td>
-</tr>
-<tr>
-<td>Bake them into the docker image</td>
-<td>56</td>
-<td>27.59</td>
-</tr>
-<tr>
-<td>Shared files system</td>
-<td>34</td>
-<td>16.75</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>20</td>
-<td>9.85</td>
-</tr>
-<tr>
-<td>I don’t know</td>
-<td>14</td>
-<td>6.9</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Using a synchronizing process (Git sync, GCS fuse, etc)</td>
+ <td style="text-align: left">79</td>
+ <td style="text-align: left">38.92</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Bake them into the docker image</td>
+ <td style="text-align: left">56</td>
+ <td style="text-align: left">27.59</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Shared files system</td>
+ <td style="text-align: left">34</td>
+ <td style="text-align: left">16.75</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">20</td>
+ <td style="text-align: left">9.85</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t know</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">6.9</td>
+ </tr>
+ </tbody>
</table>
<p>The most popular way of distributing DAGs seems to be using a synchronizing process. About
40% of users use this process together with Kubernetes deployments.</p>
<h2 id="future-of-airflow">Future of Airflow</h2>
<p><strong>In your opinion, what could be improved in Airflow? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Web UI</td>
-<td>100</td>
-<td>49.26</td>
-</tr>
-<tr>
-<td>Examples, how-to, onboarding documentation</td>
-<td>90</td>
-<td>44.33</td>
-</tr>
-<tr>
-<td>Logging, monitoring and alerting</td>
-<td>90</td>
-<td>44.33</td>
-</tr>
-<tr>
-<td>Technical documentation</td>
-<td>90</td>
-<td>44.33</td>
-</tr>
-<tr>
-<td>Scheduler performance</td>
-<td>83</td>
-<td>40.89</td>
-</tr>
-<tr>
-<td>DAG authoring</td>
-<td>64</td>
-<td>31.53</td>
-</tr>
-<tr>
-<td>Authentication and authorization</td>
-<td>58</td>
-<td>28.57</td>
-</tr>
-<tr>
-<td>REST API</td>
-<td>51</td>
-<td>25.12</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>44</td>
-<td>21.67</td>
-</tr>
-<tr>
-<td>Reliability</td>
-<td>41</td>
-<td>20.2</td>
-</tr>
-<tr>
-<td>External integration e.g. AWS, GCP, Apache products</td>
-<td>36</td>
-<td>17.73</td>
-</tr>
-<tr>
-<td>Security</td>
-<td>28</td>
-<td>13.79</td>
-</tr>
-<tr>
-<td>CLI</td>
-<td>20</td>
-<td>9.85</td>
-</tr>
-<tr>
-<td>Everything work fine for me</td>
-<td>14</td>
-<td>6.9</td>
-</tr>
-<tr>
-<td>I don’t know</td>
-<td>4</td>
-<td>1.97</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Web UI</td>
+ <td style="text-align: left">100</td>
+ <td style="text-align: left">49.26</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Examples, how-to, onboarding documentation</td>
+ <td style="text-align: left">90</td>
+ <td style="text-align: left">44.33</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Logging, monitoring and alerting</td>
+ <td style="text-align: left">90</td>
+ <td style="text-align: left">44.33</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Technical documentation</td>
+ <td style="text-align: left">90</td>
+ <td style="text-align: left">44.33</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Scheduler performance</td>
+ <td style="text-align: left">83</td>
+ <td style="text-align: left">40.89</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DAG authoring</td>
+ <td style="text-align: left">64</td>
+ <td style="text-align: left">31.53</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Authentication and authorization</td>
+ <td style="text-align: left">58</td>
+ <td style="text-align: left">28.57</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">REST API</td>
+ <td style="text-align: left">51</td>
+ <td style="text-align: left">25.12</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">44</td>
+ <td style="text-align: left">21.67</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Reliability</td>
+ <td style="text-align: left">41</td>
+ <td style="text-align: left">20.2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">External integration e.g. AWS, GCP, Apache products</td>
+ <td style="text-align: left">36</td>
+ <td style="text-align: left">17.73</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Security</td>
+ <td style="text-align: left">28</td>
+ <td style="text-align: left">13.79</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">CLI</td>
+ <td style="text-align: left">20</td>
+ <td style="text-align: left">9.85</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Everything work fine for me</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">6.9</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t know</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">1.97</td>
+ </tr>
+ </tbody>
</table>
<p><strong>Which features would most interest you? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>DAG versioning</td>
-<td>109</td>
-<td>53.69</td>
-</tr>
-<tr>
-<td>Builtin statistics</td>
-<td>71</td>
-<td>34.98</td>
-</tr>
-<tr>
-<td>Improved data lineage</td>
-<td>65</td>
-<td>32.02</td>
-</tr>
-<tr>
-<td>Scheduling at the start of the interval</td>
-<td>63</td>
-<td>31.03</td>
-</tr>
-<tr>
-<td>Stateless workers</td>
-<td>59</td>
-<td>29.06</td>
-</tr>
-<tr>
-<td>More option to configure schedules (time units, increments)</td>
-<td>57</td>
-<td>28.08</td>
-</tr>
-<tr>
-<td>Multi-tenant deployment</td>
-<td>49</td>
-<td>24.14</td>
-</tr>
-<tr>
-<td>DAG fetcher (AIP-5)</td>
-<td>39</td>
-<td>19.21</td>
-</tr>
-<tr>
-<td>Generic transfer operator</td>
-<td>34</td>
-<td>16.75</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>33</td>
-<td>16.26</td>
-</tr>
-<tr>
-<td>I have everything I need</td>
-<td>11</td>
-<td>5.42</td>
-</tr>
-<tr>
-<td>Nothing</td>
-<td>11</td>
-<td>5.42</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">DAG versioning</td>
+ <td style="text-align: left">109</td>
+ <td style="text-align: left">53.69</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Builtin statistics</td>
+ <td style="text-align: left">71</td>
+ <td style="text-align: left">34.98</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Improved data lineage</td>
+ <td style="text-align: left">65</td>
+ <td style="text-align: left">32.02</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Scheduling at the start of the interval</td>
+ <td style="text-align: left">63</td>
+ <td style="text-align: left">31.03</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Stateless workers</td>
+ <td style="text-align: left">59</td>
+ <td style="text-align: left">29.06</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">More option to configure schedules (time units, increments)</td>
+ <td style="text-align: left">57</td>
+ <td style="text-align: left">28.08</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Multi-tenant deployment</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">24.14</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DAG fetcher (AIP-5)</td>
+ <td style="text-align: left">39</td>
+ <td style="text-align: left">19.21</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Generic transfer operator</td>
+ <td style="text-align: left">34</td>
+ <td style="text-align: left">16.75</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">33</td>
+ <td style="text-align: left">16.26</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I have everything I need</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">5.42</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Nothing</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">5.42</td>
+ </tr>
+ </tbody>
</table>
<p><strong>Will you consider migrating to Airflow 2.0? (single choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Yes, as soon as possible</td>
-<td>81</td>
-<td>39.9</td>
-</tr>
-<tr>
-<td>Yes, once it’s mature (for example after 2.1)</td>
-<td>72</td>
-<td>35.47</td>
-</tr>
-<tr>
-<td>I am already using Airflow 2.0+</td>
-<td>39</td>
-<td>19.21</td>
-</tr>
-<tr>
-<td>I don’t know yet</td>
-<td>8</td>
-<td>3.94</td>
-</tr>
-<tr>
-<td>No, I do not plan to migrate</td>
-<td>3</td>
-<td>1.48</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Yes, as soon as possible</td>
+ <td style="text-align: left">81</td>
+ <td style="text-align: left">39.9</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, once it’s mature (for example after 2.1)</td>
+ <td style="text-align: left">72</td>
+ <td style="text-align: left">35.47</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I am already using Airflow 2.0+</td>
+ <td style="text-align: left">39</td>
+ <td style="text-align: left">19.21</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t know yet</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">3.94</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No, I do not plan to migrate</td>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">1.48</td>
+ </tr>
+ </tbody>
</table>
<p><strong>What are the features of Airflow 2.0 you are most excited about? (multiple choice)</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>General performance improvements</td>
-<td>133</td>
-<td>65.52</td>
-</tr>
-<tr>
-<td>Refreshed WebUI</td>
-<td>102</td>
-<td>50.25</td>
-</tr>
-<tr>
-<td>Scheduler HA</td>
-<td>99</td>
-<td>48.77</td>
-</tr>
-<tr>
-<td>Official docker image</td>
-<td>84</td>
-<td>41.38</td>
-</tr>
-<tr>
-<td>@task decorator</td>
-<td>56</td>
-<td>27.59</td>
-</tr>
-<tr>
-<td>Official helm chart</td>
-<td>51</td>
-<td>25.12</td>
-</tr>
-<tr>
-<td>Providers packages</td>
-<td>41</td>
-<td>20.2</td>
-</tr>
-<tr>
-<td>Configurable XCom backends</td>
-<td>33</td>
-<td>16.26</td>
-</tr>
-<tr>
-<td>CeleryKubernetesExecutor</td>
-<td>31</td>
-<td>15.27</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>12</td>
-<td>5.91</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">General performance improvements</td>
+ <td style="text-align: left">133</td>
+ <td style="text-align: left">65.52</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Refreshed WebUI</td>
+ <td style="text-align: left">102</td>
+ <td style="text-align: left">50.25</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Scheduler HA</td>
+ <td style="text-align: left">99</td>
+ <td style="text-align: left">48.77</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Official docker image</td>
+ <td style="text-align: left">84</td>
+ <td style="text-align: left">41.38</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">@task decorator</td>
+ <td style="text-align: left">56</td>
+ <td style="text-align: left">27.59</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Official helm chart</td>
+ <td style="text-align: left">51</td>
+ <td style="text-align: left">25.12</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Providers packages</td>
+ <td style="text-align: left">41</td>
+ <td style="text-align: left">20.2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Configurable XCom backends</td>
+ <td style="text-align: left">33</td>
+ <td style="text-align: left">16.26</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">CeleryKubernetesExecutor</td>
+ <td style="text-align: left">31</td>
+ <td style="text-align: left">15.27</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">5.91</td>
+ </tr>
+ </tbody>
</table>
<h2 id="summary">Summary</h2>
<p>From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
@@ -1929,7 +1939,7 @@ to share their experience and tools.</p>
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-survey-2020/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-survey-2020>
@@ -2081,13 +2091,13 @@ to share their experience and tools.</p>
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-survey-2022/index.html b/blog/airflow-survey-2022/index.html
index 29b29e0395..d68efdb536 100644
--- a/blog/airflow-survey-2022/index.html
+++ b/blog/airflow-survey-2022/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,36 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Airflow Survey 2022 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2022" />
-<meta property="og:description" content="2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey-2022/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2022-06-17T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Airflow Survey 2022 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-survey-2022/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Airflow Survey 2022">
+ <meta property="og:description" content="2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2022-06-17T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Community">
+ <meta property="article:tag" content="Survey">
+ <meta property="article:tag" content="Users">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Airflow Survey 2022">
-<meta itemprop="description" content="2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward."><meta itemprop="datePublished" content="2022-06-17T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="3865"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="community,survey,users," />
+ <meta itemprop="name" content="Airflow Survey 2022">
+ <meta itemprop="description" content="2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward.">
+ <meta itemprop="datePublished" content="2022-06-17T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="3865">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Community,Survey,Users">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +96,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -482,1644 +492,1644 @@
</ul>
<h2 id="overview-of-the-user-1">Overview of the user</h2>
<h3 id="what-best-describes-your-current-occupation-single-choice">What best describes your current occupation? (single choice)</h3>
-<p><img src="images/image1.png" alt="alt_text" title="user_occupations"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image1.png" title="user_occupations"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Data Engineer</td>
-<td>114</td>
-<td>54%</td>
-</tr>
-<tr>
-<td>Solutions Architect</td>
-<td>27</td>
-<td>13%</td>
-</tr>
-<tr>
-<td>Developer</td>
-<td>25</td>
-<td>12%</td>
-</tr>
-<tr>
-<td>DevOps</td>
-<td>12</td>
-<td>6%</td>
-</tr>
-<tr>
-<td>Data Scientist</td>
-<td>8</td>
-<td>4%</td>
-</tr>
-<tr>
-<td>Support Engineer</td>
-<td>5</td>
-<td>2%</td>
-</tr>
-<tr>
-<td>Data Analyst</td>
-<td>3</td>
-<td>1%</td>
-</tr>
-<tr>
-<td>Business Analyst</td>
-<td>2</td>
-<td>1%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>14</td>
-<td>7%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Data Engineer</td>
+ <td style="text-align: left">114</td>
+ <td style="text-align: left">54%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Solutions Architect</td>
+ <td style="text-align: left">27</td>
+ <td style="text-align: left">13%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Developer</td>
+ <td style="text-align: left">25</td>
+ <td style="text-align: left">12%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DevOps</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Data Scientist</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Support Engineer</td>
+ <td style="text-align: left">5</td>
+ <td style="text-align: left">2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Data Analyst</td>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Business Analyst</td>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">7%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey, more than half of Airflow users are Data Engineers (54%). Roles of the remaining Airflow users might be broken down into Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%). The 2022 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/">those from 2019</a> and <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> with a slight increase in the representation of Solu [...]
<h3 id="how-often-do-you-interact-with-airflow-single-choice">How often do you interact with Airflow? (single choice)</h3>
-<p><img src="images/image2.png" alt="alt_text" title="interaction_frequency"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image2.png" title="interaction_frequency"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Every day</td>
-<td>154</td>
-<td>73%</td>
-</tr>
-<tr>
-<td>At least once per week</td>
-<td>36</td>
-<td>17%</td>
-</tr>
-<tr>
-<td>At least once per month</td>
-<td>11</td>
-<td>5%</td>
-</tr>
-<tr>
-<td>Less than once per month</td>
-<td>9</td>
-<td>4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Every day</td>
+ <td style="text-align: left">154</td>
+ <td style="text-align: left">73%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">At least once per week</td>
+ <td style="text-align: left">36</td>
+ <td style="text-align: left">17%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">At least once per month</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Less than once per month</td>
+ <td style="text-align: left">9</td>
+ <td style="text-align: left">4%</td>
+ </tr>
+ </tbody>
</table>
<p>Users who took the survey are actively using Airflow as part of their current role. 73% of Airflow users who responded use it on a daily basis, 17% weekly.</p>
<h3 id="how-many-people-work-at-your-company-single-choice">How many people work at your company? (single choice)</h3>
-<p><img src="images/image3.png" alt="alt_text" title="company_size"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image3.png" title="company_size"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>201-5000</td>
-<td>85</td>
-<td>41%</td>
-</tr>
-<tr>
-<td>5000+</td>
-<td>49</td>
-<td>23%</td>
-</tr>
-<tr>
-<td>51-200</td>
-<td>46</td>
-<td>22%</td>
-</tr>
-<tr>
-<td>11-50</td>
-<td>20</td>
-<td>10%</td>
-</tr>
-<tr>
-<td>1-10</td>
-<td>9</td>
-<td>4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">201-5000</td>
+ <td style="text-align: left">85</td>
+ <td style="text-align: left">41%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">5000+</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">23%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">51-200</td>
+ <td style="text-align: left">46</td>
+ <td style="text-align: left">22%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">11-50</td>
+ <td style="text-align: left">20</td>
+ <td style="text-align: left">10%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1-10</td>
+ <td style="text-align: left">9</td>
+ <td style="text-align: left">4%</td>
+ </tr>
+ </tbody>
</table>
<p>Airflow is a framework that is used and popular in bigger companies, 64% of Airflow users who responded (compared to 52.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>) work for companies bigger than 200 employees (41% in companies size 201-5000 and 23% in companies size 5000+).</p>
<h3 id="how-many-people-at-your-company-use-airflow-single-choice">How many people at your company use Airflow? (single choice)</h3>
-<p><img src="images/image4.png" alt="alt_text" title="airflow_usage"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image4.png" title="airflow_usage"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>6-20</td>
-<td>80</td>
-<td>38%</td>
-</tr>
-<tr>
-<td>1-5</td>
-<td>61</td>
-<td>29%</td>
-</tr>
-<tr>
-<td>51-200</td>
-<td>49</td>
-<td>24%</td>
-</tr>
-<tr>
-<td>200+</td>
-<td>18</td>
-<td>9%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">6-20</td>
+ <td style="text-align: left">80</td>
+ <td style="text-align: left">38%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1-5</td>
+ <td style="text-align: left">61</td>
+ <td style="text-align: left">29%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">51-200</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">24%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">200+</td>
+ <td style="text-align: left">18</td>
+ <td style="text-align: left">9%</td>
+ </tr>
+ </tbody>
</table>
<p>Airflow is generally used by small to medium-sized teams. 62% of the survey participants have more than 6 Airflow users in their company (38% have between 6 and 200 users, 24% between 51-200 users).</p>
<h3 id="how-likely-are-you-to-recommend-apache-airflow-single-choice">How likely are you to recommend Apache Airflow? (single choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>% 2019</td>
-<td>% 2020</td>
-<td>% 2022</td>
-</tr>
-<tr>
-<td>Very Likely</td>
-<td>45.4%</td>
-<td>61.6%</td>
-<td>65.9%</td>
-</tr>
-<tr>
-<td>Likely</td>
-<td>40.3%</td>
-<td>30.4%</td>
-<td>26.9%</td>
-</tr>
-<tr>
-<td>Neutral</td>
-<td>10.7%</td>
-<td>5.4%</td>
-<td>6.3%</td>
-</tr>
-<tr>
-<td>Unlikely</td>
-<td>2.6%</td>
-<td>1.5%</td>
-<td>0.5%</td>
-</tr>
-<tr>
-<td>Very Unlikely</td>
-<td>1%</td>
-<td>1%</td>
-<td>0.5%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">% 2019</td>
+ <td style="text-align: left">% 2020</td>
+ <td style="text-align: left">% 2022</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Very Likely</td>
+ <td style="text-align: left">45.4%</td>
+ <td style="text-align: left">61.6%</td>
+ <td style="text-align: left">65.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Likely</td>
+ <td style="text-align: left">40.3%</td>
+ <td style="text-align: left">30.4%</td>
+ <td style="text-align: left">26.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Neutral</td>
+ <td style="text-align: left">10.7%</td>
+ <td style="text-align: left">5.4%</td>
+ <td style="text-align: left">6.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Unlikely</td>
+ <td style="text-align: left">2.6%</td>
+ <td style="text-align: left">1.5%</td>
+ <td style="text-align: left">0.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Very Unlikely</td>
+ <td style="text-align: left">1%</td>
+ <td style="text-align: left">1%</td>
+ <td style="text-align: left">0.5%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey, more Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow (92% in <a href="https://airflow.apache.org/blog/airflow-surve [...]
<h3 id="what-is-your-source-of-information-about-airflow-multiple-choice">What is your source of information about Airflow? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Documentation</td>
-<td>189</td>
-<td>90.4%</td>
-</tr>
-<tr>
-<td>Airflow website (Blog, etc.)</td>
-<td>142</td>
-<td>67.9%</td>
-</tr>
-<tr>
-<td>Stack Overflow</td>
-<td>126</td>
-<td>60.3%</td>
-</tr>
-<tr>
-<td>Github Issues</td>
-<td>104</td>
-<td>49.8%</td>
-</tr>
-<tr>
-<td>Slack</td>
-<td>96</td>
-<td>45.9%</td>
-</tr>
-<tr>
-<td>Airflow Summit Videos</td>
-<td>88</td>
-<td>42.1%</td>
-</tr>
-<tr>
-<td>GitHub Discussions</td>
-<td>76</td>
-<td>36.4%</td>
-</tr>
-<tr>
-<td>Airflow Community Webinars</td>
-<td>41</td>
-<td>19.6%</td>
-</tr>
-<tr>
-<td>Astronomer Registry</td>
-<td>51</td>
-<td>24.4%</td>
-</tr>
-<tr>
-<td>Airflow Mailing List</td>
-<td>34</td>
-<td>16.3%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Documentation</td>
+ <td style="text-align: left">189</td>
+ <td style="text-align: left">90.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Airflow website (Blog, etc.)</td>
+ <td style="text-align: left">142</td>
+ <td style="text-align: left">67.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Stack Overflow</td>
+ <td style="text-align: left">126</td>
+ <td style="text-align: left">60.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Github Issues</td>
+ <td style="text-align: left">104</td>
+ <td style="text-align: left">49.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Slack</td>
+ <td style="text-align: left">96</td>
+ <td style="text-align: left">45.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Airflow Summit Videos</td>
+ <td style="text-align: left">88</td>
+ <td style="text-align: left">42.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">GitHub Discussions</td>
+ <td style="text-align: left">76</td>
+ <td style="text-align: left">36.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Airflow Community Webinars</td>
+ <td style="text-align: left">41</td>
+ <td style="text-align: left">19.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Astronomer Registry</td>
+ <td style="text-align: left">51</td>
+ <td style="text-align: left">24.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Airflow Mailing List</td>
+ <td style="text-align: left">34</td>
+ <td style="text-align: left">16.3%</td>
+ </tr>
+ </tbody>
</table>
<p>Airflow documentation is a critical source of information, with more than 90% of survey participants using the documentation. It is of increasing importance compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> where documentation was at about 75% level. Moreover, more than 60% of users are getting information from the Airflow website (67.9% ) and Stack Overflow (60.3%) which is also a big increase compared to 36% level i [...]
<h2 id="deployments-1">Deployments</h2>
<h3 id="how-many-active-dags-do-you-have-in-your-largest-airflow-instance-single-choice">How many active DAGs do you have in your largest Airflow instance? (single choice)</h3>
-<p><img src="images/image5.png" alt="alt_text" title="active_dags"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image5.png" title="active_dags"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>51-250</td>
-<td>66</td>
-<td>31.7%</td>
-</tr>
-<tr>
-<td>11-50</td>
-<td>64</td>
-<td>30.8%</td>
-</tr>
-<tr>
-<td>5-10</td>
-<td>25</td>
-<td>12.0%</td>
-</tr>
-<tr>
-<td>251-500</td>
-<td>20</td>
-<td>9.6%</td>
-</tr>
-<tr>
-<td><5</td>
-<td>14</td>
-<td>6.7%</td>
-</tr>
-<tr>
-<td>1000+</td>
-<td>10</td>
-<td>4.8%</td>
-</tr>
-<tr>
-<td>501-1000</td>
-<td>9</td>
-<td>4.3%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">51-250</td>
+ <td style="text-align: left">66</td>
+ <td style="text-align: left">31.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">11-50</td>
+ <td style="text-align: left">64</td>
+ <td style="text-align: left">30.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">5-10</td>
+ <td style="text-align: left">25</td>
+ <td style="text-align: left">12.0%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">251-500</td>
+ <td style="text-align: left">20</td>
+ <td style="text-align: left">9.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><5</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">6.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1000+</td>
+ <td style="text-align: left">10</td>
+ <td style="text-align: left">4.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">501-1000</td>
+ <td style="text-align: left">9</td>
+ <td style="text-align: left">4.3%</td>
+ </tr>
+ </tbody>
</table>
<p>62.5% of the Airflow users surveyed have between 11 to 250 DAGs in their largest Airflow instance.</p>
<h3 id="how-many-active-airflow-instances-do-you-have-single-choice">How many active Airflow instances do you have? (single choice)</h3>
-<p><img src="images/image6.png" alt="alt_text" title="image_tooltip"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image6.png" title="image_tooltip"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>1</td>
-<td>52</td>
-<td>25.2%</td>
-</tr>
-<tr>
-<td>2</td>
-<td>46</td>
-<td>22.3%</td>
-</tr>
-<tr>
-<td>4-7</td>
-<td>40</td>
-<td>19.4%</td>
-</tr>
-<tr>
-<td>3</td>
-<td>37</td>
-<td>18.0%</td>
-</tr>
-<tr>
-<td>20+</td>
-<td>19</td>
-<td>9.2%</td>
-</tr>
-<tr>
-<td>8-10</td>
-<td>7</td>
-<td>3.4%</td>
-</tr>
-<tr>
-<td>11-20</td>
-<td>5</td>
-<td>2.4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">52</td>
+ <td style="text-align: left">25.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">46</td>
+ <td style="text-align: left">22.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">4-7</td>
+ <td style="text-align: left">40</td>
+ <td style="text-align: left">19.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">37</td>
+ <td style="text-align: left">18.0%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">20+</td>
+ <td style="text-align: left">19</td>
+ <td style="text-align: left">9.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">8-10</td>
+ <td style="text-align: left">7</td>
+ <td style="text-align: left">3.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">11-20</td>
+ <td style="text-align: left">5</td>
+ <td style="text-align: left">2.4%</td>
+ </tr>
+ </tbody>
</table>
<p>85% of the Airflow users surveyed have between 1 and 7 active Airflow instances, and nearly 50% have only 1 or 2.</p>
<h3 id="what-is-the-maximum-number-of-tasks-that-you-have-used-in-a-single-dagsingle-choice">What is the maximum number of tasks that you have used in a single DAG?(single choice)</h3>
-<p><img src="images/image7.png" alt="alt_text" title="maximum tasks"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image7.png" title="maximum tasks"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>11-25</td>
-<td>51</td>
-<td>24.5%</td>
-</tr>
-<tr>
-<td>26-50</td>
-<td>41</td>
-<td>19.7%</td>
-</tr>
-<tr>
-<td>51-100</td>
-<td>35</td>
-<td>16.8%</td>
-</tr>
-<tr>
-<td><10</td>
-<td>29</td>
-<td>13.9%</td>
-</tr>
-<tr>
-<td>101-250</td>
-<td>23</td>
-<td>11.1%</td>
-</tr>
-<tr>
-<td>501-1000</td>
-<td>9</td>
-<td>4.3%</td>
-</tr>
-<tr>
-<td>1000-2500</td>
-<td>8</td>
-<td>3.8%</td>
-</tr>
-<tr>
-<td>251-500</td>
-<td>8</td>
-<td>3.8%</td>
-</tr>
-<tr>
-<td>2500-5000</td>
-<td>4</td>
-<td>1.9%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">11-25</td>
+ <td style="text-align: left">51</td>
+ <td style="text-align: left">24.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">26-50</td>
+ <td style="text-align: left">41</td>
+ <td style="text-align: left">19.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">51-100</td>
+ <td style="text-align: left">35</td>
+ <td style="text-align: left">16.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><10</td>
+ <td style="text-align: left">29</td>
+ <td style="text-align: left">13.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">101-250</td>
+ <td style="text-align: left">23</td>
+ <td style="text-align: left">11.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">501-1000</td>
+ <td style="text-align: left">9</td>
+ <td style="text-align: left">4.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1000-2500</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">3.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">251-500</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">3.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2500-5000</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">1.9%</td>
+ </tr>
+ </tbody>
</table>
<p>75% of the surveyed Airflow users have between 1 and 100 tasks per DAG.</p>
<h3 id="how-many-schedulers-do-you-have-in-your-largest-airflow-instance-single-choice">How many schedulers do you have in your largest Airflow instance? (single choice)</h3>
-<p><img src="images/image8.png" alt="alt_text" title="max_schedulers"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image8.png" title="max_schedulers"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>1</td>
-<td>113</td>
-<td>55.1%</td>
-</tr>
-<tr>
-<td>2</td>
-<td>61</td>
-<td>29.8%</td>
-</tr>
-<tr>
-<td>3</td>
-<td>18</td>
-<td>8.8%</td>
-</tr>
-<tr>
-<td>4+</td>
-<td>13</td>
-<td>6.3%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">113</td>
+ <td style="text-align: left">55.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">61</td>
+ <td style="text-align: left">29.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">18</td>
+ <td style="text-align: left">8.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">4+</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6.3%</td>
+ </tr>
+ </tbody>
</table>
<p>More than half of Airflow users who responded to the survey have 1 scheduler in their largest Airflow instance, however it’s important to notice that the second half of Airflow users decided to have 2 schedulers and more.</p>
<h3 id="what-executor-type-do-you-use-multiple-choice">What executor type do you use? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Celery</td>
-<td>107</td>
-<td>52.7 %</td>
-</tr>
-<tr>
-<td>Kubernetes</td>
-<td>80</td>
-<td>39.4%</td>
-</tr>
-<tr>
-<td>Local</td>
-<td>49</td>
-<td>24.1%</td>
-</tr>
-<tr>
-<td>Sequential</td>
-<td>21</td>
-<td>10.3%</td>
-</tr>
-<tr>
-<td>CeleryKubernetes</td>
-<td>14</td>
-<td>6.9%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Celery</td>
+ <td style="text-align: left">107</td>
+ <td style="text-align: left">52.7 %</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Kubernetes</td>
+ <td style="text-align: left">80</td>
+ <td style="text-align: left">39.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Local</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">24.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Sequential</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">10.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">CeleryKubernetes</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">6.9%</td>
+ </tr>
+ </tbody>
</table>
<p>Celery (52.7%) and Kubernetes (39.4%) are the most common executors used. CeleryKubernetes (6.9%) executor also started to be noticed and used by Airflow users.</p>
<h3 id="if-you-use-the-celery-executor-how-many-workers-do-you-have-in-your-largest-airflow-instance-single-choice">If you use the Celery executor, how many workers do you have in your largest Airflow instance? (single choice)</h3>
-<p><img src="images/image9.png" alt="alt_text" title="max_workers"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image9.png" title="max_workers"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>2-5</td>
-<td>64</td>
-<td>44.8%</td>
-</tr>
-<tr>
-<td>10+</td>
-<td>28</td>
-<td>19.6%</td>
-</tr>
-<tr>
-<td>1</td>
-<td>26</td>
-<td>18.2%</td>
-</tr>
-<tr>
-<td>6-10</td>
-<td>25</td>
-<td>17.5%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2-5</td>
+ <td style="text-align: left">64</td>
+ <td style="text-align: left">44.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">10+</td>
+ <td style="text-align: left">28</td>
+ <td style="text-align: left">19.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">26</td>
+ <td style="text-align: left">18.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">6-10</td>
+ <td style="text-align: left">25</td>
+ <td style="text-align: left">17.5%</td>
+ </tr>
+ </tbody>
</table>
<p>Amongst Celery executor users who responded to the survey, close to half the number (44.8%) have between 2 to 5 workers in their largest Airflow instance. It’s notable that nearly a fifth (19.6%) have more than 10 workers.</p>
<h3 id="which-version-of-airflow-do-you-currently-use-single-choice">Which version of Airflow do you currently use? (single choice)</h3>
-<p><img src="images/image10.png" alt="alt_text" title="airflow_version"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image10.png" title="airflow_version"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>1.10.14 or older</td>
-<td>13</td>
-<td>6.3%</td>
-</tr>
-<tr>
-<td>1.10.15</td>
-<td>19</td>
-<td>9.2%</td>
-</tr>
-<tr>
-<td>2.0.x</td>
-<td>23</td>
-<td>11.1%</td>
-</tr>
-<tr>
-<td>2.1.x</td>
-<td>24</td>
-<td>11.6%</td>
-</tr>
-<tr>
-<td>2.2.x</td>
-<td>79</td>
-<td>38.2%</td>
-</tr>
-<tr>
-<td>2.3.x</td>
-<td>49</td>
-<td>23.7%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.14 or older</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1.10.15</td>
+ <td style="text-align: left">19</td>
+ <td style="text-align: left">9.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2.0.x</td>
+ <td style="text-align: left">23</td>
+ <td style="text-align: left">11.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2.1.x</td>
+ <td style="text-align: left">24</td>
+ <td style="text-align: left">11.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2.2.x</td>
+ <td style="text-align: left">79</td>
+ <td style="text-align: left">38.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">2.3.x</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">23.7%</td>
+ </tr>
+ </tbody>
</table>
<p>It’s good to see that close to 85% of users who responded to the survey use one of the Airflow 2 versions, 9.2% users still use 1.10.15, while the remaining 6.3% are still using older Airflow 1.10 versions.</p>
<p>The good news is that the majority of users on Airflow 1 are planning migration to Airflow 2 quite soon, as for now they have capacity constraints to undertake such a significant effort in their opinion. However, it can also be noticed in the survey’s comments that some users are generally skeptical towards migration to Airflow 2, they have negative opinions about the new scheduler or compatibility with the helm chart.</p>
<p>As to plans about migration to the newest version of Airflow 2, users who responded to the survey are committed and waiting especially for the features related to dynamic DAGs. However, some users also reported that they are waiting to solve some dependencies they have or they prefer to wait a little bit more for the community to test the new version before they decide to move on.</p>
<h3 id="what-metrics-do-you-use-to-monitor-airflow-multiple-choice">What metrics do you use to monitor Airflow? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>External monitoring service</td>
-<td>81</td>
-<td>40.7%</td>
-</tr>
-<tr>
-<td>Information from metadatabase</td>
-<td>71</td>
-<td>35.7%</td>
-</tr>
-<tr>
-<td>Statsd</td>
-<td>54</td>
-<td>27.1%</td>
-</tr>
-<tr>
-<td>I do not use monitoring</td>
-<td>47</td>
-<td>23.6%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>14</td>
-<td>7%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">External monitoring service</td>
+ <td style="text-align: left">81</td>
+ <td style="text-align: left">40.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Information from metadatabase</td>
+ <td style="text-align: left">71</td>
+ <td style="text-align: left">35.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Statsd</td>
+ <td style="text-align: left">54</td>
+ <td style="text-align: left">27.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I do not use monitoring</td>
+ <td style="text-align: left">47</td>
+ <td style="text-align: left">23.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">7%</td>
+ </tr>
+ </tbody>
</table>
<p>In comparison to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>, more users are monitoring airflow in some way. External monitoring services (40.7%) and information from metabase (35.7%) started to play a more important role in Airflow monitoring.</p>
<h3 id="how-do-you-deploy-airflow-multiple-choice">How do you deploy Airflow? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>On virtual machines (for example using AWS EC2)</td>
-<td>63</td>
-<td>30.6 %</td>
-</tr>
-<tr>
-<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
-<td>54</td>
-<td>26.2 %</td>
-</tr>
-<tr>
-<td>On Kubernetes (using Apache Airflow’s helm chart)</td>
-<td>46</td>
-<td>22.3%</td>
-</tr>
-<tr>
-<td>On premises</td>
-<td>43</td>
-<td>20.9%</td>
-</tr>
-<tr>
-<td>On Kubernetes (using custom deployments)</td>
-<td>39</td>
-<td>18.9%</td>
-</tr>
-<tr>
-<td>On Kubernetes (using another helm chart)</td>
-<td>21</td>
-<td>10.2%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>13</td>
-<td>6.5%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On virtual machines (for example using AWS EC2)</td>
+ <td style="text-align: left">63</td>
+ <td style="text-align: left">30.6 %</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
+ <td style="text-align: left">54</td>
+ <td style="text-align: left">26.2 %</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On Kubernetes (using Apache Airflow’s helm chart)</td>
+ <td style="text-align: left">46</td>
+ <td style="text-align: left">22.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On premises</td>
+ <td style="text-align: left">43</td>
+ <td style="text-align: left">20.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On Kubernetes (using custom deployments)</td>
+ <td style="text-align: left">39</td>
+ <td style="text-align: left">18.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">On Kubernetes (using another helm chart)</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">10.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6.5%</td>
+ </tr>
+ </tbody>
</table>
<p>More than half of Airflow users who responded (51.4%) deploy Airflow on Kubernetes. This is about 20 percent more than in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>. The remaining top deployment methods are on virtual machines (30.6%) and via managed services (26.2%).</p>
<h3 id="how-do-you-distribute-your-dags-from-your-developer-environment-to-the-cloud-single-choice">How do you distribute your DAGs from your developer environment to the cloud? (single choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
-<td>100</td>
-<td>49%</td>
-</tr>
-<tr>
-<td>Bake them into the docker image</td>
-<td>51</td>
-<td>25%</td>
-</tr>
-<tr>
-<td>Shared files system</td>
-<td>30</td>
-<td>14.7%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>16</td>
-<td>7.9%</td>
-</tr>
-<tr>
-<td>I don’t know</td>
-<td>7</td>
-<td>3.4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using a synchronizing process (Git sync, GCS fuse, etc)</td>
+ <td style="text-align: left">100</td>
+ <td style="text-align: left">49%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Bake them into the docker image</td>
+ <td style="text-align: left">51</td>
+ <td style="text-align: left">25%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Shared files system</td>
+ <td style="text-align: left">30</td>
+ <td style="text-align: left">14.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">16</td>
+ <td style="text-align: left">7.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t know</td>
+ <td style="text-align: left">7</td>
+ <td style="text-align: left">3.4%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey responses, the most popular way of distributing DAGs is a synchronizing process, about half of Airflow users (49%) use this process to distribute DAGs from developer environments to the cloud.</p>
<h2 id="usage-1">Usage</h2>
<h3 id="do-you-have-any-customisation-of-airflow-single-choice">Do you have any customisation of Airflow? (single choice)</h3>
-<p><img src="images/image11.png" alt="alt_text" title="customization"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image11.png" title="customization"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>No, we use vanilla airflow</td>
-<td>165</td>
-<td>81.3%</td>
-</tr>
-<tr>
-<td>Yes, we have a separate fork</td>
-<td>13</td>
-<td>6.4%</td>
-</tr>
-<tr>
-<td>Yes, we use a 3rd-party fork</td>
-<td>12</td>
-<td>5.9%</td>
-</tr>
-<tr>
-<td>Yes, we’ve backpropagated bug fixes to an older version</td>
-<td>13</td>
-<td>6.4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No, we use vanilla airflow</td>
+ <td style="text-align: left">165</td>
+ <td style="text-align: left">81.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, we have a separate fork</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, we use a 3rd-party fork</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">5.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, we’ve backpropagated bug fixes to an older version</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6.4%</td>
+ </tr>
+ </tbody>
</table>
<p>More Airflow users (81.3%) don’t have any customisation of Airflow (compared to 75.9% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>). Those Airflow users who have customisations (18.7%) decided to introduce them mainly to separate development and production workflows, to backport bug fixes, due to security fixes or to run a backfill command on Kubernetes pod.</p>
<h3 id="which-metadata-database-do-you-use-single-choice">Which Metadata Database do you use? (single choice)</h3>
-<p><img src="images/image12.png" alt="alt_text" title="database"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image12.png" title="database"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%I</td>
-</tr>
-<tr>
-<td>PostgreSQL 13</td>
-<td>86</td>
-<td>43.9%</td>
-</tr>
-<tr>
-<td>PostgreSQL 12</td>
-<td>74</td>
-<td>37.8%</td>
-</tr>
-<tr>
-<td>MySQL 8</td>
-<td>22</td>
-<td>11.2%</td>
-</tr>
-<tr>
-<td>MySQL 5</td>
-<td>9</td>
-<td>4.6%</td>
-</tr>
-<tr>
-<td>MariaDB</td>
-<td>4</td>
-<td>2.0%</td>
-</tr>
-<tr>
-<td>MsSQL</td>
-<td>1</td>
-<td>0.5%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%I</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">PostgreSQL 13</td>
+ <td style="text-align: left">86</td>
+ <td style="text-align: left">43.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">PostgreSQL 12</td>
+ <td style="text-align: left">74</td>
+ <td style="text-align: left">37.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">MySQL 8</td>
+ <td style="text-align: left">22</td>
+ <td style="text-align: left">11.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">MySQL 5</td>
+ <td style="text-align: left">9</td>
+ <td style="text-align: left">4.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">MariaDB</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">2.0%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">MsSQL</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.5%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey responses, the most popular metadata databases are PostgreSQL 13 (43.9%) and PostgreSQL 12 (37.8%). This represents a sharp increase from 2020, up from 68.9% to 81.7% total on PostgreSQL, with a corresponding decrease in MySQL, down from 23% to 15%. This is an interesting result taking into account community discussion about not adding support for more database backend or even deciding on single database support.</p>
<h3 id="whats-the-primary-method-by-which-you-integrate-with-providers-and-external-services-in-your-airflow-dags-single-choice">What’s the primary method by which you integrate with providers and external services in your Airflow DAGs? (single choice)</h3>
-<p><img src="images/image13.png" alt="alt_text" title="providers_interface"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image13.png" title="providers_interface"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Using existing dedicated operators / hooks</td>
-<td>70</td>
-<td>34.5%</td>
-</tr>
-<tr>
-<td>Using Bash/Python operators</td>
-<td>58</td>
-<td>28.6%</td>
-</tr>
-<tr>
-<td>Using custom operators / hooks</td>
-<td>50</td>
-<td>24.6%</td>
-</tr>
-<tr>
-<td>Using KubernetesPodOperator</td>
-<td>25</td>
-<td>12.3%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using existing dedicated operators / hooks</td>
+ <td style="text-align: left">70</td>
+ <td style="text-align: left">34.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using Bash/Python operators</td>
+ <td style="text-align: left">58</td>
+ <td style="text-align: left">28.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using custom operators / hooks</td>
+ <td style="text-align: left">50</td>
+ <td style="text-align: left">24.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Using KubernetesPodOperator</td>
+ <td style="text-align: left">25</td>
+ <td style="text-align: left">12.3%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey responses, the following ways of using Airflow to connect to external services are the most popular: Using existing dedicated operators / hooks (34.5%), Using Bash/Python operators (28.6%), Using custom operators / hooks (24.6%). Using KubernetesPodOperator (12.3%) is less popular regarding the survey responses. The integration with providers and external services methods ranking is similar to the one from <a href="https://airflow.apache.org/blog/airflow-survey [...]
<h3 id="what-providers-do-you-use-in-your-airflow-dags-multiple-choice">What providers do you use in your Airflow DAGs? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Amazon Web Services</td>
-<td>112</td>
-<td>55.4%</td>
-</tr>
-<tr>
-<td>Google Cloud Platform / Google APIs</td>
-<td>79</td>
-<td>39.1%</td>
-</tr>
-<tr>
-<td>Internal company systems</td>
-<td>75</td>
-<td>37.1%</td>
-</tr>
-<tr>
-<td>Hadoop / Spark / Flink / Other Apache software</td>
-<td>57</td>
-<td>28.2%</td>
-</tr>
-<tr>
-<td>Microsoft Azure</td>
-<td>17</td>
-<td>8.4%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>21</td>
-<td>10.5%</td>
-</tr>
-<tr>
-<td>I do not use external services in my Airflow DAGs</td>
-<td>14</td>
-<td>6.9%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Amazon Web Services</td>
+ <td style="text-align: left">112</td>
+ <td style="text-align: left">55.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Google Cloud Platform / Google APIs</td>
+ <td style="text-align: left">79</td>
+ <td style="text-align: left">39.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Internal company systems</td>
+ <td style="text-align: left">75</td>
+ <td style="text-align: left">37.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Hadoop / Spark / Flink / Other Apache software</td>
+ <td style="text-align: left">57</td>
+ <td style="text-align: left">28.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Microsoft Azure</td>
+ <td style="text-align: left">17</td>
+ <td style="text-align: left">8.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">10.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I do not use external services in my Airflow DAGs</td>
+ <td style="text-align: left">14</td>
+ <td style="text-align: left">6.9%</td>
+ </tr>
+ </tbody>
</table>
<p>It’s not surprising that Amazon Web Services (55.4% vs 59.6% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/">2020</a>), on the next three positions Google Cloud Platform (39.1% vs 47.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/">2020</a> ), Internal company systems (37.1% vs 55.6% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/">2020</a>), and other Apache products (28.2% vs 35.47% in <a href="https://airflow.apache.org/blog/ai [...]
<h3 id="how-frequently-do-you-upgrade-airflow-environments-single-choice">How frequently do you upgrade Airflow environments? (single choice)</h3>
-<p><img src="images/image14.png" alt="alt_text" title="upgrade_frequency"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image14.png" title="upgrade_frequency"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>every 12 months</td>
-<td>46</td>
-<td>22.9%</td>
-</tr>
-<tr>
-<td>every 6 months</td>
-<td>49</td>
-<td>24.4%</td>
-</tr>
-<tr>
-<td>once a quarter</td>
-<td>47</td>
-<td>23.4%</td>
-</tr>
-<tr>
-<td>Whenever there is a newer version</td>
-<td>59</td>
-<td>29.4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">every 12 months</td>
+ <td style="text-align: left">46</td>
+ <td style="text-align: left">22.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">every 6 months</td>
+ <td style="text-align: left">49</td>
+ <td style="text-align: left">24.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">once a quarter</td>
+ <td style="text-align: left">47</td>
+ <td style="text-align: left">23.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Whenever there is a newer version</td>
+ <td style="text-align: left">59</td>
+ <td style="text-align: left">29.4%</td>
+ </tr>
+ </tbody>
</table>
<p>Different frequencies of Airflow environments upgrades are almost equally popular amongst Airflow users who responded to the survey.</p>
<h3 id="do-you-upgrade-providers-separately-from-the-core-single-choice">Do you upgrade providers separately from the core? (single choice)</h3>
-<p><img src="images/image15.png" alt="alt_text" title="providers_upgrade"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image15.png" title="providers_upgrade"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>When I need it</td>
-<td>83</td>
-<td>42.8%</td>
-</tr>
-<tr>
-<td>Never - always use the providers that come with Airflow</td>
-<td>68</td>
-<td>35.1%</td>
-</tr>
-<tr>
-<td>I did not know I can upgrade providers separately</td>
-<td>32</td>
-<td>16.5%</td>
-</tr>
-<tr>
-<td>I upgrade providers when they are released</td>
-<td>11</td>
-<td>5.7%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">When I need it</td>
+ <td style="text-align: left">83</td>
+ <td style="text-align: left">42.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Never - always use the providers that come with Airflow</td>
+ <td style="text-align: left">68</td>
+ <td style="text-align: left">35.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I did not know I can upgrade providers separately</td>
+ <td style="text-align: left">32</td>
+ <td style="text-align: left">16.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I upgrade providers when they are released</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">5.7%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey responses, Airflow users most often upgrade providers when they need it (42.8%) or prefer to stay with providers that come with Airflow (35.1%). It’s surprising that 16.5% of Airflow users who responded to the survey were not aware that they can upgrade their providers separately from the core Airflow.</p>
<h3 id="how-do-you-pass-inputs-and-outputs-between-tasks-multiple-choice">How do you pass inputs and outputs between tasks? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Xcom</td>
-<td>141</td>
-<td>69.8%</td>
-</tr>
-<tr>
-<td>Saving and retrieving from Storage</td>
-<td>99</td>
-<td>49%</td>
-</tr>
-<tr>
-<td>TaskFlow</td>
-<td>37</td>
-<td>18.3%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>5</td>
-<td>2.5%</td>
-</tr>
-<tr>
-<td>We don’t</td>
-<td>29</td>
-<td>14.4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Xcom</td>
+ <td style="text-align: left">141</td>
+ <td style="text-align: left">69.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Saving and retrieving from Storage</td>
+ <td style="text-align: left">99</td>
+ <td style="text-align: left">49%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">TaskFlow</td>
+ <td style="text-align: left">37</td>
+ <td style="text-align: left">18.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">5</td>
+ <td style="text-align: left">2.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">We don’t</td>
+ <td style="text-align: left">29</td>
+ <td style="text-align: left">14.4%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey responses, Xcom (69.8%) is the most popular method to pass inputs and outputs between tasks, however Saving and Retrieving Inputs and Outputs from Storage still plays an important role (49%). It’s interesting that close to 15% of Airflow users who responded to the survey declare to not pass any outputs or inputs between tasks.</p>
<h3 id="do-you-use-a-data-lineage-backend-multiple-choice">Do you use a data lineage backend? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>No, but I will use such feature if fully supported in Airflow</td>
-<td>95</td>
-<td>47.5%</td>
-</tr>
-<tr>
-<td>I’m not familiar with data lineage</td>
-<td>58</td>
-<td>29%</td>
-</tr>
-<tr>
-<td>No, data lineage isn’t a concern for my usage</td>
-<td>26</td>
-<td>13%</td>
-</tr>
-<tr>
-<td>Yes, I send lineage to an Open Source lineage repository</td>
-<td>15</td>
-<td>7.5%</td>
-</tr>
-<tr>
-<td>Yes, I send lineage to an Enterprise lineage repository</td>
-<td>7</td>
-<td>3.5%</td>
-</tr>
-<tr>
-<td>Yes, I send lineage to a custom internal lineage repository</td>
-<td>9</td>
-<td>4.5%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No, but I will use such feature if fully supported in Airflow</td>
+ <td style="text-align: left">95</td>
+ <td style="text-align: left">47.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I’m not familiar with data lineage</td>
+ <td style="text-align: left">58</td>
+ <td style="text-align: left">29%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">No, data lineage isn’t a concern for my usage</td>
+ <td style="text-align: left">26</td>
+ <td style="text-align: left">13%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, I send lineage to an Open Source lineage repository</td>
+ <td style="text-align: left">15</td>
+ <td style="text-align: left">7.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, I send lineage to an Enterprise lineage repository</td>
+ <td style="text-align: left">7</td>
+ <td style="text-align: left">3.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, I send lineage to a custom internal lineage repository</td>
+ <td style="text-align: left">9</td>
+ <td style="text-align: left">4.5%</td>
+ </tr>
+ </tbody>
</table>
<p>When asked what lineage backend Airflow users use, the answers indicated that, while lineage itself is a quite new topic, there is interest in the feature as a whole. Most Airflow users responded that they don’t use lineage solutions currently but might be interested in the future if supported by Airflow (47.5%), are not familiar with data lineage (29%) or that data lineage is not their concern (13%).</p>
<h3 id="which-interfaces-of-airflow-do-you-use-as-part-of-your-current-role-multiple-choice">Which interfaces of Airflow do you use as part of your current role? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Original Airflow Graphical User Interface</td>
-<td>189</td>
-<td>94%</td>
-</tr>
-<tr>
-<td>CLI</td>
-<td>98</td>
-<td>48.8%</td>
-</tr>
-<tr>
-<td>API</td>
-<td>80</td>
-<td>39.8%</td>
-</tr>
-<tr>
-<td>Custom (own created) Airflow Graphical User Interface</td>
-<td>12</td>
-<td>6%</td>
-</tr>
-<tr>
-<td>GCP Composer</td>
-<td>1</td>
-<td>0.5%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Original Airflow Graphical User Interface</td>
+ <td style="text-align: left">189</td>
+ <td style="text-align: left">94%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">CLI</td>
+ <td style="text-align: left">98</td>
+ <td style="text-align: left">48.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">API</td>
+ <td style="text-align: left">80</td>
+ <td style="text-align: left">39.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Custom (own created) Airflow Graphical User Interface</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">GCP Composer</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.5%</td>
+ </tr>
+ </tbody>
</table>
<p>It’s clear that usage of Airflow web UI is important as 94% of users who responded to the survey declare to use it as a part of their current role. Usage of CLI (48.8%) and API (39.8%) goes in pairs but are not so common compared to Airflow web UI usage.</p>
<h3 id="if-gui-marked-what-do-you-use-the-gui-for-multiple-choice">(If GUI Marked) What do you use the GUI for? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Monitoring Runs</td>
-<td>188</td>
-<td>95.9%</td>
-</tr>
-<tr>
-<td>Accessing Task Logs</td>
-<td>176</td>
-<td>89.8%</td>
-</tr>
-<tr>
-<td>Manually triggering DAGs</td>
-<td>167</td>
-<td>85.2%</td>
-</tr>
-<tr>
-<td>Clearing Tasks</td>
-<td>162</td>
-<td>82.7%</td>
-</tr>
-<tr>
-<td>Marking Tasks as successful</td>
-<td>119</td>
-<td>60.7%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>6</td>
-<td>3%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Monitoring Runs</td>
+ <td style="text-align: left">188</td>
+ <td style="text-align: left">95.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Accessing Task Logs</td>
+ <td style="text-align: left">176</td>
+ <td style="text-align: left">89.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Manually triggering DAGs</td>
+ <td style="text-align: left">167</td>
+ <td style="text-align: left">85.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Clearing Tasks</td>
+ <td style="text-align: left">162</td>
+ <td style="text-align: left">82.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Marking Tasks as successful</td>
+ <td style="text-align: left">119</td>
+ <td style="text-align: left">60.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">6</td>
+ <td style="text-align: left">3%</td>
+ </tr>
+ </tbody>
</table>
<p>Airflow web UI is used heavily for monitoring: Monitoring Runs (95.9%) and troubleshooting: Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%).</p>
<h3 id="if-cli-marked-what-do-you-use-the-cli-for-multiple-choice">(if CLI Marked) What do you use the CLI For? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Backfilling</td>
-<td>63</td>
-<td>56.8%</td>
-</tr>
-<tr>
-<td>Manually triggering DAGs</td>
-<td>52</td>
-<td>46.8%</td>
-</tr>
-<tr>
-<td>Clearing Tasks</td>
-<td>26</td>
-<td>23.4%</td>
-</tr>
-<tr>
-<td>Monitoring Runs</td>
-<td>25</td>
-<td>22.5%</td>
-</tr>
-<tr>
-<td>Accessing Task Logs</td>
-<td>21</td>
-<td>18.9%</td>
-</tr>
-<tr>
-<td>Marking Tasks as successful</td>
-<td>11</td>
-<td>9.9%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>17</td>
-<td>15.3%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Backfilling</td>
+ <td style="text-align: left">63</td>
+ <td style="text-align: left">56.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Manually triggering DAGs</td>
+ <td style="text-align: left">52</td>
+ <td style="text-align: left">46.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Clearing Tasks</td>
+ <td style="text-align: left">26</td>
+ <td style="text-align: left">23.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Monitoring Runs</td>
+ <td style="text-align: left">25</td>
+ <td style="text-align: left">22.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Accessing Task Logs</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">18.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Marking Tasks as successful</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">9.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">17</td>
+ <td style="text-align: left">15.3%</td>
+ </tr>
+ </tbody>
</table>
<p>Compared to Airflow web UI, Airflow CLI is used mainly for Backfilling (56.8%) and Manually triggering DAGs (46.8%).</p>
<h3 id="in-airflow-which-ui-views-are-important-for-you-multiple-choice">In Airflow, which UI views are important for you? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>List of DAGs</td>
-<td>178</td>
-<td>89.4%</td>
-</tr>
-<tr>
-<td>Task Logs</td>
-<td>162</td>
-<td>81.4%</td>
-</tr>
-<tr>
-<td>DAG Runs</td>
-<td>160</td>
-<td>80.4%</td>
-</tr>
-<tr>
-<td>Graph view</td>
-<td>147</td>
-<td>73.9%</td>
-</tr>
-<tr>
-<td>Grid/Tree View</td>
-<td>138</td>
-<td>69.3%</td>
-</tr>
-<tr>
-<td>Run Details</td>
-<td>117</td>
-<td>58.8%</td>
-</tr>
-<tr>
-<td>DAG details</td>
-<td>111</td>
-<td>55.8%</td>
-</tr>
-<tr>
-<td>Task Instances</td>
-<td>102</td>
-<td>51.3%</td>
-</tr>
-<tr>
-<td>Task Duration</td>
-<td>91</td>
-<td>45.7%</td>
-</tr>
-<tr>
-<td>Code</td>
-<td>90</td>
-<td>45.2%</td>
-</tr>
-<tr>
-<td>Task Tries</td>
-<td>60</td>
-<td>30.2%</td>
-</tr>
-<tr>
-<td>Gantt</td>
-<td>48</td>
-<td>21.4%</td>
-</tr>
-<tr>
-<td>Landing Times</td>
-<td>27</td>
-<td>13.6%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>4</td>
-<td>2%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">List of DAGs</td>
+ <td style="text-align: left">178</td>
+ <td style="text-align: left">89.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Task Logs</td>
+ <td style="text-align: left">162</td>
+ <td style="text-align: left">81.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DAG Runs</td>
+ <td style="text-align: left">160</td>
+ <td style="text-align: left">80.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Graph view</td>
+ <td style="text-align: left">147</td>
+ <td style="text-align: left">73.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Grid/Tree View</td>
+ <td style="text-align: left">138</td>
+ <td style="text-align: left">69.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Run Details</td>
+ <td style="text-align: left">117</td>
+ <td style="text-align: left">58.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DAG details</td>
+ <td style="text-align: left">111</td>
+ <td style="text-align: left">55.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Task Instances</td>
+ <td style="text-align: left">102</td>
+ <td style="text-align: left">51.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Task Duration</td>
+ <td style="text-align: left">91</td>
+ <td style="text-align: left">45.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Code</td>
+ <td style="text-align: left">90</td>
+ <td style="text-align: left">45.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Task Tries</td>
+ <td style="text-align: left">60</td>
+ <td style="text-align: left">30.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Gantt</td>
+ <td style="text-align: left">48</td>
+ <td style="text-align: left">21.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Landing Times</td>
+ <td style="text-align: left">27</td>
+ <td style="text-align: left">13.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">2%</td>
+ </tr>
+ </tbody>
</table>
<p>UI views importance ranking shows that the majority Airflow users use Web UI mostly for monitoring and/or troubleshooting purposes, where the top 3 views are List of DAGs (89.4%), Task Logs (81.4%) and DAG Runs (80.4%). The results are very similar to those from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>.</p>
<h2 id="community-and-contribution-1">Community and contribution</h2>
<h3 id="are-you-participating-in-the-airflow-community-discussions-single-choice">Are you participating in the Airflow community discussions? (single choice)</h3>
-<p><img src="images/image16.png" alt="alt_text" title="discussions_engagement"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image16.png" title="discussions_engagement"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>I see them from time to time</td>
-<td>99</td>
-<td>48.3%</td>
-</tr>
-<tr>
-<td>I regularly follow what’s being discussed but don’t participate</td>
-<td>53</td>
-<td>25.9%</td>
-</tr>
-<tr>
-<td>I didn’t know I could</td>
-<td>41</td>
-<td>20.0%</td>
-</tr>
-<tr>
-<td>I actively participate in the discussions</td>
-<td>12</td>
-<td>5.9%</td>
-</tr>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>I know I can but I do not contribute</td>
-<td>116</td>
-<td>57.1%</td>
-</tr>
-<tr>
-<td>Very rarely when it relates to what I need</td>
-<td>44</td>
-<td>21.7%</td>
-</tr>
-<tr>
-<td>I do not know I could</td>
-<td>30</td>
-<td>14.8%</td>
-</tr>
-<tr>
-<td>I regularly contribute by discussing, reviewing and submitting PR</td>
-<td>13</td>
-<td>6.4%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I see them from time to time</td>
+ <td style="text-align: left">99</td>
+ <td style="text-align: left">48.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I regularly follow what’s being discussed but don’t participate</td>
+ <td style="text-align: left">53</td>
+ <td style="text-align: left">25.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I didn’t know I could</td>
+ <td style="text-align: left">41</td>
+ <td style="text-align: left">20.0%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I actively participate in the discussions</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">5.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I know I can but I do not contribute</td>
+ <td style="text-align: left">116</td>
+ <td style="text-align: left">57.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Very rarely when it relates to what I need</td>
+ <td style="text-align: left">44</td>
+ <td style="text-align: left">21.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I do not know I could</td>
+ <td style="text-align: left">30</td>
+ <td style="text-align: left">14.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I regularly contribute by discussing, reviewing and submitting PR</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">6.4%</td>
+ </tr>
+ </tbody>
</table>
<p>Results related to the Airflow contribution are very similar to those about participating in the Airflow community discussions. Most of the Airflow users (57.1%) who responded to the survey are aware but do not contribute or contribute very rarely (21.7%). 14.8% of users were not aware they could contribute. Once again, it’s a clear indicator that there is much more to be done to engage our community to be more active contributors and raise the current 6.4% of users who actively contr [...]
<h3 id="if-you-do-not-contribute---why">If you do not contribute - why?</h3>
-<p><img src="images/image18.png" alt="alt_text" title="contribution_reasons"></p>
+<p><img alt="alt_text" src="/blog/airflow-survey-2022/images/image18.png" title="contribution_reasons"></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>I have no time to contribute even if would like to</td>
-<td>65</td>
-<td>38.9%</td>
-</tr>
-<tr>
-<td>I don’t know how to start</td>
-<td>63</td>
-<td>37.7%</td>
-</tr>
-<tr>
-<td>I don’t have a need to contribute</td>
-<td>19</td>
-<td>11.4%</td>
-</tr>
-<tr>
-<td>I didn’t know I could</td>
-<td>12</td>
-<td>7.2%</td>
-</tr>
-<tr>
-<td>My employer has policy that makes it difficult to contribute</td>
-<td>8</td>
-<td>4.8%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I have no time to contribute even if would like to</td>
+ <td style="text-align: left">65</td>
+ <td style="text-align: left">38.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t know how to start</td>
+ <td style="text-align: left">63</td>
+ <td style="text-align: left">37.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t have a need to contribute</td>
+ <td style="text-align: left">19</td>
+ <td style="text-align: left">11.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I didn’t know I could</td>
+ <td style="text-align: left">12</td>
+ <td style="text-align: left">7.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">My employer has policy that makes it difficult to contribute</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">4.8%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey results, the most important blocker for the Airflow contribution is limited time (38.9%), but surprisingly interesting and important blocker is also lack of knowledge on how to start (37.7%), followed by lack of knowledge that it’s possible to contribute (7.2%).</p>
<h2 id="the-future-of-airflow-1">The future of Airflow</h2>
<h3 id="in-your-opinion-what-could-be-improved-in-airflow-multiple-choice">In your opinion, what could be improved in Airflow? (multiple choice)</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>Web UI</td>
-<td>100</td>
-<td>49.5%</td>
-</tr>
-<tr>
-<td>Logging, monitoring and alerting</td>
-<td>97</td>
-<td>48.0%</td>
-</tr>
-<tr>
-<td>Examples, how-to, onboarding documentation</td>
-<td>74</td>
-<td>36.6%</td>
-</tr>
-<tr>
-<td>Technical documentation</td>
-<td>74</td>
-<td>36.6%</td>
-</tr>
-<tr>
-<td>Scheduler performance</td>
-<td>56</td>
-<td>27.7%</td>
-</tr>
-<tr>
-<td>Reliability</td>
-<td>52</td>
-<td>25.7%</td>
-</tr>
-<tr>
-<td>DAG authoring</td>
-<td>48</td>
-<td>23.8%</td>
-</tr>
-<tr>
-<td>REST API</td>
-<td>43</td>
-<td>21.3%</td>
-</tr>
-<tr>
-<td>Authentication and authorization</td>
-<td>41</td>
-<td>20.3%</td>
-</tr>
-<tr>
-<td>External integration e.g. AWS, GCP, Apache products</td>
-<td>41</td>
-<td>20.3%</td>
-</tr>
-<tr>
-<td>Better support for various deployments (Docker-compose/Nomad/Others)</td>
-<td>39</td>
-<td>19.3%</td>
-</tr>
-<tr>
-<td>Everything works fine for me</td>
-<td>19</td>
-<td>9.4%</td>
-</tr>
-<tr>
-<td>I don’t know</td>
-<td>4</td>
-<td>2.0%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Web UI</td>
+ <td style="text-align: left">100</td>
+ <td style="text-align: left">49.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Logging, monitoring and alerting</td>
+ <td style="text-align: left">97</td>
+ <td style="text-align: left">48.0%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Examples, how-to, onboarding documentation</td>
+ <td style="text-align: left">74</td>
+ <td style="text-align: left">36.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Technical documentation</td>
+ <td style="text-align: left">74</td>
+ <td style="text-align: left">36.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Scheduler performance</td>
+ <td style="text-align: left">56</td>
+ <td style="text-align: left">27.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Reliability</td>
+ <td style="text-align: left">52</td>
+ <td style="text-align: left">25.7%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DAG authoring</td>
+ <td style="text-align: left">48</td>
+ <td style="text-align: left">23.8%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">REST API</td>
+ <td style="text-align: left">43</td>
+ <td style="text-align: left">21.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Authentication and authorization</td>
+ <td style="text-align: left">41</td>
+ <td style="text-align: left">20.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">External integration e.g. AWS, GCP, Apache products</td>
+ <td style="text-align: left">41</td>
+ <td style="text-align: left">20.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Better support for various deployments (Docker-compose/Nomad/Others)</td>
+ <td style="text-align: left">39</td>
+ <td style="text-align: left">19.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Everything works fine for me</td>
+ <td style="text-align: left">19</td>
+ <td style="text-align: left">9.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t know</td>
+ <td style="text-align: left">4</td>
+ <td style="text-align: left">2.0%</td>
+ </tr>
+ </tbody>
</table>
<p>The results are quite self-explanatory. According to the survey results, the top area for improvement is still the Airflow web UI (49.5%), closely followed by more telemetry for logging, monitoring and alerting purposes (48%). However all those efforts should go in line with improved documentation (36.6.%) and resources about using the Airflow, especially when we take into account the need of onboarding new users (36.6%).</p>
<h3 id="which-features-would-you-like-to-see-in-airflow">Which features would you like to see in Airflow?</h3>
<table>
-<thead>
-<tr>
-<th></th>
-<th></th>
-<th></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td></td>
-<td>No.</td>
-<td>%</td>
-</tr>
-<tr>
-<td>DAG Versioning</td>
-<td>129</td>
-<td>66.2%</td>
-</tr>
-<tr>
-<td>Dependency management and Data-driven scheduling</td>
-<td>83</td>
-<td>42.6%</td>
-</tr>
-<tr>
-<td>More dynamic task structure</td>
-<td>82</td>
-<td>42.1%</td>
-</tr>
-<tr>
-<td>Multi-Tenancy</td>
-<td>74</td>
-<td>37.9%</td>
-</tr>
-<tr>
-<td>Signal-based scheduling</td>
-<td>67</td>
-<td>34.4%</td>
-</tr>
-<tr>
-<td>Better Security (Isolation)</td>
-<td>65</td>
-<td>33.3%</td>
-</tr>
-<tr>
-<td>Submitting new DAGs externally via API</td>
-<td>53</td>
-<td>27.2%</td>
-</tr>
-<tr>
-<td>Composable Operators</td>
-<td>46</td>
-<td>23.6%</td>
-</tr>
-<tr>
-<td>Support for native cloud executors (AWS/GCP/Azure etc.)</td>
-<td>44</td>
-<td>22.6%</td>
-</tr>
-<tr>
-<td>Better support for Machine Learning</td>
-<td>38</td>
-<td>19.5%</td>
-</tr>
-<tr>
-<td>Remote CLI</td>
-<td>36</td>
-<td>18.5%</td>
-</tr>
-<tr>
-<td>Support for hybrid executors</td>
-<td>22</td>
-<td>11.3%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ <th style="text-align: left"></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"></td>
+ <td style="text-align: left">No.</td>
+ <td style="text-align: left">%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DAG Versioning</td>
+ <td style="text-align: left">129</td>
+ <td style="text-align: left">66.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Dependency management and Data-driven scheduling</td>
+ <td style="text-align: left">83</td>
+ <td style="text-align: left">42.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">More dynamic task structure</td>
+ <td style="text-align: left">82</td>
+ <td style="text-align: left">42.1%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Multi-Tenancy</td>
+ <td style="text-align: left">74</td>
+ <td style="text-align: left">37.9%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Signal-based scheduling</td>
+ <td style="text-align: left">67</td>
+ <td style="text-align: left">34.4%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Better Security (Isolation)</td>
+ <td style="text-align: left">65</td>
+ <td style="text-align: left">33.3%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Submitting new DAGs externally via API</td>
+ <td style="text-align: left">53</td>
+ <td style="text-align: left">27.2%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Composable Operators</td>
+ <td style="text-align: left">46</td>
+ <td style="text-align: left">23.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Support for native cloud executors (AWS/GCP/Azure etc.)</td>
+ <td style="text-align: left">44</td>
+ <td style="text-align: left">22.6%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Better support for Machine Learning</td>
+ <td style="text-align: left">38</td>
+ <td style="text-align: left">19.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Remote CLI</td>
+ <td style="text-align: left">36</td>
+ <td style="text-align: left">18.5%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Support for hybrid executors</td>
+ <td style="text-align: left">22</td>
+ <td style="text-align: left">11.3%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey results, DAG Versioning is a winner for new features in Airflow, and it’s not a surprise as this feature may positively impact daily work of Airflow users. It is followed by three other ideas: Dependency management and Data-driven scheduling (42.6%), More dynamic task structure (42.1%) and Multi-Tenancy (37.9%). Another interesting point from that question is that only 11.3% think that support for hybrid executors is needed in Airflow.</p>
<h2 id="data">Data</h2>
@@ -2295,7 +2305,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-survey-2022/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-survey-2022>
@@ -2447,13 +2457,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-survey-2023/index.html b/blog/airflow-survey-2023/index.html
index 241787aef7..a7885924ae 100644
--- a/blog/airflow-survey-2023/index.html
+++ b/blog/airflow-survey-2023/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,36 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Airflow Survey 2023 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2023" />
-<meta property="og:description" content="Airflow 2 has seen rapid adoption, accompanied by continuous community growth. This annual survey helps us understand how people use Airflow and where we can best focus our efforts as we advance." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey-2023/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2023-09-21T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Airflow Survey 2023 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-survey-2023/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Airflow Survey 2023">
+ <meta property="og:description" content="Airflow 2 has seen rapid adoption, accompanied by continuous community growth. This annual survey helps us understand how people use Airflow and where we can best focus our efforts as we advance.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2023-09-21T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Community">
+ <meta property="article:tag" content="Survey">
+ <meta property="article:tag" content="Users">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Airflow Survey 2023">
-<meta itemprop="description" content="Airflow 2 has seen rapid adoption, accompanied by continuous community growth. This annual survey helps us understand how people use Airflow and where we can best focus our efforts as we advance."><meta itemprop="datePublished" content="2023-09-21T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="3"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="community,survey,users," />
+ <meta itemprop="name" content="Airflow Survey 2023">
+ <meta itemprop="description" content="Airflow 2 has seen rapid adoption, accompanied by continuous community growth. This annual survey helps us understand how people use Airflow and where we can best focus our efforts as we advance.">
+ <meta itemprop="datePublished" content="2023-09-21T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="3">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Community,Survey,Users">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +96,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -432,12 +442,12 @@
<p class="blogpost-content__metadata--description">Airflow 2 has seen rapid adoption, accompanied by continuous community growth. This annual survey helps us understand how people use Airflow and where we can best focus our efforts as we advance.</p>
</div>
<div class="markdown-content">
- <p><img src="images/Astronomer_Demographics.png" alt="Demographics" title="airflow_usage">
-<img src="images/Astronomer_Community_and_Contribution.png" alt="Community and Contribution" title="community_and_contributions">
-<img src="images/Airflow-Survey-2023-Results--Airflow-Usage-Page-1-Revised.png" alt="Airflow Usage Page 1">
-<img src="images/Astronomer-Airflow-Survey-2023-Results-Airflow-Usage-Page-2-Landscape.png" alt="Airflow Usage Page 2">
-<img src="images/Airflow-Survey-2023-Results-Airflow-Usage-Page-3-Revised-Landscape@2x.png" alt="Airflow Usage Page 3">
-<img src="images/Astronomer-Airflow-Survey-2023-Results-Future-Landscape@2x.png" alt="Future"></p>
+ <p><img alt="Demographics" src="/blog/airflow-survey-2023/images/Astronomer_Demographics.png" title="airflow_usage">
+<img alt="Community and Contribution" src="/blog/airflow-survey-2023/images/Astronomer_Community_and_Contribution.png" title="community_and_contributions">
+<img alt="Airflow Usage Page 1" src="/blog/airflow-survey-2023/images/Airflow-Survey-2023-Results--Airflow-Usage-Page-1-Revised.png">
+<img alt="Airflow Usage Page 2" src="/blog/airflow-survey-2023/images/Astronomer-Airflow-Survey-2023-Results-Airflow-Usage-Page-2-Landscape.png">
+<img alt="Airflow Usage Page 3" src="/blog/airflow-survey-2023/images/Airflow-Survey-2023-Results-Airflow-Usage-Page-3-Revised-Landscape@2x.png">
+<img alt="Future" src="/blog/airflow-survey-2023/images/Astronomer-Airflow-Survey-2023-Results-Future-Landscape@2x.png"></p>
<p><a href="https://docs.google.com/forms/d/1wYm6c5Gn379zkg7zD7vcWB-1fCjnOocT0oZm-tjft_Q/viewanalytics">View Raw Data</a></p>
</div>
@@ -610,7 +620,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-survey-2023/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-survey-2023>
@@ -762,13 +772,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-survey-2024/index.html b/blog/airflow-survey-2024/index.html
index e6d4f55a8e..3915cc159b 100644
--- a/blog/airflow-survey-2024/index.html
+++ b/blog/airflow-survey-2024/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,36 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Airflow Survey 2024 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2024" />
-<meta property="og:description" content="With more than 5,250 responses from 116 countries, this is the largest data engineering survey to date. Conducted annually, it offers valuable insights into Airflow usage and helps guide our future efforts." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey-2024/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2025-02-27T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Airflow Survey 2024 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-survey-2024/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Airflow Survey 2024">
+ <meta property="og:description" content="With more than 5,250 responses from 116 countries, this is the largest data engineering survey to date. Conducted annually, it offers valuable insights into Airflow usage and helps guide our future efforts.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2025-02-27T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Community">
+ <meta property="article:tag" content="Survey">
+ <meta property="article:tag" content="Users">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Airflow Survey 2024">
-<meta itemprop="description" content="With more than 5,250 responses from 116 countries, this is the largest data engineering survey to date. Conducted annually, it offers valuable insights into Airflow usage and helps guide our future efforts."><meta itemprop="datePublished" content="2025-02-27T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="3"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="community,survey,users," />
+ <meta itemprop="name" content="Airflow Survey 2024">
+ <meta itemprop="description" content="With more than 5,250 responses from 116 countries, this is the largest data engineering survey to date. Conducted annually, it offers valuable insights into Airflow usage and helps guide our future efforts.">
+ <meta itemprop="datePublished" content="2025-02-27T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="3">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Community,Survey,Users">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +96,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -445,7 +455,7 @@
<p class="blogpost-content__metadata--description">With more than 5,250 responses from 116 countries, this is the largest data engineering survey to date. Conducted annually, it offers valuable insights into Airflow usage and helps guide our future efforts.</p>
</div>
<div class="markdown-content">
- <p><img src="images/Airflow-Survey-2024-Results-v2.png" alt="Airflow Survey 2024" title="airflow_survey_2024"></p>
+ <p><img alt="Airflow Survey 2024" src="/blog/airflow-survey-2024/images/Airflow-Survey-2024-Results-v2.png" title="airflow_survey_2024"></p>
<p><a href="https://astronomer.typeform.com/report/SF2VGNTc/fRSeRcKKJ3kgYXVl">View Raw Data</a></p>
</div>
@@ -618,7 +628,7 @@
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en/blog/airflow-survey-2024/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/main/landing-pages/site/content/en//blog/airflow-survey-2024>
@@ -770,13 +780,13 @@
-<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
+<script src="/js/main.min.7d5b92c4889cf0554b0ea5e2309b0b81b196aa7c9a235632938a25d9fc4d8a7c.js" integrity="sha256-fVuSxIic8FVLDqXiMJsLgbGWqnyaI1Yyk4ol2fxNinw=" crossorigin="anonymous"></script>
</body>
- <script src="/main.60e9d.js"></script>
+ <script src="/main.8cb1f.js"></script>
</html>
diff --git a/blog/airflow-survey/index.html b/blog/airflow-survey/index.html
index 9c474befa2..20deb9f0b6 100644
--- a/blog/airflow-survey/index.html
+++ b/blog/airflow-survey/index.html
@@ -7,7 +7,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.91.2" />
+<meta name="generator" content="Hugo 0.135.0">
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -31,26 +31,36 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Airflow Survey 2019 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2019" />
-<meta property="og:description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss." />
-<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
-<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2025-09-25T22:04:04+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<title>Airflow Survey 2019 | Apache Airflow</title><meta property="og:url" content="/blog/airflow-survey/">
+ <meta property="og:site_name" content="Apache Airflow">
+ <meta property="og:title" content="Airflow Survey 2019">
+ <meta property="og:description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss.">
+ <meta property="og:locale" content="en">
+ <meta property="og:type" content="article">
+ <meta property="article:section" content="blog">
+ <meta property="article:published_time" content="2019-12-11T00:00:00+00:00">
+ <meta property="article:modified_time" content="2025-09-25T22:52:02+01:00">
+ <meta property="article:tag" content="Community">
+ <meta property="article:tag" content="Survey">
+ <meta property="article:tag" content="Users">
+ <meta property="og:image" content="/images/feature-image.png">
-<meta itemprop="name" content="Airflow Survey 2019">
-<meta itemprop="description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss."><meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2025-09-25T22:04:04+01:00" />
-<meta itemprop="wordCount" content="1723"><meta itemprop="image" content="/images/feature-image.png"/>
-<meta itemprop="keywords" content="community,survey,users," />
+ <meta itemprop="name" content="Airflow Survey 2019">
+ <meta itemprop="description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss.">
+ <meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00">
+ <meta itemprop="dateModified" content="2025-09-25T22:52:02+01:00">
+ <meta itemprop="wordCount" content="1723">
+ <meta itemprop="image" content="/images/feature-image.png">
+ <meta itemprop="keywords" content="Community,Survey,Users">
-<link rel="preload" href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" as="style">
-<link href="/scss/main.min.b17aebabd27496c9901cfab812b30fb885ece0ac9dedd4969f14edf568b76157.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" as="style">
+<link href="/scss/main.min.88249c1339fe16a3fabf9b92c623dfad7cedd424c3202bfec29db4dcb0326427.css" rel="stylesheet" integrity="">
+
<script
src="/external/js/jquery-3.3.1.min.js"
@@ -86,10 +96,10 @@
-<link rel="preload" href="/main.60e9d.js" as="script">
-<link rel="preload" href="/chunk-1.60e9d.js" as="script">
+<link rel="preload" href="/main.8cb1f.js" as="script">
+<link rel="preload" href="/chunk-608.8cb1f.js" as="script">
-<link rel="preload" href="/chunk-4.60e9d.js" as="script">
+<link rel="preload" href="/" as="script">
@@ -453,117 +463,117 @@ Let’s see who Airflow users are, how they play with it, and what they miss.</p
<h1 id="overview-of-the-user">Overview of the user</h1>
<p><strong>What best describes your current occupation?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Data Engineer</td>
-<td>194</td>
-<td>62.99%</td>
-</tr>
-<tr>
-<td>Developer</td>
-<td>34</td>
-<td>11.04%</td>
-</tr>
-<tr>
-<td>Architect</td>
-<td>23</td>
-<td>7.47%</td>
-</tr>
-<tr>
-<td>Data Scientist</td>
-<td>19</td>
-<td>6.17%</td>
-</tr>
-<tr>
-<td>Data Analyst</td>
-<td>13</td>
-<td>4.22%</td>
-</tr>
-<tr>
-<td>DevOps</td>
-<td>13</td>
-<td>4.22%</td>
-</tr>
-<tr>
-<td>IT Administrator</td>
-<td>2</td>
-<td>0.65%</td>
-</tr>
-<tr>
-<td>Machine Learning Engineer</td>
-<td>2</td>
-<td>0.65%</td>
-</tr>
-<tr>
-<td>Manager</td>
-<td>2</td>
-<td>0.65%</td>
-</tr>
-<tr>
-<td>Operations</td>
-<td>2</td>
-<td>0.65%</td>
-</tr>
-<tr>
-<td>Chief Data Officer</td>
-<td>1</td>
-<td>0.32%</td>
-</tr>
-<tr>
-<td>Engineering Manager</td>
-<td>1</td>
-<td>0.32%</td>
-</tr>
-<tr>
-<td>Intern</td>
-<td>1</td>
-<td>0.32%</td>
-</tr>
-<tr>
-<td>Product owner</td>
-<td>1</td>
-<td>0.32%</td>
-</tr>
-<tr>
-<td>Quant</td>
-<td>1</td>
-<td>0.32%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Data Engineer</td>
+ <td style="text-align: left">194</td>
+ <td style="text-align: left">62.99%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Developer</td>
+ <td style="text-align: left">34</td>
+ <td style="text-align: left">11.04%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Architect</td>
+ <td style="text-align: left">23</td>
+ <td style="text-align: left">7.47%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Data Scientist</td>
+ <td style="text-align: left">19</td>
+ <td style="text-align: left">6.17%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Data Analyst</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">4.22%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">DevOps</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">4.22%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">IT Administrator</td>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">0.65%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Machine Learning Engineer</td>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">0.65%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Manager</td>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">0.65%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Operations</td>
+ <td style="text-align: left">2</td>
+ <td style="text-align: left">0.65%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Chief Data Officer</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.32%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Engineering Manager</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.32%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Intern</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.32%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Product owner</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.32%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Quant</td>
+ <td style="text-align: left">1</td>
+ <td style="text-align: left">0.32%</td>
+ </tr>
+ </tbody>
</table>
<p><strong>In your day to day job, what do you use Airflow for?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Data processing (ETL)</td>
-<td>298</td>
-<td>96.75%</td>
-</tr>
-<tr>
-<td>Artificial Intelligence and Machine Learning Pipelines</td>
-<td>90</td>
-<td>29.22%</td>
-</tr>
-<tr>
-<td>Automating DevOps operations</td>
-<td>64</td>
-<td>20.78%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Data processing (ETL)</td>
+ <td style="text-align: left">298</td>
+ <td style="text-align: left">96.75%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Artificial Intelligence and Machine Learning Pipelines</td>
+ <td style="text-align: left">90</td>
+ <td style="text-align: left">29.22%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Automating DevOps operations</td>
+ <td style="text-align: left">64</td>
+ <td style="text-align: left">20.78%</td>
+ </tr>
+ </tbody>
</table>
<p>According to the survey, most of the Airflow users are the “data” people. Moreover,
28.57% uses Airflow to both ETL and ML pipelines meaning that those two fields
@@ -572,154 +582,154 @@ That means that other 59 people who use Airflow for DevOps stuff use it also for
ETL / ML purposes.</p>
<p><strong>How many active DAGs do you have in your largest Airflow instance?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>0-20</td>
-<td>115</td>
-<td>37.34%</td>
-</tr>
-<tr>
-<td>21-40</td>
-<td>65</td>
-<td>21.10%</td>
-</tr>
-<tr>
-<td>41-60</td>
-<td>44</td>
-<td>14.29%</td>
-</tr>
-<tr>
-<td>61-100</td>
-<td>28</td>
-<td>9.09%</td>
-</tr>
-<tr>
-<td>101-200</td>
-<td>28</td>
-<td>9.09%</td>
-</tr>
-<tr>
-<td>201-300</td>
-<td>7</td>
-<td>2.27%</td>
-</tr>
-<tr>
-<td>301-999</td>
-<td>8</td>
-<td>2.60%</td>
-</tr>
-<tr>
-<td>1000+</td>
-<td>13</td>
-<td>4.22%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">0-20</td>
+ <td style="text-align: left">115</td>
+ <td style="text-align: left">37.34%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">21-40</td>
+ <td style="text-align: left">65</td>
+ <td style="text-align: left">21.10%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">41-60</td>
+ <td style="text-align: left">44</td>
+ <td style="text-align: left">14.29%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">61-100</td>
+ <td style="text-align: left">28</td>
+ <td style="text-align: left">9.09%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">101-200</td>
+ <td style="text-align: left">28</td>
+ <td style="text-align: left">9.09%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">201-300</td>
+ <td style="text-align: left">7</td>
+ <td style="text-align: left">2.27%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">301-999</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">2.60%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">1000+</td>
+ <td style="text-align: left">13</td>
+ <td style="text-align: left">4.22%</td>
+ </tr>
+ </tbody>
</table>
<p>The majority of users do not exceed 100 active DAGs per Airflow instance. However,
as we can see there are users who exceed thousands of DAGs with a maximum number 5000.</p>
<p><strong>What is the maximum number of tasks that you have used in one DAG?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>0-10</td>
-<td>61</td>
-<td>19.81%</td>
-</tr>
-<tr>
-<td>11-20</td>
-<td>60</td>
-<td>19.48%</td>
-</tr>
-<tr>
-<td>21-30</td>
-<td>31</td>
-<td>10.06%</td>
-</tr>
-<tr>
-<td>31-40</td>
-<td>21</td>
-<td>6.82%</td>
-</tr>
-<tr>
-<td>41-50</td>
-<td>26</td>
-<td>8.44%</td>
-</tr>
-<tr>
-<td>51-100</td>
-<td>36</td>
-<td>11.69%</td>
-</tr>
-<tr>
-<td>101-200</td>
-<td>28</td>
-<td>9.09%</td>
-</tr>
-<tr>
-<td>201-500</td>
-<td>21</td>
-<td>6.82%</td>
-</tr>
-<tr>
-<td>501+</td>
-<td>24</td>
-<td>11.54%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">0-10</td>
+ <td style="text-align: left">61</td>
+ <td style="text-align: left">19.81%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">11-20</td>
+ <td style="text-align: left">60</td>
+ <td style="text-align: left">19.48%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">21-30</td>
+ <td style="text-align: left">31</td>
+ <td style="text-align: left">10.06%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">31-40</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">6.82%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">41-50</td>
+ <td style="text-align: left">26</td>
+ <td style="text-align: left">8.44%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">51-100</td>
+ <td style="text-align: left">36</td>
+ <td style="text-align: left">11.69%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">101-200</td>
+ <td style="text-align: left">28</td>
+ <td style="text-align: left">9.09%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">201-500</td>
+ <td style="text-align: left">21</td>
+ <td style="text-align: left">6.82%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">501+</td>
+ <td style="text-align: left">24</td>
+ <td style="text-align: left">11.54%</td>
+ </tr>
+ </tbody>
</table>
<p>The given maximum number of tasks in a single DAG was 10 000 (!). The number of tasks
depends on the purposes of a DAG, so it’s rather hard to say if users have “simple”
or “complicated” workflows.</p>
<p><strong>When onboarding new members to Airflow, what is the biggest problem?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>No guide on best practises on developing DAGs</td>
-<td>160</td>
-<td>51.95%</td>
-</tr>
-<tr>
-<td>Small number of tutorials on different aspects of using Airflow</td>
-<td>57</td>
-<td>18.51%</td>
-</tr>
-<tr>
-<td>Documentation is not clear enough</td>
-<td>42</td>
-<td>13.64%</td>
-</tr>
-<tr>
-<td>Small number of blogs regarding Airflow</td>
-<td>6</td>
-<td>1.95%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>43</td>
-<td>13.96%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">No guide on best practises on developing DAGs</td>
+ <td style="text-align: left">160</td>
+ <td style="text-align: left">51.95%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Small number of tutorials on different aspects of using Airflow</td>
+ <td style="text-align: left">57</td>
+ <td style="text-align: left">18.51%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Documentation is not clear enough</td>
+ <td style="text-align: left">42</td>
+ <td style="text-align: left">13.64%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Small number of blogs regarding Airflow</td>
+ <td style="text-align: left">6</td>
+ <td style="text-align: left">1.95%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">43</td>
+ <td style="text-align: left">13.96%</td>
+ </tr>
+ </tbody>
</table>
<p>This is an important result. Using Airflow is all about writing and scheduling DAGs.
No guide or any other complete resource on best practices for developing Dags is a big
@@ -731,40 +741,40 @@ problem. Diving deep in the “other” answers, we can find that:</p>
</ul>
<p><strong>How likely are you to recommend Apache Airflow?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Very Likely</td>
-<td>140</td>
-<td>45.45%</td>
-</tr>
-<tr>
-<td>Likely</td>
-<td>124</td>
-<td>40.26%</td>
-</tr>
-<tr>
-<td>Neutral</td>
-<td>33</td>
-<td>10.71%</td>
-</tr>
-<tr>
-<td>Unlikely</td>
-<td>8</td>
-<td>2.60%</td>
-</tr>
-<tr>
-<td>Very unlikely</td>
-<td>3</td>
-<td>0.97%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Very Likely</td>
+ <td style="text-align: left">140</td>
+ <td style="text-align: left">45.45%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Likely</td>
+ <td style="text-align: left">124</td>
+ <td style="text-align: left">40.26%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Neutral</td>
+ <td style="text-align: left">33</td>
+ <td style="text-align: left">10.71%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Unlikely</td>
+ <td style="text-align: left">8</td>
+ <td style="text-align: left">2.60%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Very unlikely</td>
+ <td style="text-align: left">3</td>
+ <td style="text-align: left">0.97%</td>
+ </tr>
+ </tbody>
</table>
<p>This means that more than 85% of people who use Airflow like it. It seems Airflow does
its job nicely. However, we have to remember that this survey is likely biased - it’s
@@ -773,55 +783,55 @@ focus then on those 11 people who did not like Airflow? It’s a good question.<
<h2 id="airflow-usage">Airflow usage</h2>
<p><strong>Which interface(s) of Airflow do you use as part of your current role?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Original Airflow Graphical User Interface</td>
-<td>297</td>
-<td>96.43%</td>
-</tr>
-<tr>
-<td>CLI</td>
-<td>126</td>
-<td>40.91%</td>
-</tr>
-<tr>
-<td>Original Airflow Graphical User Interface, CLI</td>
-<td>117</td>
-<td>37.99%</td>
-</tr>
-<tr>
-<td>API</td>
-<td>60</td>
-<td>19.48%</td>
-</tr>
-<tr>
-<td>Original Airflow Graphical User Interface, CLI, API</td>
-<td>32</td>
-<td>10.39%</td>
-</tr>
-<tr>
-<td>Custom (own created) Airflow Graphical User Interface</td>
-<td>25</td>
-<td>8.12%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Original Airflow Graphical User Interface</td>
+ <td style="text-align: left">297</td>
+ <td style="text-align: left">96.43%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">CLI</td>
+ <td style="text-align: left">126</td>
+ <td style="text-align: left">40.91%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Original Airflow Graphical User Interface, CLI</td>
+ <td style="text-align: left">117</td>
+ <td style="text-align: left">37.99%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">API</td>
+ <td style="text-align: left">60</td>
+ <td style="text-align: left">19.48%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Original Airflow Graphical User Interface, CLI, API</td>
+ <td style="text-align: left">32</td>
+ <td style="text-align: left">10.39%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Custom (own created) Airflow Graphical User Interface</td>
+ <td style="text-align: left">25</td>
+ <td style="text-align: left">8.12%</td>
+ </tr>
+ </tbody>
</table>
<p>It’s visible that usage of CLI goes in pair with using Airflow web UI. Our
survey included some UX related questions to allow us to understand how users
use Airflow webserver.</p>
<p><strong>What do you use the Graphical User Interface for?</strong></p>
-<p><img src="plot1.png" alt=""></p>
+<p><img src="/blog/airflow-survey/plot1.png"></p>
<p><strong>What do you use CLI for?</strong></p>
-<p><img src="plot2.png" alt=""></p>
+<p><img src="/blog/airflow-survey/plot2.png"></p>
<p><strong>In Airflow, which UI view(s) are important for you?</strong></p>
-<p><img src="plot3.png" alt=""></p>
+<p><img src="/blog/airflow-survey/plot3.png"></p>
<p>Here we see that the majority uses Web UI mostly for monitoring purposes:</p>
<ul>
<li>Monitoring DAGs</li>
@@ -831,40 +841,40 @@ use Airflow webserver.</p>
there’s no other way than to do it by CLI.</p>
<p><strong>What executor type do you use?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Celery</td>
-<td>138</td>
-<td>44.81%</td>
-</tr>
-<tr>
-<td>Local</td>
-<td>85</td>
-<td>27.60%</td>
-</tr>
-<tr>
-<td>Kubernetes</td>
-<td>52</td>
-<td>16.88%</td>
-</tr>
-<tr>
-<td>Sequential</td>
-<td>22</td>
-<td>7.14%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>11</td>
-<td>3.57</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Celery</td>
+ <td style="text-align: left">138</td>
+ <td style="text-align: left">44.81%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Local</td>
+ <td style="text-align: left">85</td>
+ <td style="text-align: left">27.60%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Kubernetes</td>
+ <td style="text-align: left">52</td>
+ <td style="text-align: left">16.88%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Sequential</td>
+ <td style="text-align: left">22</td>
+ <td style="text-align: left">7.14%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">11</td>
+ <td style="text-align: left">3.57</td>
+ </tr>
+ </tbody>
</table>
<p>The other option mostly consisted of information that someone uses a few types or is
migrating from one executor to another. What can be observed is an increase in usage
@@ -872,45 +882,45 @@ of Local and Kubernetes executors when compared to results from an earlier <a hr
by Ash</a>.</p>
<p><strong>Do you use Kubernetes-based deployments for Airflow?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>No - we do not plan to use Kubernetes near term</td>
-<td>88</td>
-<td>28.57%</td>
-</tr>
-<tr>
-<td>Yes - setup on our own via Helm Chart or similar</td>
-<td>65</td>
-<td>21.10%</td>
-</tr>
-<tr>
-<td>Not yet - but we use Kubernetes in our organization and we could move</td>
-<td>61</td>
-<td>19.81%</td>
-</tr>
-<tr>
-<td>Yes - via managed service in the cloud (Composer / Astronomer etc.)</td>
-<td>45</td>
-<td>14.61%</td>
-</tr>
-<tr>
-<td>Not yet - but we plan to deploy Kubernetes in our organization soon</td>
-<td>42</td>
-<td>13.64%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>7</td>
-<td>2.27%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">No - we do not plan to use Kubernetes near term</td>
+ <td style="text-align: left">88</td>
+ <td style="text-align: left">28.57%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes - setup on our own via Helm Chart or similar</td>
+ <td style="text-align: left">65</td>
+ <td style="text-align: left">21.10%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Not yet - but we use Kubernetes in our organization and we could move</td>
+ <td style="text-align: left">61</td>
+ <td style="text-align: left">19.81%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes - via managed service in the cloud (Composer / Astronomer etc.)</td>
+ <td style="text-align: left">45</td>
+ <td style="text-align: left">14.61%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Not yet - but we plan to deploy Kubernetes in our organization soon</td>
+ <td style="text-align: left">42</td>
+ <td style="text-align: left">13.64%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">7</td>
+ <td style="text-align: left">2.27%</td>
+ </tr>
+ </tbody>
</table>
<p>The most interesting thing is that there’s nearly 30% of users who do not use Kubernetes,
and they are not going to move. This means we should keep other deployment options in
@@ -918,74 +928,74 @@ mind when working on Airflow 2.0. On the other hand, almost 70% of the users alr
use Kubernetes, or it’s a viable option for them.</p>
<p><strong>Do you combine multiple DAGs?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>No, I don’t combine multiple DAGs</td>
-<td>127</td>
-<td>41.23%</td>
-</tr>
-<tr>
-<td>Yes, through SubDAG</td>
-<td>73</td>
-<td>23.70%</td>
-</tr>
-<tr>
-<td>Yes, by triggering another DAG</td>
-<td>72</td>
-<td>23.38%</td>
-</tr>
-<tr>
-<td>Other</td>
-<td>36</td>
-<td>11.69%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">No, I don’t combine multiple DAGs</td>
+ <td style="text-align: left">127</td>
+ <td style="text-align: left">41.23%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, through SubDAG</td>
+ <td style="text-align: left">73</td>
+ <td style="text-align: left">23.70%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Yes, by triggering another DAG</td>
+ <td style="text-align: left">72</td>
+ <td style="text-align: left">23.38%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Other</td>
+ <td style="text-align: left">36</td>
+ <td style="text-align: left">11.69%</td>
+ </tr>
+ </tbody>
</table>
<p>In the other category, 9 people explicitly mentioned using <code>ExternalTaskSensor</code>,
and I think it could be treated as running subDAGs by triggering other DAGs.</p>
<p><strong>Do you use Airflow Plugins? If yes, what do you use it for?</strong></p>
<table>
-<thead>
-<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Adding new operators/sensors and hooks</td>
-<td>187</td>
-<td>60.71%</td>
-</tr>
-<tr>
-<td>I don’t use Airflow plugins</td>
-<td>109</td>
-<td>35.39%</td>
-</tr>
-<tr>
-<td>Adding AppBuilder views & menu items</td>
-<td>31</td>
-<td>10.06%</td>
-</tr>
-<tr>
-<td>Adding new executor</td>
-<td>18</td>
-<td>5.84%</td>
-</tr>
-<tr>
-<td>Adding OperatorExtraLinks</td>
-<td>7</td>
-<td>2.27%</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left"></th>
+ <th style="text-align: left">No.</th>
+ <th style="text-align: left">%</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">Adding new operators/sensors and hooks</td>
+ <td style="text-align: left">187</td>
+ <td style="text-align: left">60.71%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">I don’t use Airflow plugins</td>
+ <td style="text-align: left">109</td>
+ <td style="text-align: left">35.39%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Adding AppBuilder views & menu items</td>
+ <td style="text-align: left">31</td>
+ <td style="text-align: left">10.06%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Adding new executor</td>
+ <td style="text-align: left">18</td>
+ <td style="text-align: left">5.84%</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">Adding OperatorExtraLinks</td>
+ <td style="text-align: left">7</td>
... 46476 lines suppressed ...