You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by gi...@apache.org on 2022/07/12 16:30:51 UTC
[airflow-site] branch gh-pages updated: Deploying to gh-pages from @ 124cd9a37ae833d6e6bdd197f5cafde66b02c3cf ๐
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/gh-pages by this push:
new d216844c6f Deploying to gh-pages from @ 124cd9a37ae833d6e6bdd197f5cafde66b02c3cf ๐
d216844c6f is described below
commit d216844c6f686670b699f1bc2afc0a688929a035
Author: potiuk <ja...@potiuk.com>
AuthorDate: Tue Jul 12 16:30:44 2022 +0000
Deploying to gh-pages from @ 124cd9a37ae833d6e6bdd197f5cafde66b02c3cf ๐
---
404.html | 20 +-
announcements/index.html | 1136 ++++-------
blog/airflow-1.10.10/index.html | 183 +-
blog/airflow-1.10.12/index.html | 150 +-
blog/airflow-1.10.8-1.10.9/index.html | 96 +-
blog/airflow-2.2.0/index.html | 92 +-
blog/airflow-2.3.0/index.html | 183 +-
blog/airflow-survey-2020/index.html | 323 +---
blog/airflow-survey-2022/index.html | 488 +----
blog/airflow-survey/index.html | 230 +--
blog/airflow-two-point-oh-is-here/index.html | 162 +-
blog/airflow_summit_2021/index.html | 47 +-
blog/airflow_summit_2022/index.html | 51 +-
blog/announcing-new-website/index.html | 47 +-
blog/apache-airflow-for-newcomers/index.html | 133 +-
.../index.html | 38 +-
.../index.html | 55 +-
.../index.html | 125 +-
.../index.html | 84 +-
.../index.html | 128 +-
blog/index.html | 22 +-
blog/index.xml | 2012 ++++----------------
.../index.html | 43 +-
blog/tags/airflow-summit/index.html | 25 +-
blog/tags/airflow-summit/index.xml | 30 +-
blog/tags/community/index.html | 31 +-
blog/tags/community/index.xml | 1139 ++---------
blog/tags/development/index.html | 25 +-
blog/tags/development/index.xml | 30 +-
blog/tags/documentation/index.html | 25 +-
blog/tags/documentation/index.xml | 91 +-
blog/tags/release/index.html | 25 +-
blog/tags/release/index.xml | 662 ++-----
blog/tags/rest-api/index.html | 25 +-
blog/tags/rest-api/index.xml | 94 +-
blog/tags/survey/index.html | 25 +-
blog/tags/survey/index.xml | 939 +--------
blog/tags/users/index.html | 25 +-
blog/tags/users/index.xml | 939 +--------
categories/index.html | 26 +-
community/index.html | 40 +-
docs/index.html | 303 +--
ecosystem/index.html | 197 +-
index.html | 52 +-
index.xml | 2012 ++++----------------
install/index.html | 20 +-
...4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js | 6 -
...f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js | 1 +
meetups/index.html | 22 +-
privacy-notice/index.html | 49 +-
roadmap/index.html | 49 +-
...d2437604a16375e2ddfe624d37e328483cfa86ad1e1.css | 1 +
...518111642bc80af69f05c204c47682d42ef29747534.css | 1 -
...8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css | 7 -
...5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css | 7 +
search/index.html | 30 +-
sitemap.xml | 300 +--
tags/index.html | 27 +-
use-cases/adobe/index.html | 47 +-
use-cases/adyen/index.html | 40 +-
use-cases/big-fish-games/index.html | 39 +-
use-cases/dish/index.html | 39 +-
use-cases/experity/index.html | 39 +-
use-cases/index.html | 20 +-
use-cases/index.xml | 94 +-
use-cases/onefootball/index.html | 41 +-
use-cases/plarium-krasnodar/index.html | 39 +-
use-cases/seniorlink/index.html | 41 +-
use-cases/sift/index.html | 45 +-
69 files changed, 2956 insertions(+), 10656 deletions(-)
diff --git a/404.html b/404.html
index f18b16d58e..9559e34122 100644
--- a/404.html
+++ b/404.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,14 +32,10 @@
<title>404 Page not found | Apache Airflow</title><meta property="og:title" content="404 Page not found" />
<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
<meta property="og:type" content="website" />
-<meta property="og:url" content="/404.html" />
+<meta property="og:url" content="/404.html" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
-<meta property="og:image" content="/images/feature-image.png" />
-<meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="404 Page not found">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
-
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows."><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="404 Page not found"/>
@@ -51,8 +47,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -65,8 +61,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -599,7 +595,7 @@
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/announcements/index.html b/announcements/index.html
index 4e6ed3ec75..ecb782f07f 100644
--- a/announcements/index.html
+++ b/announcements/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -33,15 +33,10 @@
<title>Announcements | Apache Airflow</title><meta property="og:title" content="Announcements" />
<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
<meta property="og:type" content="website" />
-<meta property="og:url" content="/announcements/" />
+<meta property="og:url" content="/announcements/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
-<meta property="og:image" content="/images/feature-image.png" />
-
-<meta property="og:updated_time" content="2020-08-08T05:30:20+05:30" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Announcements">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
-
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows."><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Announcements"/>
@@ -53,8 +48,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -67,8 +62,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -429,1059 +424,678 @@
<main role="main" class="td-main container">
-
-
-<p>
- </p>
-
-<p><strong>Note:</strong> Follow <a href="https://twitter.com/ApacheAirflow" target="_blank">@ApacheAirflow</a> on Twitter for the latest news and announcements!</p>
-
+<p>ย
+ย </p>
+<p><strong>Note:</strong> Follow <a href="https://twitter.com/ApacheAirflow">@ApacheAirflow</a> on Twitter for the latest news and announcements!</p>
<h1 id="july-09-2022">July 09, 2022</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.3.3</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.3/" target="_blank">https://pypi.org/project/apache-airflow/2.3.3/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.3" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.3</a> <br />
-๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.3/release_notes.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.3/release_notes.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.3/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.3/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.3/">https://pypi.org/project/apache-airflow/2.3.3/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.3">https://airflow.apache.org/docs/apache-airflow/2.3.3</a> <br>
+๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.3/release_notes.html">https://airflow.apache.org/docs/apache-airflow/2.3.3/release_notes.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.3/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.3.3/installation/installing-from-sources.html</a></p>
<h1 id="june-04-2022">June 04, 2022</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.3.2</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.2/" target="_blank">https://pypi.org/project/apache-airflow/2.3.2/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.2" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.2</a> <br />
-๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.2/release_notes.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.2/release_notes.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.2/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.2/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.2/">https://pypi.org/project/apache-airflow/2.3.2/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.2">https://airflow.apache.org/docs/apache-airflow/2.3.2</a> <br>
+๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.2/release_notes.html">https://airflow.apache.org/docs/apache-airflow/2.3.2/release_notes.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.2/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.3.2/installation/installing-from-sources.html</a></p>
<h1 id="may-25-2022">May 25, 2022</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.3.1</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.1/" target="_blank">https://pypi.org/project/apache-airflow/2.3.1/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.1" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.1</a> <br />
-๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.1/release_notes.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.1/release_notes.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.1/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.1/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.1/">https://pypi.org/project/apache-airflow/2.3.1/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.1">https://airflow.apache.org/docs/apache-airflow/2.3.1</a> <br>
+๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.1/release_notes.html">https://airflow.apache.org/docs/apache-airflow/2.3.1/release_notes.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.1/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.3.1/installation/installing-from-sources.html</a></p>
<h1 id="may-20-2022">May 20, 2022</h1>
-
<p>We’ve just released Apache <strong>Airflow Helm chart 1.6.0</strong>.</p>
-
-<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow" target="_blank">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.6.0/" target="_blank">https://airflow.apache.org/docs/helm-chart/1.6.0/</a> <br />
-๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/helm-chart/1.6.0/release_notes.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.6.0/release_notes.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.6.0/installing-helm-chart-from-sources.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.6.0/installing-helm-chart-from-sources.html</a></p>
-
+<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.6.0/">https://airflow.apache.org/docs/helm-chart/1.6.0/</a> <br>
+๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/helm-chart/1.6.0/release_notes.html">https://airflow.apache.org/docs/helm-chart/1.6.0/release_notes.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.6.0/installing-helm-chart-from-sources.html">https://airflow.apache.org/docs/helm-chart/1.6.0/installing-helm-chart-from-sources.html</a></p>
<h1 id="april-30-2022">April 30, 2022</h1>
-
-<p>Weโve just released Apache <strong>Airflow 2.3.0</strong>. You can read more in the <a href="/blog/airflow-2.3.0/" target="_blank">What’s new in Apache Airflow 2.3.0</a> blog post.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.0/" target="_blank">https://pypi.org/project/apache-airflow/2.3.0/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.0</a> <br />
-๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.0/installation/installing-from-sources.html</a></p>
-
+<p>Weโve just released Apache <strong>Airflow 2.3.0</strong>. You can read more in the <a href="/blog/airflow-2.3.0/">What’s new in Apache Airflow 2.3.0</a> blog post.</p>
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.0/">https://pypi.org/project/apache-airflow/2.3.0/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0">https://airflow.apache.org/docs/apache-airflow/2.3.0</a> <br>
+๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html">https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.3.0/installation/installing-from-sources.html</a></p>
<h1 id="april-04-2022">April 04, 2022</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.2.5</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.5/" target="_blank">https://pypi.org/project/apache-airflow/2.2.5/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.5/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.5/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.5/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.5/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.5/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.5/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.5/">https://pypi.org/project/apache-airflow/2.2.5/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.5/">https://airflow.apache.org/docs/apache-airflow/2.2.5/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.5/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.2.5/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.5/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.2.5/installation/installing-from-sources.html</a></p>
<h1 id="march-11-2022">March 11, 2022</h1>
-
<p>We’ve just released Apache <strong>Airflow Helm chart 1.5.0</strong>.</p>
-
-<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow" target="_blank">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.5.0/" target="_blank">https://airflow.apache.org/docs/helm-chart/1.5.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.5.0/changelog.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.5.0/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.5.0/installing-helm-chart-from-sources.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.5.0/installing-helm-chart-from-sources.html</a></p>
-
+<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.5.0/">https://airflow.apache.org/docs/helm-chart/1.5.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.5.0/changelog.html">https://airflow.apache.org/docs/helm-chart/1.5.0/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.5.0/installing-helm-chart-from-sources.html">https://airflow.apache.org/docs/helm-chart/1.5.0/installing-helm-chart-from-sources.html</a></p>
<h1 id="march-11-2022-1">March 11, 2022</h1>
-
-<p><a href="https://airflowsummit.org" target="_blank">Airflow Summit 2022</a></p>
-
+<p><a href="https://airflowsummit.org">Airflow Summit 2022</a></p>
<p>The biggest Airflow Event of the Year returns May 23โ27! Airflow Summit 2022 will bring together the global
community of Apache Airflow practitioners and data leaders.</p>
-
-<h3 id="what-s-on-the-agenda">Whatโs on the Agenda</h3>
-
+<h3 id="whats-on-the-agenda">Whatโs on the Agenda</h3>
<p>During the free conference, you will hear about Apache Airflow best practices, trends in building data
pipelines, data governance, Airflow and machine learning, and the future of Airflow.
There will also be a series of presentations on non-code contributions driving the open-source project.</p>
-
<h3 id="how-to-attend">How to Attend</h3>
-
<p>This yearโs edition will include a variety of online sessions across different time zones. Additionally,
you can take part in local in-person events organized worldwide for data communities
to watch the event and network.</p>
-
<h3 id="interested">Interested?</h3>
-
-<p>๐ชถ <a href="https://www.crowdcast.io/e/airflowsummit2022/register?utm_campaign=Astronomer_marketing&utm_source=Astronomer%20website&utm_medium=website&utm_term=Airflow%20Summit" target="_blank">Register for Airflow Summit 2022</a> today! <br />
-๐ฃ๏ธ If you have an Airflow story to share, <a href="https://sessionize.com/airflow-summit-2022/" target="_blank">join as a speaker</a> <br />
-โจ <a href="https://www.linkedin.com/company/airflow-summit/" target="_blank">Follow Airflow Summit on LinkedIn</a> to stay current with the latest updates. <br />
-๐ค <a href="https://airflowsummit.org/in-person-events/" target="_blank">Check out the in-person events</a> planned for Airflow Summit 2022.</p>
-
+<p>๐ชถ <a href="https://www.crowdcast.io/e/airflowsummit2022/register?utm_campaign=Astronomer_marketing&utm_source=Astronomer%20website&utm_medium=website&utm_term=Airflow%20Summit">Register for Airflow Summit 2022</a> today! <br>
+๐ฃ๏ธ If you have an Airflow story to share, <a href="https://sessionize.com/airflow-summit-2022/">join as a speaker</a> <br>
+โจ <a href="https://www.linkedin.com/company/airflow-summit/">Follow Airflow Summit on LinkedIn</a> to stay current with the latest updates. <br>
+๐ค <a href="https://airflowsummit.org/in-person-events/">Check out the in-person events</a> planned for Airflow Summit 2022.</p>
<h1 id="february-22-2022">February 22, 2022</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.2.4</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.4/" target="_blank">https://pypi.org/project/apache-airflow/2.2.4/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.4/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.4/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.4/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.4/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.4/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.4/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.4/">https://pypi.org/project/apache-airflow/2.2.4/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.4/">https://airflow.apache.org/docs/apache-airflow/2.2.4/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.4/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.2.4/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.4/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.2.4/installation/installing-from-sources.html</a></p>
<h1 id="february-19-2022">February 19, 2022</h1>
-
<p>Airflow PMC welcomes two new Airlow Committers:</p>
-
<ul>
-<li><strong>Malthe Borch</strong> (<a href="https://github.com/malthe" target="_blank">@malthe</a>)</li>
-<li><strong>Josh Fell</strong> (<a href="https://github.com/josh-fell" target="_blank">@josh-fell</a>)</li>
+<li><strong>Malthe Borch</strong> (<a href="https://github.com/malthe">@malthe</a>)</li>
+<li><strong>Josh Fell</strong> (<a href="https://github.com/josh-fell">@josh-fell</a>)</li>
</ul>
-
<h1 id="january-10-2022">January 10, 2022</h1>
-
<p>We’ve just released Apache <strong>Airflow Helm chart 1.4.0</strong>.</p>
-
-<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow" target="_blank">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.4.0/" target="_blank">https://airflow.apache.org/docs/helm-chart/1.4.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.4.0/changelog.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.4.0/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.4.0/installing-helm-chart-from-sources.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.4.0/installing-helm-chart-from-sources.html</a></p>
-
+<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.4.0/">https://airflow.apache.org/docs/helm-chart/1.4.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.4.0/changelog.html">https://airflow.apache.org/docs/helm-chart/1.4.0/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.4.0/installing-helm-chart-from-sources.html">https://airflow.apache.org/docs/helm-chart/1.4.0/installing-helm-chart-from-sources.html</a></p>
<h1 id="january-4-2022">January 4, 2022</h1>
-
-<p>Airflow PMC welcomes <strong>Jed Cunningham</strong> (<a href="https://github.com/jedcunningham" target="_blank">@jedcunningham</a>) as the newest addition to Airflow PMC.</p>
-
+<p>Airflow PMC welcomes <strong>Jed Cunningham</strong> (<a href="https://github.com/jedcunningham">@jedcunningham</a>) as the newest addition to Airflow PMC.</p>
<h1 id="december-21-2021">December 21, 2021</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.2.3</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.3/" target="_blank">https://pypi.org/project/apache-airflow/2.2.3/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.3/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.3/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.3/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.3/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.3/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.3/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.3/">https://pypi.org/project/apache-airflow/2.2.3/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.3/">https://airflow.apache.org/docs/apache-airflow/2.2.3/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.3/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.2.3/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.3/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.2.3/installation/installing-from-sources.html</a></p>
<h1 id="november-15-2021">November 15, 2021</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.2.2</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.2/" target="_blank">https://pypi.org/project/apache-airflow/2.2.2/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.2/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.2/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.2/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.2/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.2/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.2/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.2/">https://pypi.org/project/apache-airflow/2.2.2/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.2/">https://airflow.apache.org/docs/apache-airflow/2.2.2/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.2/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.2.2/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.2/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.2.2/installation/installing-from-sources.html</a></p>
<h1 id="november-8-2021">November 8, 2021</h1>
-
<p>We’ve just released Apache <strong>Airflow Helm chart 1.3.0</strong>.</p>
-
-<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow" target="_blank">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.3.0/" target="_blank">https://airflow.apache.org/docs/helm-chart/1.3.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.3.0/changelog.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.3.0/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.3.0/installing-helm-chart-from-sources.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.3.0/installing-helm-chart-from-sources.html</a></p>
-
+<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.3.0/">https://airflow.apache.org/docs/helm-chart/1.3.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.3.0/changelog.html">https://airflow.apache.org/docs/helm-chart/1.3.0/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.3.0/installing-helm-chart-from-sources.html">https://airflow.apache.org/docs/helm-chart/1.3.0/installing-helm-chart-from-sources.html</a></p>
<h1 id="october-29-2021">October 29, 2021</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.2.1</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.1/" target="_blank">https://pypi.org/project/apache-airflow/2.2.1/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.1/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.1/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.1/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.1/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.1/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.1/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.1/">https://pypi.org/project/apache-airflow/2.2.1/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.1/">https://airflow.apache.org/docs/apache-airflow/2.2.1/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.1/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.2.1/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.1/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.2.1/installation/installing-from-sources.html</a></p>
<h1 id="october-11-2021">October 11, 2021</h1>
-
-<p>Weโve just released Apache <strong>Airflow 2.2.0</strong>. You can read more in the <a href="/blog/airflow-2.2.0/" target="_blank">What’s new in Apache Airflow 2.2.0</a> blog post.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.0/" target="_blank">https://pypi.org/project/apache-airflow/2.2.0/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.0/installation/installing-from-sources.html</a></p>
-
+<p>Weโve just released Apache <strong>Airflow 2.2.0</strong>. You can read more in the <a href="/blog/airflow-2.2.0/">What’s new in Apache Airflow 2.2.0</a> blog post.</p>
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.0/">https://pypi.org/project/apache-airflow/2.2.0/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/">https://airflow.apache.org/docs/apache-airflow/2.2.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.2.0/installation/installing-from-sources.html</a></p>
<h1 id="september-28-2021">September 28, 2021</h1>
-
<p>We’ve just released Apache <strong>Airflow Helm chart 1.2.0</strong>.</p>
-
-<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow" target="_blank">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.2.0/" target="_blank">https://airflow.apache.org/docs/helm-chart/1.2.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.2.0/changelog.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.2.0/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.2.0/installing-helm-chart-from-sources.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.2.0/installing-helm-chart-from-sources.html</a></p>
-
+<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/helm-chart/1.2.0/">https://airflow.apache.org/docs/helm-chart/1.2.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/helm-chart/1.2.0/changelog.html">https://airflow.apache.org/docs/helm-chart/1.2.0/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/helm-chart/1.2.0/installing-helm-chart-from-sources.html">https://airflow.apache.org/docs/helm-chart/1.2.0/installing-helm-chart-from-sources.html</a></p>
<h1 id="september-18-2021">September 18, 2021</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.1.4</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.4/" target="_blank">https://pypi.org/project/apache-airflow/2.1.4/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.4/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.4/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.4/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.4/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.4/installation/installing-from-sources.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.4/installation/installing-from-sources.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.4/">https://pypi.org/project/apache-airflow/2.1.4/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.4/">https://airflow.apache.org/docs/apache-airflow/2.1.4/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.4/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.1.4/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.4/installation/installing-from-sources.html">https://airflow.apache.org/docs/apache-airflow/2.1.4/installation/installing-from-sources.html</a></p>
<h1 id="august-31-2021">August 31, 2021</h1>
-
<p>Airflow PMC welcomes 2 new members to the Airflow PMC:</p>
-
<ul>
-<li><strong>Ephraim Anierobi</strong> (<a href="https://github.com/ephraimbuddy" target="_blank">@ephraimbuddy</a>)</li>
-<li><strong>Elad Kalif</strong> (<a href="https://github.com/eladkal" target="_blank">@eladkal</a>)</li>
+<li><strong>Ephraim Anierobi</strong> (<a href="https://github.com/ephraimbuddy">@ephraimbuddy</a>)</li>
+<li><strong>Elad Kalif</strong> (<a href="https://github.com/eladkal">@eladkal</a>)</li>
</ul>
-
<h1 id="august-27-2021">August 27, 2021</h1>
-
-<p>Airflow PMC welcomes <strong>Brent Bovenzi</strong> (<a href="https://github.com/bbovenzi" target="_blank">@bbovenzi</a>) as the newest Airflow Committer. ๐๐</p>
-
+<p>Airflow PMC welcomes <strong>Brent Bovenzi</strong> (<a href="https://github.com/bbovenzi">@bbovenzi</a>) as the newest Airflow Committer. ๐๐</p>
<h1 id="august-23-2021">August 23, 2021</h1>
-
<p>Weโve just released Apache <strong>Airflow 2.1.3</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.3/" target="_blank">https://pypi.org/project/apache-airflow/2.1.3/</a> <br />
-๐ Docs: โจ<a href="https://airflow.apache.org/docs/apache-airflow/2.1.3/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.3/</a> <br />
-๐ ๏ธ Changelog: โจ<a href="https://airflow.apache.org/docs/apache-airflow/2.1.3/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.3/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.3/installation.html#installing-airflow-from-released-sources-and-packages" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.3/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.3/">https://pypi.org/project/apache-airflow/2.1.3/</a> <br>
+๐ Docs: โจhttps://airflow.apache.org/docs/apache-airflow/2.1.3/ <br>
+๐ ๏ธ Changelog: โจhttps://airflow.apache.org/docs/apache-airflow/2.1.3/changelog.html <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.3/installation.html#installing-airflow-from-released-sources-and-packages">https://airflow.apache.org/docs/apache-airflow/2.1.3/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
<h1 id="july-7-2021">July 7, 2021</h1>
-
<p>We’ve just released Apache <strong>Airflow 2.1.2</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.2" target="_blank">https://pypi.org/project/apache-airflow/2.1.2</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.2/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.2/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.2/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.2/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.2/installation.html#installing-airflow-from-released-sources-and-packages" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.2/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.2">https://pypi.org/project/apache-airflow/2.1.2</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.2/">https://airflow.apache.org/docs/apache-airflow/2.1.2/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.2/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.1.2/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.2/installation.html#installing-airflow-from-released-sources-and-packages">https://airflow.apache.org/docs/apache-airflow/2.1.2/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
<h1 id="july-6-2021">July 6, 2021</h1>
-
-<p>Airflow PMC welcomes <strong>Aneesh Joseph</strong> (<a href="https://github.com/aneesh-joseph" target="_blank">@aneesh-joseph</a>) as the newest Airflow Committer. ๐๐</p>
-
+<p>Airflow PMC welcomes <strong>Aneesh Joseph</strong> (<a href="https://github.com/aneesh-joseph">@aneesh-joseph</a>) as the newest Airflow Committer. ๐๐</p>
<h1 id="july-2-2021">July 2, 2021</h1>
-
<p>We’ve just released Apache <strong>Airflow 2.1.1</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.1" target="_blank">https://pypi.org/project/apache-airflow/2.1.1</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.1/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.1/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.1/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.1/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.1/installation.html#installing-airflow-from-released-sources-and-packages" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.1/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.1">https://pypi.org/project/apache-airflow/2.1.1</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.1/">https://airflow.apache.org/docs/apache-airflow/2.1.1/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.1/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.1.1/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.1/installation.html#installing-airflow-from-released-sources-and-packages">https://airflow.apache.org/docs/apache-airflow/2.1.1/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
<h1 id="june-26-2021">June 26, 2021</h1>
-
<p>Airflow PMC welcomes 2 new committers:</p>
-
<ul>
-<li><strong>Jed Cunningham</strong> (<a href="https://github.com/jedcunningham" target="_blank">@jedcunningham</a>)</li>
-<li><strong>Tzu-ping Chung</strong> (<a href="https://github.com/uranusjr" target="_blank">@uranusjr</a>)</li>
+<li><strong>Jed Cunningham</strong> (<a href="https://github.com/jedcunningham">@jedcunningham</a>)</li>
+<li><strong>Tzu-ping Chung</strong> (<a href="https://github.com/uranusjr">@uranusjr</a>)</li>
</ul>
-
<h1 id="may-21-2021">May 21, 2021</h1>
-
<p>I’m happy to announce that Apache <strong>Airflow 2.1.0</strong> was just released. This one includes a raft of fixes and other small improvements, but some notable additions include:</p>
-
<ul>
-<li><p>A Create a DAG Calendar View to show the status of your DAG run across time more easily</p></li>
-
-<li><p>The cross-dag-dependencies view (which used to be an external plugin) is now part of core</p></li>
-
-<li><p>Mask passwords and sensitive info in task logs and UI (finally!)</p></li>
-
-<li><p>Improvmenets to webserver start up time (mostly around time spent syncing DAG permissions)</p></li>
+<li>
+<p>A Create a DAG Calendar View to show the status of your DAG run across time more easily</p>
+</li>
+<li>
+<p>The cross-dag-dependencies view (which used to be an external plugin) is now part of core</p>
+</li>
+<li>
+<p>Mask passwords and sensitive info in task logs and UI (finally!)</p>
+</li>
+<li>
+<p>Improvmenets to webserver start up time (mostly around time spent syncing DAG permissions)</p>
+</li>
</ul>
-
<p>Please note that <em>this</em> release no long includes the <code>HTTP</code> extra provider by default, as we discovered that it pulls in an LGPL dependency (via the <code>requests</code> module of all places) so it is now optional.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.0" target="_blank">https://pypi.org/project/apache-airflow/2.1.0</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.0/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.0/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.0/changelog.html</a> <br />
-๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.0/installation.html#installing-airflow-from-released-sources-and-packages" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.1.0/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.1.0">https://pypi.org/project/apache-airflow/2.1.0</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.0/">https://airflow.apache.org/docs/apache-airflow/2.1.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.0/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.1.0/changelog.html</a> <br>
+๐ชถ Sources: <a href="https://airflow.apache.org/docs/apache-airflow/2.1.0/installation.html#installing-airflow-from-released-sources-and-packages">https://airflow.apache.org/docs/apache-airflow/2.1.0/installation.html#installing-airflow-from-released-sources-and-packages</a></p>
<h1 id="may-20-2021">May 20, 2021</h1>
-
<p>The first release of our official helm chart for Apache Airflow is here!</p>
-
-<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow" target="_blank">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br />
-๐ Docs: โจ<a href="https://airflow.apache.org/docs/helm-chart/1.0" target="_blank">https://airflow.apache.org/docs/helm-chart/1.0</a>. <br />
-๐ Quick Start Installation Guide: <a href="https://airflow.apache.org/docs/helm-chart/1.0.0/quick-start.html" target="_blank">https://airflow.apache.org/docs/helm-chart/1.0.0/quick-start.html</a></p>
-
+<p>๐ฆ ArtifactHub: <a href="https://artifacthub.io/packages/helm/apache-airflow/airflow">https://artifacthub.io/packages/helm/apache-airflow/airflow</a> <br>
+๐ Docs: โจhttps://airflow.apache.org/docs/helm-chart/1.0. <br>
+๐ Quick Start Installation Guide: <a href="https://airflow.apache.org/docs/helm-chart/1.0.0/quick-start.html">https://airflow.apache.org/docs/helm-chart/1.0.0/quick-start.html</a></p>
<h1 id="april-6-2021">April 6, 2021</h1>
-
<p>Airflow PMC welcomes 2 new committers:</p>
-
<ul>
-<li><strong>Qian Yu</strong> (<a href="https://github.com/yuqian90" target="_blank">@yuqian90</a>)</li>
-<li><strong>Xinbin Huang</strong> (<a href="https://github.com/xinbinhuang" target="_blank">@xinbinhuang</a>)</li>
+<li><strong>Qian Yu</strong> (<a href="https://github.com/yuqian90">@yuqian90</a>)</li>
+<li><strong>Xinbin Huang</strong> (<a href="https://github.com/xinbinhuang">@xinbinhuang</a>)</li>
</ul>
-
<h1 id="march-21-2021">March 21, 2021</h1>
-
<p>Airflow Summit will be held online July 8-16, 2021. To register or propose a talk go to
-<a href="https://airflowsummit.org" target="_blank">official Airflow Summit website</a>.</p>
-
+<a href="https://airflowsummit.org">official Airflow Summit website</a>.</p>
<h1 id="march-18-2021">March 18, 2021</h1>
-
<p>We’ve just released <strong>Airflow Backport Provider Packages 2020.10.5</strong>.</p>
-
<h1 id="march-17-2021">March 17, 2021</h1>
-
<p>We’ve just released Apache <strong>Airflow 1.10.15</strong>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/1.10.15" target="_blank">https://pypi.org/project/apache-airflow/1.10.15</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/1.10.15/" target="_blank">https://airflow.apache.org/docs/apache-airflow/1.10.15/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/1.10.15">https://pypi.org/project/apache-airflow/1.10.15</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/1.10.15/">https://airflow.apache.org/docs/apache-airflow/1.10.15/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html">https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html</a></p>
<p>19 bug fixes, 3 Improvements & a couple of doc updates since 1.10.14</p>
-
<p><strong>Apache Airflow Elasticsearch Provider 1.0.3</strong> released</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.3/" target="_blank">https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.3/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/1.0.3/index.html#changelog" target="_blank">https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/1.0.3/index.html#changelog</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.3/">https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.3/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/1.0.3/index.html#changelog">https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/1.0.3/index.html#changelog</a></p>
<h1 id="march-14-2021">March 14, 2021</h1>
-
<p>Weโve just released <strong>Apache Airflow Upgrade Check 1.3.0</strong>:</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow-upgrade-check/1.3.0/" target="_blank">https://pypi.org/project/apache-airflow-upgrade-check/1.3.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://github.com/apache/airflow/tree/upgrade-check/1.3.0/airflow/upgrade#changelog" target="_blank">https://github.com/apache/airflow/tree/upgrade-check/1.3.0/airflow/upgrade#changelog</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow-upgrade-check/1.3.0/">https://pypi.org/project/apache-airflow-upgrade-check/1.3.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://github.com/apache/airflow/tree/upgrade-check/1.3.0/airflow/upgrade#changelog">https://github.com/apache/airflow/tree/upgrade-check/1.3.0/airflow/upgrade#changelog</a></p>
<p>This powers the <code>airflow upgrade_check</code> command to make upgrading to Apache Airflow 2.0 easier.</p>
-
<h1 id="march-1-2021">March 1, 2021</h1>
-
<p>Airflow PMC welcomes 3 new committers:</p>
-
<ul>
-<li><strong>James Timmins</strong> (<a href="https://github.com/jhtimmins" target="_blank">@James Timmins</a>)</li>
-<li><strong>Elad Kalif</strong> (<a href="https://github.com/eladkal" target="_blank">@eladkal</a>)</li>
-<li><strong>Daniel Standish</strong> (<a href="https://github.com/dstandish" target="_blank">@dstandish</a>)</li>
+<li><strong>James Timmins</strong> (<a href="https://github.com/jhtimmins">@James Timmins</a>)</li>
+<li><strong>Elad Kalif</strong> (<a href="https://github.com/eladkal">@eladkal</a>)</li>
+<li><strong>Daniel Standish</strong> (<a href="https://github.com/dstandish">@dstandish</a>)</li>
</ul>
-
<h1 id="february-23-2021">February 23, 2021</h1>
-
-<p>Airflow PMC welcomes <strong>Ephraim Anierobi</strong> (<a href="https://github.com/ephraimbuddy" target="_blank">@ephraimbuddy</a>) as the newest Airflow committer.</p>
-
+<p>Airflow PMC welcomes <strong>Ephraim Anierobi</strong> (<a href="https://github.com/ephraimbuddy">@ephraimbuddy</a>) as the newest Airflow committer.</p>
<h1 id="february-8-2021">February 8, 2021</h1>
-
<p>We’ve just released Apache Airflow 2.0.1.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.0.1" target="_blank">https://pypi.org/project/apache-airflow/2.0.1</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.0.1/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.0.1/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.0.1/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.0.1/changelog.html</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.0.1">https://pypi.org/project/apache-airflow/2.0.1</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.0.1/">https://airflow.apache.org/docs/apache-airflow/2.0.1/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.0.1/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.0.1/changelog.html</a></p>
<p>We also released 61 updated and 2 new providers.</p>
-
<h1 id="january-5-2021">January 5, 2021</h1>
-
-<p>Airflow PMC welcomes <strong>Vikram Koka</strong> (<a href="https://github.com/vikramkoka" target="_blank">@vikramkoka</a>) as the newest Airflow Committer.</p>
-
+<p>Airflow PMC welcomes <strong>Vikram Koka</strong> (<a href="https://github.com/vikramkoka">@vikramkoka</a>) as the newest Airflow Committer.</p>
<h1 id="december-25-2020">December 25, 2020</h1>
-
-<p>Airflow PMC welcomes <strong>Xiaodong Deng</strong> (<a href="https://github.com/XD-DENG" target="_blank">@XD-DENG</a>) as the newest addition to Airflow PMC.</p>
-
+<p>Airflow PMC welcomes <strong>Xiaodong Deng</strong> (<a href="https://github.com/XD-DENG">@XD-DENG</a>) as the newest addition to Airflow PMC.</p>
<h1 id="december-19-2020">December 19, 2020</h1>
-
<p>Jeremiah Lowin has resigned from the Airflow PMC.</p>
-
<p>Thank you Jeremiah for all your contributions and involvement in Airflow’s early years.</p>
-
<h1 id="december-17-2020">December 17, 2020</h1>
-
-<p>We’ve just released Apache Airflow 2.0.0. You can read more about what 2.0 brings in the <a href="/blog/airflow-two-point-oh-is-here/" target="_blank">announcement post</a>.</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.0.0" target="_blank">https://pypi.org/project/apache-airflow/2.0.0</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.0.0/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.0.0/</a></p>
-
+<p>We’ve just released Apache Airflow 2.0.0. You can read more about what 2.0 brings in the <a href="/blog/airflow-two-point-oh-is-here/">announcement post</a>.</p>
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.0.0">https://pypi.org/project/apache-airflow/2.0.0</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.0.0/">https://airflow.apache.org/docs/apache-airflow/2.0.0/</a></p>
<h1 id="december-10-2020">December 10, 2020</h1>
-
<p>We’ve just released Apache Airflow 1.10.14. This is a “bridge” release for Airflow 2.0.</p>
-
-<p>PyPI: <a href="https://pypi.org/project/apache-airflow/1.10.14" target="_blank">https://pypi.org/project/apache-airflow/1.10.14</a> <br />
-Docs: <a href="https://airflow.readthedocs.io/en/1.10.14/" target="_blank">https://airflow.readthedocs.io/en/1.10.14/</a> <br />
-Changelog: <a href="https://airflow.readthedocs.io/en/1.10.14/changelog.html" target="_blank">https://airflow.readthedocs.io/en/1.10.14/changelog.html</a></p>
-
+<p>PyPI: <a href="https://pypi.org/project/apache-airflow/1.10.14">https://pypi.org/project/apache-airflow/1.10.14</a> <br>
+Docs: <a href="https://airflow.readthedocs.io/en/1.10.14/">https://airflow.readthedocs.io/en/1.10.14/</a> <br>
+Changelog: <a href="https://airflow.readthedocs.io/en/1.10.14/changelog.html">https://airflow.readthedocs.io/en/1.10.14/changelog.html</a></p>
<h1 id="november-25-2020">November 25, 2020</h1>
-
<p>We’ve just released Apache Airflow 1.10.13</p>
-
-<p>PyPI: <a href="https://pypi.org/project/apache-airflow/1.10.13" target="_blank">https://pypi.org/project/apache-airflow/1.10.13</a> <br />
-Docs: <a href="https://airflow.readthedocs.io/en/1.10.13/" target="_blank">https://airflow.readthedocs.io/en/1.10.13/</a> <br />
-Changelog: <a href="https://airflow.readthedocs.io/en/1.10.13/changelog.html" target="_blank">https://airflow.readthedocs.io/en/1.10.13/changelog.html</a></p>
-
+<p>PyPI: <a href="https://pypi.org/project/apache-airflow/1.10.13">https://pypi.org/project/apache-airflow/1.10.13</a> <br>
+Docs: <a href="https://airflow.readthedocs.io/en/1.10.13/">https://airflow.readthedocs.io/en/1.10.13/</a> <br>
+Changelog: <a href="https://airflow.readthedocs.io/en/1.10.13/changelog.html">https://airflow.readthedocs.io/en/1.10.13/changelog.html</a></p>
<p>405 commits since 1.10.12 (6 New Features, 31 Improvements, 30 Bug Fixes, and tons of doc changes and internal changes)</p>
-
<h1 id="november-2-2020">November 2, 2020</h1>
-
<p>We’ve just released Airflow Backport Provider Packages 2020.10.5.</p>
-
<h1 id="october-28-2020">October 28, 2020</h1>
-
-<p>Airflow PMC welcomes <strong>Ryan Hamilton</strong> (<a href="https://github.com/ryanahamilton" target="_blank">@ryanahamilton</a>) as the new Airflow Committer.</p>
-
+<p>Airflow PMC welcomes <strong>Ryan Hamilton</strong> (<a href="https://github.com/ryanahamilton">@ryanahamilton</a>) as the new Airflow Committer.</p>
<h1 id="october-6-2020">October 6, 2020</h1>
-
<p>We’ve just released Airflow Backport Provider Packages 2020.10.5.</p>
-
<p>57 packages have been released.</p>
-
<h1 id="august-25-2020">August 25, 2020</h1>
-
<p>We’ve just released Airflow v1.10.12</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.12/" target="_blank">https://pypi.org/project/apache-airflow/1.10.12/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/docs/1.10.12/" target="_blank">https://airflow.apache.org/docs/1.10.12/</a></p>
-
-<p>ChangeLog - <a href="https://airflow.apache.org/docs/1.10.12/changelog.html" target="_blank">https://airflow.apache.org/docs/1.10.12/changelog.html</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.12/">https://pypi.org/project/apache-airflow/1.10.12/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/docs/1.10.12/">https://airflow.apache.org/docs/1.10.12/</a></p>
+<p>ChangeLog - <a href="https://airflow.apache.org/docs/1.10.12/changelog.html">https://airflow.apache.org/docs/1.10.12/changelog.html</a></p>
<p>113 commits since 1.10.10 (5 New Features, 23 Improvements, 23 Bug Fixes, and 14 doc changes)</p>
-
<h1 id="july-20-2020">July 20, 2020</h1>
-
-<p>Airflow PMC welcomes <strong>Leah Cole</strong> (<a href="https://github.com/leahecole" target="_blank">@leahecole</a>) and <strong>Ry Walker</strong> (<a href="https://github.com/ryw" target="_blank">@ryw</a>) as new Airflow Committers.</p>
-
+<p>Airflow PMC welcomes <strong>Leah Cole</strong> (<a href="https://github.com/leahecole">@leahecole</a>) and <strong>Ry Walker</strong> (<a href="https://github.com/ryw">@ryw</a>) as new Airflow Committers.</p>
<h1 id="july-10-2020">July 10, 2020</h1>
-
<p>We’ve just released Airflow v1.10.11</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.11/" target="_blank">https://pypi.org/project/apache-airflow/1.10.11/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/docs/1.10.11/" target="_blank">https://airflow.apache.org/docs/1.10.11/</a></p>
-
-<p>ChangeLog - <a href="https://airflow.apache.org/docs/1.10.11/changelog.html" target="_blank">https://airflow.apache.org/docs/1.10.11/changelog.html</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.11/">https://pypi.org/project/apache-airflow/1.10.11/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/docs/1.10.11/">https://airflow.apache.org/docs/1.10.11/</a></p>
+<p>ChangeLog - <a href="https://airflow.apache.org/docs/1.10.11/changelog.html">https://airflow.apache.org/docs/1.10.11/changelog.html</a></p>
<p>306 commits since 1.10.10 (12 New Features, 90 Improvements, 53 Bug Fixes, and several doc changes)</p>
-
<h1 id="july-8-2020">July 8, 2020</h1>
-
-<p>Airflow PMC welcomes <strong>Daniel Imberman</strong> (<a href="https://github.com/dimberman" target="_blank">@dimberman</a>), <strong>Tomek Turbaszek</strong> (<a href="https://github.com/turbaszek" target="_blank">@turbaszek</a>), and <strong>Kamil Breguลa</strong> (<a href="https://github.com/mik-laj" target="_blank">@mik-laj</a>) as new PMC members, and <strong>QP Hou</strong> (<a href="https://github.com/houqp" target="_blank">@houqp</a>) as a committer. Congrats!</p>
-
+<p>Airflow PMC welcomes <strong>Daniel Imberman</strong> (<a href="https://github.com/dimberman">@dimberman</a>), <strong>Tomek Turbaszek</strong> (<a href="https://github.com/turbaszek">@turbaszek</a>), and <strong>Kamil Breguลa</strong> (<a href="https://github.com/mik-laj">@mik-laj</a>) as new PMC members, and <strong>QP Hou</strong> (<a href="https://github.com/houqp">@houqp</a>) as a committer. Congrats!</p>
<h1 id="july-6-2020">July 6, 2020</h1>
-
-<p>The (virtual) Airflow Summit has begun โ you can watch along at <a href="https://airflowsummit.org/" target="_blank">airflowsummit.org</a></p>
-
+<p>The (virtual) Airflow Summit has begun โ you can watch along at <a href="https://airflowsummit.org/">airflowsummit.org</a></p>
<h1 id="jun-24-2020">Jun 24, 2020</h1>
-
<p>We’ve just released Airflow Backport Provider Packages 2020.6.24</p>
-
<p>The Backport provider packages make it possible to easily use Airflow 2.0 Operators, Hooks, Sensors, Secrets, Transfers in Airflow 1.10. More stats below, but the Backport Provider packages increase the number of easily-available integrations for Airflow 1.10 users by a whopping <strong>55%</strong>.</p>
-
<ul>
<li>We have <strong>58</strong> backport packages in total. <strong>599</strong> classes (Operators, Hooks, Transfers, Sensors, Secrets)</li>
<li>We have <strong>213</strong> new (!) classes that have not been easily available to 1.10 users so far:
-
<ul>
<li>Operators: 150</li>
<li>Transfers: 12</li>
<li>Sensors: 14</li>
<li>Hooks: 37</li>
<li>Secrets: 0</li>
-</ul></li>
+</ul>
+</li>
<li>We have 386 classes that were moved. Quite a number of those (hard to say exactly how many) got new features, options, parameters.
-
<ul>
<li>Operators: 204</li>
<li>Transfers: 36</li>
<li>Sensors: 46</li>
<li>Hooks: 96</li>
<li>Secrets: 4</li>
-</ul></li>
</ul>
-
+</li>
+</ul>
<p><strong>List of the backport provider packages:</strong></p>
-
<blockquote>
<ol>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.6.24/" target="_blank">Amazon</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.6.24/" target="_blank">Apache HDFS</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.6.24/" target="_blank">Apache Hive</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.6.24/" target="_blank">Apache Livy</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.6.24/" target="_blank">Apache Pig</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.6.24/" target="_blank">Apache Pinot</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.6.24/" target="_blank">Apache Spark</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.6.24/" target="_blank">Apache Sqoop</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.6.24/" target="_blank">Azure</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.6.24/" target="_blank">Cassandra</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-celery/2020.6.24/" target="_blank">Celery</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.6.24/" target="_blank">Cloudant</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.6.24/" target="_blank">Databricks</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.6.24/" target="_blank">Datadog</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.6.24/" target="_blank">Dingding</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-discord/2020.6.24/" target="_blank">Discord</a></li>
-<li><a href="https://p0ypi.org/project/apache-airflow-backport-providers-docker/2020.6.24/" target="_blank">Docker</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.6.24/" target="_blank">Druid</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.6.24/" target="_blank">Elasticsearch</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-email/2020.6.24/" target="_blank">Email</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.6.24/" target="_blank">Exasol</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.6.24/" target="_blank">Facebook</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.6.24/" target="_blank">FTP</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-google/2020.6.24/" target="_blank">Google</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.6.24/" target="_blank">GRPC</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.6.24/" target="_blank">Hashicorp</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-http/2020.6.24/" target="_blank">HTTP</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-imap/2020.6.24/" target="_blank">IMAP</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.6.24/" target="_blank">JDBC</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.6.24/" target="_blank">Jenkins</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-jira/2020.6.24/" target="_blank">JIRA</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.6.24/" target="_blank">Mongo</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.6.24/" target="_blank">MsSQL</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.6.24/" target="_blank">MySQL</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.6.24/" target="_blank">ODBC</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.6.24/" target="_blank">OpenFAAS</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.6.24/" target="_blank">OpsGenie</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.6.24/" target="_blank">Oracle</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.6.24/" target="_blank">PagerDuty</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.6.24/" target="_blank">Postgres</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-presto/2020.6.24/" target="_blank">Presto</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.6.24/" target="_blank">Qubole</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-redis/2020.6.24/" target="_blank">Redis</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.6.24/" target="_blank">Salesforce</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-samba/2020.6.24/" target="_blank">Samba</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-segment/2020.6.24/" target="_blank">Segment</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.6.24/" target="_blank">SFTP</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.6.24/" target="_blank">Singularity</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-slack/2020.6.24/" target="_blank">Slack</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.6.24/" target="_blank">Snowflake</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.6.24/" target="_blank">Sqlite</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.6.24/" target="_blank">SSH</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.6.24/" target="_blank">Vertica</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.6.24/" target="_blank">Winrm</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.6.24/" target="_blank">Yandex</a></li>
-<li><a href="https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.6.24/" target="_blank">Zendesk</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.6.24/">Amazon</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.6.24/">Apache HDFS</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.6.24/">Apache Hive</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.6.24/">Apache Livy</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.6.24/">Apache Pig</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.6.24/">Apache Pinot</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.6.24/">Apache Spark</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.6.24/">Apache Sqoop</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.6.24/">Azure</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.6.24/">Cassandra</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-celery/2020.6.24/">Celery</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.6.24/">Cloudant</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.6.24/">Databricks</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.6.24/">Datadog</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.6.24/">Dingding</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-discord/2020.6.24/">Discord</a></li>
+<li><a href="https://p0ypi.org/project/apache-airflow-backport-providers-docker/2020.6.24/">Docker</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.6.24/">Druid</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.6.24/">Elasticsearch</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-email/2020.6.24/">Email</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.6.24/">Exasol</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.6.24/">Facebook</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.6.24/">FTP</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-google/2020.6.24/">Google</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.6.24/">GRPC</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.6.24/">Hashicorp</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-http/2020.6.24/">HTTP</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-imap/2020.6.24/">IMAP</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.6.24/">JDBC</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.6.24/">Jenkins</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-jira/2020.6.24/">JIRA</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.6.24/">Mongo</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.6.24/">MsSQL</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.6.24/">MySQL</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.6.24/">ODBC</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.6.24/">OpenFAAS</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.6.24/">OpsGenie</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.6.24/">Oracle</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.6.24/">PagerDuty</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.6.24/">Postgres</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-presto/2020.6.24/">Presto</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.6.24/">Qubole</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-redis/2020.6.24/">Redis</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.6.24/">Salesforce</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-samba/2020.6.24/">Samba</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-segment/2020.6.24/">Segment</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.6.24/">SFTP</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.6.24/">Singularity</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-slack/2020.6.24/">Slack</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.6.24/">Snowflake</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.6.24/">Sqlite</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.6.24/">SSH</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.6.24/">Vertica</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.6.24/">Winrm</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.6.24/">Yandex</a></li>
+<li><a href="https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.6.24/">Zendesk</a></li>
</ol>
</blockquote>
-
<h1 id="apr-9-2020">Apr 9, 2020</h1>
-
<p>We’ve just released Airflow v1.10.10</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.10/" target="_blank">https://pypi.org/project/apache-airflow/1.10.10/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/docs/1.10.10/" target="_blank">https://airflow.apache.org/docs/1.10.10/</a></p>
-
-<p>Changelog - <a href="https://airflow.apache.org/docs/1.10.10/changelog.html" target="_blank">https://airflow.apache.org/docs/1.10.10/changelog.html</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.10/">https://pypi.org/project/apache-airflow/1.10.10/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/docs/1.10.10/">https://airflow.apache.org/docs/1.10.10/</a></p>
+<p>Changelog - <a href="https://airflow.apache.org/docs/1.10.10/changelog.html">https://airflow.apache.org/docs/1.10.10/changelog.html</a></p>
<p>199 commits since 1.10.9 (11 New Features, 43 Improvements, 44 Bug Fixes, and several doc changes)</p>
-
<h1 id="mar-31-2020">Mar 31, 2020</h1>
-
-<p>Airflow PMC welcomes <strong>Jiajie Zhong</strong> (<a href="https://github.com/zhongjiajie" target="_blank">@zhongjiajie</a>) as its new committer. Congrats!</p>
-
+<p>Airflow PMC welcomes <strong>Jiajie Zhong</strong> (<a href="https://github.com/zhongjiajie">@zhongjiajie</a>) as its new committer. Congrats!</p>
<p>and</p>
-
-<p><a href="https://www.meetup.com/pl-PL/Warsaw-Airflow-Meetup/events/269763588/" target="_blank">Second Warsaw Airflow Meetup</a> is happening next week and it will be an online event!</p>
-
+<p><a href="https://www.meetup.com/pl-PL/Warsaw-Airflow-Meetup/events/269763588/">Second Warsaw Airflow Meetup</a> is happening next week and it will be an online event!</p>
<h1 id="feb-7-2020">Feb 7, 2020</h1>
-
<p>We’ve just released Airflow v1.10.8</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.8/" target="_blank">https://pypi.org/project/apache-airflow/1.10.8/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/docs/1.10.8/" target="_blank">https://airflow.apache.org/docs/1.10.8/</a></p>
-
-<p>Changelog - <a href="http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07" target="_blank">http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.8/">https://pypi.org/project/apache-airflow/1.10.8/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/docs/1.10.8/">https://airflow.apache.org/docs/1.10.8/</a></p>
+<p>Changelog - <a href="http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07">http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07</a></p>
<p>160 commits since 1.10.7 (4 new features, 42 improvements, 36 bug fixes, and several doc changes)</p>
-
<p>and</p>
-
<p>We’ve just released Airflow 1.10.9 (this one is a quick fix to work around the breaking release of Werkzeug 1.0)</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.9/" target="_blank">https://pypi.org/project/apache-airflow/1.10.9/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/docs/1.10.9/" target="_blank">https://airflow.apache.org/docs/1.10.9/</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.9/">https://pypi.org/project/apache-airflow/1.10.9/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/docs/1.10.9/">https://airflow.apache.org/docs/1.10.9/</a></p>
<p>2 commits since 1.10.8 :)</p>
-
<h1 id="dec-24-2019">Dec 24, 2019</h1>
-
<p>We’ve just released Airflow v1.10.7</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.7/" target="_blank">https://pypi.org/project/apache-airflow/1.10.7/</a></p>
-
-<p>Docs - <a href="https://airflow.readthedocs.io/en/1.10.7/" target="_blank">https://airflow.readthedocs.io/en/1.10.7/</a></p>
-
-<p>Changelog - <a href="https://github.com/apache/airflow/blob/1.10.7/CHANGELOG.txt" target="_blank">https://github.com/apache/airflow/blob/1.10.7/CHANGELOG.txt</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.7/">https://pypi.org/project/apache-airflow/1.10.7/</a></p>
+<p>Docs - <a href="https://airflow.readthedocs.io/en/1.10.7/">https://airflow.readthedocs.io/en/1.10.7/</a></p>
+<p>Changelog - <a href="https://github.com/apache/airflow/blob/1.10.7/CHANGELOG.txt">https://github.com/apache/airflow/blob/1.10.7/CHANGELOG.txt</a></p>
<p>217 commits since 1.10.6 including some Critical Bugfixes and Performance Improvements (17 new features, 57 improvements, 52 bug fixes, and several doc changes)</p>
-
<h1 id="dec-20-2019">Dec 20, 2019</h1>
-
-<p>Airflow PMC welcomes <strong>Tomasz Urbaszek</strong> (<a href="https://github.com/turbaszek" target="_blank">@turbaszek</a>) as its new committer. Congrats!</p>
-
+<p>Airflow PMC welcomes <strong>Tomasz Urbaszek</strong> (<a href="https://github.com/turbaszek">@turbaszek</a>) as its new committer. Congrats!</p>
<h1 id="nov-30-2019">Nov 30, 2019</h1>
-
-<p>Airflow PMC welcomes <strong>Kengo Seki</strong> (<a href="https://github.com/sekikn" target="_blank">@sekikn</a>) to both its committer and PMC ranks. Congrats!</p>
-
+<p>Airflow PMC welcomes <strong>Kengo Seki</strong> (<a href="https://github.com/sekikn">@sekikn</a>) to both its committer and PMC ranks. Congrats!</p>
<h1 id="nov-27-2019">Nov 27, 2019</h1>
-
-<p>New website for Apache Airflow is live : <a href="https://airflow.apache.org" target="_blank">https://airflow.apache.org</a></p>
-
+<p>New website for Apache Airflow is live : <a href="https://airflow.apache.org">https://airflow.apache.org</a></p>
<p>Same URL with more & better webpages</p>
-
<h1 id="nov-21-2019">Nov 21, 2019</h1>
-
-<p>Airflow PMC has voted in & promoted <strong>Aizhamal Nurmamat kyzy</strong> (<a href="https://github.com/Aizhamal" target="_blank">@Aizhamal</a>) and Kevin Yang (<a href="https://github.com/KevinYang21" target="_blank">@KevinYang21</a>) to be a part of Airflow PMC.</p>
-
+<p>Airflow PMC has voted in & promoted <strong>Aizhamal Nurmamat kyzy</strong> (<a href="https://github.com/Aizhamal">@Aizhamal</a>) and Kevin Yang (<a href="https://github.com/KevinYang21">@KevinYang21</a>) to be a part of Airflow PMC.</p>
<h1 id="oct-28-2019">Oct 28, 2019</h1>
-
<p>We’ve just released ApacheAirflow v1.10.6</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.6/" target="_blank">https://pypi.org/project/apache-airflow/1.10.6/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/1.10.6/" target="_blank">https://airflow.apache.org/1.10.6/</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.6/">https://pypi.org/project/apache-airflow/1.10.6/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/1.10.6/">https://airflow.apache.org/1.10.6/</a></p>
<p>168 commits since 1.10.5 (including 6 new features, 35 improvements, 38 bug fixes, and the usual swathe of doc fixes and CI improvements)</p>
-
<h1 id="oct-18-2019">Oct 18, 2019</h1>
-
-<p>Airflow PMC has voted in & promoted Jarek Potiuk (<a href="https://github.com/higrys" target="_blank">@higrys</a>) to be a PMC Member.</p>
-
+<p>Airflow PMC has voted in & promoted Jarek Potiuk (<a href="https://github.com/higrys">@higrys</a>) to be a PMC Member.</p>
<p>Jarek has been one of the most active community members and has spread the word about Airflow Well deserved Jarek, congratulations</p>
-
<h1 id="sep-24-2019">Sep 24, 2019</h1>
-
<p>What do you think of our new logo? <em>#AirflowLogo</em> <em>#ApacheAirflow</em></p>
-
-<p><img src="/images/airflow_dark_bg.png" alt="Airflow" /></p>
-
+<p><img src="/images/airflow_dark_bg.png" alt="Airflow"></p>
<h1 id="sep-18-2019">Sep 18, 2019</h1>
-
<p>Two meetups are happening soon:</p>
-
<ol>
-<li><a href="https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/events/263013440/" target="_blank">Seattle, WA on Thursday 19th</a></li>
-<li><a href="https://www.meetup.com/London-Apache-Airflow-Meetup/events/264521028/" target="_blank">London, UK on Tuesday 24th</a></li>
+<li><a href="https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/events/263013440/">Seattle, WA on Thursday 19th</a></li>
+<li><a href="https://www.meetup.com/London-Apache-Airflow-Meetup/events/264521028/">London, UK on Tuesday 24th</a></li>
</ol>
-
<p>We’ll put links to any recordings here once they are available.</p>
-
<h1 id="sep-9-2019">Sep 9, 2019</h1>
-
<p>The PMC has just added two new Airflow Committers :</p>
-
<ul>
-<li><p>Felix Uellendall (<a href="https://github.com/feluelle/" target="_blank">@feluelle</a>)</p></li>
-
-<li><p>Chao-han Tsai (<a href="https://github.com/milton0825/" target="_blank">@milton0825</a>)</p></li>
+<li>
+<p>Felix Uellendall (<a href="https://github.com/feluelle/">@feluelle</a>)</p>
+</li>
+<li>
+<p>Chao-han Tsai (<a href="https://github.com/milton0825/">@milton0825</a>)</p>
+</li>
</ul>
-
<h1 id="sep-4-2019">Sep 4, 2019</h1>
-
<p>We’ve just released ApacheAirflow v1.10.5</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.5/" target="_blank">https://pypi.org/project/apache-airflow/1.10.5/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/1.10.5/" target="_blank">https://airflow.apache.org/1.10.5/</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.5/">https://pypi.org/project/apache-airflow/1.10.5/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/1.10.5/">https://airflow.apache.org/1.10.5/</a></p>
<p>95 commits since 1.10.4 (including 3 new features, 17 improvements, 23 bug fixes, and lots more doc fixes and CI improvements)</p>
-
<h1 id="aug-6-2019">Aug 6, 2019</h1>
-
<p>We’ve just released ApacheAirflow v1.10.4</p>
-
-<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.4/" target="_blank">https://pypi.org/project/apache-airflow/1.10.4/</a></p>
-
-<p>Docs - <a href="https://airflow.apache.org/1.10.4/" target="_blank">https://airflow.apache.org/1.10.4/</a></p>
-
+<p>PyPI - <a href="https://pypi.org/project/apache-airflow/1.10.4/">https://pypi.org/project/apache-airflow/1.10.4/</a></p>
+<p>Docs - <a href="https://airflow.apache.org/1.10.4/">https://airflow.apache.org/1.10.4/</a></p>
<p>377 commits since 1.10.3 (including 50 new features, 107 improvements, 91 bug fixes, and lots more doc fixes and CI improvements)</p>
-
<h1 id="may-15-2019">May 15, 2019</h1>
-
-<p>The community welcomes the latest Apache Airflow meetup group: The <a href="https://www.meetup.com/Melbourne-Apache-Airflow-Meetup/" target="_blank">Melbourne Apache Airflow meetup</a>. Stay tuned for its first meetup event!</p>
-
+<p>The community welcomes the latest Apache Airflow meetup group: The <a href="https://www.meetup.com/Melbourne-Apache-Airflow-Meetup/">Melbourne Apache Airflow meetup</a>. Stay tuned for its first meetup event!</p>
<h1 id="may-2-2019">May 2, 2019</h1>
-
<p>The Apache Airflow PMC welcomes a slew of new committers to it ranks! The following contributors are now Airflow Committers:</p>
-
<ul>
-<li>Daniel Imberman (<a href="https://github.com/dimberman" target="_blank">@dimberman</a>)</li>
-<li>Jarek Potiuk (<a href="https://github.com/higrys" target="_blank">@higrys</a>)</li>
-<li>Bas Harenslak (<a href="https://github.com/BasPH" target="_blank">@BasPH</a>)</li>
-<li>Joshua Carp (<a href="https://github.com/jmcarp" target="_blank">@jmcarp</a>)</li>
-<li>Kevin Yang (<a href="https://github.com/KevinYang21" target="_blank">@KevinYang21</a>)</li>
-<li>Kamil Bregula (<a href="https://github.com/mik-laj" target="_blank">@mik-laj</a>)</li>
+<li>Daniel Imberman (<a href="https://github.com/dimberman">@dimberman</a>)</li>
+<li>Jarek Potiuk (<a href="https://github.com/higrys">@higrys</a>)</li>
+<li>Bas Harenslak (<a href="https://github.com/BasPH">@BasPH</a>)</li>
+<li>Joshua Carp (<a href="https://github.com/jmcarp">@jmcarp</a>)</li>
+<li>Kevin Yang (<a href="https://github.com/KevinYang21">@KevinYang21</a>)</li>
+<li>Kamil Bregula (<a href="https://github.com/mik-laj">@mik-laj</a>)</li>
</ul>
-
<p>Congratulations folks - Very well deserved.</p>
-
<h1 id="apr-10-2019">Apr 10, 2019</h1>
-
<p>New Apache Airflow release 1.10.3 is out!</p>
-
-<p>Pypi - <a href="https://pypi.org/project/apache-airflow/1.10.3/" target="_blank">https://pypi.org/project/apache-airflow/1.10.3/</a> (Run <code>pip install apache-airflow</code>)</p>
-
-<p>Changelog - <a href="https://airflow.apache.org/1.10.3/changelog.html#airflow-1-10-3-2019-04-09" target="_blank">https://airflow.apache.org/1.10.3/changelog.html#airflow-1-10-3-2019-04-09</a></p>
-
+<p>Pypi - <a href="https://pypi.org/project/apache-airflow/1.10.3/">https://pypi.org/project/apache-airflow/1.10.3/</a> (Run <code>pip install apache-airflow</code>)</p>
+<p>Changelog - <a href="https://airflow.apache.org/1.10.3/changelog.html#airflow-1-10-3-2019-04-09">https://airflow.apache.org/1.10.3/changelog.html#airflow-1-10-3-2019-04-09</a></p>
<p>This release no longer needs the SLUGIFY_USES_TEXT_UNIDECODE/AIRFLOW_GPL_UNIDECODE environment variables at install time to avoid the GPL dependency!</p>
-
<h1 id="apr-3-2019">Apr 3, 2019</h1>
-
-<p>The <a href="https://www.meetup.com/NYC-Apache-Airflow-Meetup/" target="_blank">Apache Airflow NYC meetup</a> is having its next meetup on April 29, 2019 โ more details <a href="https://www.meetup.com/NYC-Apache-Airflow-Meetup/events/260257700/" target="_blank">here</a>.</p>
-
+<p>The <a href="https://www.meetup.com/NYC-Apache-Airflow-Meetup/">Apache Airflow NYC meetup</a> is having its next meetup on April 29, 2019 โ more details <a href="https://www.meetup.com/NYC-Apache-Airflow-Meetup/events/260257700/">here</a>.</p>
<p>If you are in or around NYC, please be sure to check it out!</p>
-
<h1 id="mar-18-2019">Mar 18, 2019</h1>
-
-<p>The Apache Airflow PMC welcomes new committer, <strong>Daniel Imberman</strong> (<a href="https://github.com/dimberman" target="_blank">@dimberman</a>).</p>
-
+<p>The Apache Airflow PMC welcomes new committer, <strong>Daniel Imberman</strong> (<a href="https://github.com/dimberman">@dimberman</a>).</p>
<h1 id="mar-7-2019">Mar 7, 2019</h1>
-
-<p>The Apache Airflow PMC welcomes new committer, <strong>Xiaodong Deng</strong> (<a href="https://github.com/XD-DENG" target="_blank">XD-DENG</a>)</p>
-
+<p>The Apache Airflow PMC welcomes new committer, <strong>Xiaodong Deng</strong> (<a href="https://github.com/XD-DENG">XD-DENG</a>)</p>
<h1 id="feb-14-2019">Feb 14, 2019</h1>
-
-<p>We are pleased to announce our newest Meetup, this one in <a href="https://www.meetup.com/Portland-Apache-Airflow-Meetup/" target="_blank">Portland, Oregon, USA</a>.</p>
-
+<p>We are pleased to announce our newest Meetup, this one in <a href="https://www.meetup.com/Portland-Apache-Airflow-Meetup/">Portland, Oregon, USA</a>.</p>
<h1 id="jan-22-2019">Jan 22, 2019</h1>
-
<p>New Apache Airflow release 1.10.2 is out!</p>
-
-<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow" target="_blank">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
-
-<p>Changelog - <a href="https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19" target="_blank">https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19</a></p>
-
+<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
+<p>Changelog - <a href="https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19">https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19</a></p>
<p>By default one of Airflow’s dependencies installs a GPL dependency (unidecode). To avoid this dependency set <strong>SLUGIFY_USES_TEXT_UNIDECODE=yes</strong> in your environment when you install or upgrade Airflow. To force installing the GPL version set <strong>AIRFLOW_GPL_UNIDECODE</strong>. One of these two environment variables must be specified.</p>
-
<h1 id="jan-9-2019">Jan 9, 2019</h1>
-
-<p>The <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/" target="_blank">Apache Airflow Paris meetup</a> is having its second meetup on Feb 6, 2019 โ more details <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/events/257702755/" target="_blank">here</a>.</p>
-
+<p>The <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/">Apache Airflow Paris meetup</a> is having its second meetup on Feb 6, 2019 โ more details <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/events/257702755/">here</a>.</p>
<p>If you are in or around Paris, please be sure to check it out!</p>
-
<h1 id="jan-8-2019">Jan 8, 2019</h1>
-
<p>Apache Airflow graduates from the Incubator and is now a TLP!</p>
-
<ul>
-<li><p>ASF PR announcements</p>
-
+<li>
+<p>ASF PR announcements</p>
<ul>
-<li><a href="http://globenewswire.com/news-release/2019/01/08/1681851/0/en/The-Apache-Software-Foundation-Announces-Apache-Airflow-as-a-Top-Level-Project.html" target="_blank">GlobeNewswire</a></li>
-<li><a href="https://s.apache.org/LeeE" target="_blank">ASF “Foundation” Blog</a></li>
-<li><a href="https://twitter.com/TheASF/status/1082593312421208064" target="_blank">The ASF Twitter Feed</a></li>
-<li><a href="https://www.linkedin.com/company/the-apache-software-foundation" target="_blank">The ASF on LinkedIn</a></li>
-<li>You can find more announcements on our twitter account as well : <a href="https://twitter.com/ApacheAirflow" target="_blank">@ApacheAirflow</a></li>
-</ul></li>
-
-<li><p>Community Blogs</p>
-
+<li><a href="http://globenewswire.com/news-release/2019/01/08/1681851/0/en/The-Apache-Software-Foundation-Announces-Apache-Airflow-as-a-Top-Level-Project.html">GlobeNewswire</a></li>
+<li><a href="https://s.apache.org/LeeE">ASF “Foundation” Blog</a></li>
+<li><a href="https://twitter.com/TheASF/status/1082593312421208064">The ASF Twitter Feed</a></li>
+<li><a href="https://www.linkedin.com/company/the-apache-software-foundation">The ASF on LinkedIn</a></li>
+<li>You can find more announcements on our twitter account as well : <a href="https://twitter.com/ApacheAirflow">@ApacheAirflow</a></li>
+</ul>
+</li>
+<li>
+<p>Community Blogs</p>
<ul>
-<li><a href="https://medium.com/wbaa/apache-airflow-announced-as-a-top-level-project-how-does-ing-wb-advanced-analytics-use-airflow-ed7990ffba9c" target="_blank">Apache Airflow announced as a Top-Level Project โ How does ING WB Advanced Analytics use Airflow?</a></li>
-<li><a href="https://medium.com/@r39132/apache-airflow-grows-up-c820ee8a8324" target="_blank">Apache Airflow Grows Up!</a></li>
-</ul></li>
+<li><a href="https://medium.com/wbaa/apache-airflow-announced-as-a-top-level-project-how-does-ing-wb-advanced-analytics-use-airflow-ed7990ffba9c">Apache Airflow announced as a Top-Level Project โ How does ING WB Advanced Analytics use Airflow?</a></li>
+<li><a href="https://medium.com/@r39132/apache-airflow-grows-up-c820ee8a8324">Apache Airflow Grows Up!</a></li>
+</ul>
+</li>
</ul>
-
<h1 id="nov-21-2018">Nov 21, 2018</h1>
-
<p>New Apache Airflow release 1.10.1-incubating is out!</p>
-
-<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow" target="_blank">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
-
-<p>Changelog - <a href="https://github.com/apache/incubator-airflow/blob/v1-10-test/CHANGELOG.txt" target="_blank">https://github.com/apache/incubator-airflow/blob/v1-10-test/CHANGELOG.txt</a></p>
-
+<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
+<p>Changelog - <a href="https://github.com/apache/incubator-airflow/blob/v1-10-test/CHANGELOG.txt">https://github.com/apache/incubator-airflow/blob/v1-10-test/CHANGELOG.txt</a></p>
<p>By default one of Airflow’s dependencies installs a GPL dependency (unidecode). To avoid this dependency set <strong>SLUGIFY_USES_TEXT_UNIDECODE=yes</strong> in your environment when you install or upgrade Airflow. To force installing the GPL version set <strong>AIRFLOW_GPL_UNIDECODE</strong>. One of these two environment variables must be specified.</p>
-
<h1 id="oct-16-2018">Oct 16, 2018</h1>
-
-<p>We are excited to announce that the <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/" target="_blank">Paris Apache Airflow Meetup</a> will be hosting its <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/events/255336471/" target="_blank">inaugural meetup on Nov 21</a></p>
-
-<p>Speakers include <a href="https://www.chauffeur-prive.com/" target="_blank">Chauffeur Prive</a> (event organizer). More to come soon!</p>
-
+<p>We are excited to announce that the <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/">Paris Apache Airflow Meetup</a> will be hosting its <a href="https://www.meetup.com/Paris-Apache-Airflow-Meetup/events/255336471/">inaugural meetup on Nov 21</a></p>
+<p>Speakers include <a href="https://www.chauffeur-prive.com/">Chauffeur Prive</a> (event organizer). More to come soon!</p>
<h1 id="sept-17-2018">Sept 17, 2018</h1>
-
-<p>The Singapore Big Data Meetup will host “Intro to Airflow” via <a href="https://github.com/XD-DENG" target="_blank">@XD-DENG</a> on Sept 27, 2018! Check it out! <a href="http://bit.ly/2PMgABT" target="_blank">http://bit.ly/2PMgABT</a></p>
-
+<p>The Singapore Big Data Meetup will host “Intro to Airflow” via <a href="https://github.com/XD-DENG">@XD-DENG</a> on Sept 27, 2018! Check it out! <a href="http://bit.ly/2PMgABT">http://bit.ly/2PMgABT</a></p>
<h1 id="sept-10-2018">Sept 10, 2018</h1>
-
-<p>We are excited to announce that the <a href="https://www.meetup.com/London-Apache-Airflow-Meetup/" target="_blank">London Apache Airflow Meetup</a> will be hosting its <a href="https://www.meetup.com/London-Apache-Airflow-Meetup/events/253076795/" target="_blank">inaugural meetup on Sept 20</a>.</p>
-
+<p>We are excited to announce that the <a href="https://www.meetup.com/London-Apache-Airflow-Meetup/">London Apache Airflow Meetup</a> will be hosting its <a href="https://www.meetup.com/London-Apache-Airflow-Meetup/events/253076795/">inaugural meetup on Sept 20</a>.</p>
<p>Speakers include London committer Ash Berlin-Taylor & Ben Marengo from Just Eat!</p>
-
<h1 id="sept-9-2018">Sept 9, 2018</h1>
-
-<p>The <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/" target="_blank">Bay Area Apache Airflow Meetup</a> group will be hosting its <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/253105418/" target="_blank">next meetup at Google on Sept 24!</a>.</p>
-
+<p>The <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/">Bay Area Apache Airflow Meetup</a> group will be hosting its <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/253105418/">next meetup at Google on Sept 24!</a>.</p>
<p>Five 20-minute talks followed by 6 lightning sessions! Come one, come all!</p>
-
<h1 id="sept-8-2018">Sept 8, 2018</h1>
-
-<p>We welcome the 7th & latest regional Apache Airflow Meetup, this one in <a href="https://lnkd.in/dYt2hE8" target="_blank">Seattle</a>.</p>
-
+<p>We welcome the 7th & latest regional Apache Airflow Meetup, this one in <a href="https://lnkd.in/dYt2hE8">Seattle</a>.</p>
<h1 id="aug-26-2018">Aug 26, 2018</h1>
-
-<p>The Apache Airflow Meetup group in Amsterdam will be hosting its <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/events/253673642/" target="_blank">2nd meetup at GoDataDriven on Sep 12</a>.</p>
-
+<p>The Apache Airflow Meetup group in Amsterdam will be hosting its <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/events/253673642/">2nd meetup at GoDataDriven on Sep 12</a>.</p>
<h1 id="aug-20-2018">Aug 20, 2018</h1>
-
<p>New Apache Airflow release 1.10.0-incubating is out!</p>
-
-<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow" target="_blank">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
-
-<p>Changelog - <a href="https://github.com/apache/incubator-airflow/blob/8100f1f/CHANGELOG.txt" target="_blank">https://github.com/apache/incubator-airflow/blob/8100f1f/CHANGELOG.txt</a></p>
-
+<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
+<p>Changelog - <a href="https://github.com/apache/incubator-airflow/blob/8100f1f/CHANGELOG.txt">https://github.com/apache/incubator-airflow/blob/8100f1f/CHANGELOG.txt</a></p>
<p>By default one of Airflow’s dependencies installs a GPL dependency (unidecode). To avoid this dependency set <strong>SLUGIFY_USES_TEXT_UNIDECODE=yes</strong> in your environment when you install or upgrade Airflow. To force installing the GPL version set <strong>AIRFLOW_GPL_UNIDECODE</strong>. One of these two environment variables must be specified.</p>
-
<h1 id="aug-3-2018">Aug 3, 2018</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, Tao Feng (<a href="https://github.com/feng-tao" target="_blank">@feng-tao</a>)</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, Tao Feng (<a href="https://github.com/feng-tao">@feng-tao</a>)</p>
<h1 id="jun-1-2018">Jun 1, 2018</h1>
-
-<p>We’re happy to welcome a new Apache Airflow Meetup group – this one in <a href="https://www.meetup.com/London-Apache-Airflow-Meetup/" target="_blank">London, UK</a></p>
-
+<p>We’re happy to welcome a new Apache Airflow Meetup group – this one in <a href="https://www.meetup.com/London-Apache-Airflow-Meetup/">London, UK</a></p>
<h1 id="may-23-2018">May 23, 2018</h1>
-
-<p>We’re happy to welcome a new Apache Airflow Meetup group – this one in <a href="https://www.meetup.com/Chicago-Apache-Airflow-incubating-Meetup/" target="_blank">Chicago</a>.</p>
-
+<p>We’re happy to welcome a new Apache Airflow Meetup group – this one in <a href="https://www.meetup.com/Chicago-Apache-Airflow-incubating-Meetup/">Chicago</a>.</p>
<h1 id="may-7-2018">May 7, 2018</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Kaxil Naik</strong> (<a href="https://github.com/kaxil" target="_blank">@Kaxil</a>)</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Kaxil Naik</strong> (<a href="https://github.com/kaxil">@Kaxil</a>)</p>
<h1 id="apr-9-2018">Apr 9, 2018</h1>
-
-<p>The <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/" target="_blank">Bay Area Apache Airflow Meetup</a> group will be hosting a <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/247127561/" target="_blank">meetup on April 11 at WePay</a>.</p>
-
+<p>The <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/">Bay Area Apache Airflow Meetup</a> group will be hosting a <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/247127561/">meetup on April 11 at WePay</a>.</p>
<p>Speakers from Slack, Google, and WePay will be presenting.</p>
-
<h1 id="feb-15-2018">Feb 15, 2018</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Ash Berlin-Taylor</strong> (<a href="https://github.com/ashb" target="_blank">@ashb</a>)</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Ash Berlin-Taylor</strong> (<a href="https://github.com/ashb">@ashb</a>)</p>
<h1 id="jan-2-2018">Jan 2, 2018</h1>
-
<p>New Apache Airflow release 1.9.0-incubating is out!</p>
-
-<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow" target="_blank">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
-
-<p>ReleaseNotes - <a href="https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt" target="_blank">https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt</a></p>
-
-<p>Special Thanks to Chris Riccomini (<a href="https://github.com/criccomini" target="_blank">@criccomini</a>) and Bolke de Bruin (<a href="https://github.com/bolkedebruin" target="_blank">@bolkedebruin</a>) for tirelessly shepherding this release!</p>
-
+<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow">https://pypi.python.org/pypi/apache-airflow</a> (Run <code>pip install apache-airflow</code>)</p>
+<p>ReleaseNotes - <a href="https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt">https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt</a></p>
+<p>Special Thanks to Chris Riccomini (<a href="https://github.com/criccomini">@criccomini</a>) and Bolke de Bruin (<a href="https://github.com/bolkedebruin">@bolkedebruin</a>) for tirelessly shepherding this release!</p>
<h1 id="nov-30-2017">Nov 30, 2017</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Joy Gao</strong> (<a href="https://github.com/joygao" target="_blank">@joygao</a>)</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Joy Gao</strong> (<a href="https://github.com/joygao">@joygao</a>)</p>
<h1 id="nov-27-2017">Nov 27, 2017</h1>
-
-<p>The Apache Airflow Meetup group in Amsterdam will be hosting its <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/events/245382408/?eventId=245382408" target="_blank">1st meetup (Heineken) on Dec 21</a>.</p>
-
+<p>The Apache Airflow Meetup group in Amsterdam will be hosting its <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/events/245382408/?eventId=245382408">1st meetup (Heineken) on Dec 21</a>.</p>
<p>Niels Zeilemaker from GoDataDriven will present Using Azure Container Instances as a cheap method to run Heineken data science workloads, and Daniel van der Ende from ING Wholesale banking Advanced Analytics will talk about Data Tests using Apache Airflow.</p>
-
<h1 id="nov-5-2017">Nov 5, 2017</h1>
-
-<p>We”re happy to welcome a new Apache Airflow Meetup group in <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/" target="_blank">Amsterdam</a></p>
-
+<p>We'’re happy to welcome a new Apache Airflow Meetup group in <a href="https://www.meetup.com/Amsterdam-Airflow-meetup/">Amsterdam</a></p>
<h1 id="nov-1-2017">Nov 1, 2017</h1>
-
-<p>Next Bay Area Airflow meet-up hosted by <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/" target="_blank">Airbnb this December (Dec 4)</a></p>
-
-<p>Slides are available <a href="https://drive.google.com/drive/folders/154jnUADKfrHXLUDvJBQOl8aiBwpaw9-w" target="_blank">here</a>.</p>
-
+<p>Next Bay Area Airflow meet-up hosted by <a href="https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/">Airbnb this December (Dec 4)</a></p>
+<p>Slides are available <a href="https://drive.google.com/drive/folders/154jnUADKfrHXLUDvJBQOl8aiBwpaw9-w">here</a>.</p>
<h1 id="oct-1-2017">Oct 1, 2017</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Fokko Driespong</strong> (<a href="https://github.com/fokko" target="_blank">@fokko</a>)</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Fokko Driespong</strong> (<a href="https://github.com/fokko">@fokko</a>)</p>
<h1 id="sept-6-2017">Sept 6, 2017</h1>
-
<p>New Apache Airflow release 1.8.2-incubating is out!</p>
-
-<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow" target="_blank">https://pypi.python.org/pypi/apache-airflow</a></p>
-
-<p>Release Notes - <a href="http://bit.ly/2gH9QFx" target="_blank">http://bit.ly/2gH9QFx</a></p>
-
-<p>Special Thanks to Maxime Beauchemin (<a href="https://github.com/mistercrunch" target="_blank">@mistercrunch</a>) for tirelessly shepherding this release!</p>
-
+<p>Pypi - <a href="https://pypi.python.org/pypi/apache-airflow">https://pypi.python.org/pypi/apache-airflow</a></p>
+<p>Release Notes - <a href="http://bit.ly/2gH9QFx">http://bit.ly/2gH9QFx</a></p>
+<p>Special Thanks to Maxime Beauchemin (<a href="https://github.com/mistercrunch">@mistercrunch</a>) for tirelessly shepherding this release!</p>
<h1 id="april-12-2017">April 12, 2017</h1>
-
-<p>We”re happy to welcome a new Apache Meetup group in <a href="https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/" target="_blank">Tokyo</a></p>
-
-<p>Their first event is being held on <a href="https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/events/238731591/" target="_blank">May 11</a></p>
-
+<p>We'’re happy to welcome a new Apache Meetup group in <a href="https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/">Tokyo</a></p>
+<p>Their first event is being held on <a href="https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/events/238731591/">May 11</a></p>
<p>Slides :</p>
-
<ul>
-<li><p><a href="https://speakerdeck.com/takus/building-data-pipelines-with-apache-airflow" target="_blank">Building data pipelines with Apache Airflow (Kaizen Platform)</a> via @takus</p></li>
-
-<li><p><a href="https://speakerdeck.com/hatappi/airflowkarakuroko2nicheng-rihuan-etawake" target="_blank">Yahoo! Japan’s path from Oozie to Apache Airflow</a> via @tmk_ueks</p></li>
-
-<li><p><a href="https://www.slideshare.net/techblogyahoo/oozieairflow-apacheairflow-oozie" target="_blank">Apache Airflow and Kuroko 2 @hatappi</a></p></li>
+<li>
+<p><a href="https://speakerdeck.com/takus/building-data-pipelines-with-apache-airflow">Building data pipelines with Apache Airflow (Kaizen Platform)</a> via @takus</p>
+</li>
+<li>
+<p><a href="https://speakerdeck.com/hatappi/airflowkarakuroko2nicheng-rihuan-etawake">Yahoo! Japan’s path from Oozie to Apache Airflow</a> via @tmk_ueks</p>
+</li>
+<li>
+<p><a href="https://www.slideshare.net/techblogyahoo/oozieairflow-apacheairflow-oozie">Apache Airflow and Kuroko 2 @hatappi</a></p>
+</li>
</ul>
-
<h1 id="march-19-2017">March 19, 2017</h1>
-
<p>The first official Apache Airflow release (1.8.0-incubating) is out!</p>
-
-<p><strong>Git Tag:</strong> <a href="https://github.com/apache/incubator-airflow/releases/tag/1.8.0%2Bapache.incubating" target="_blank">https://github.com/apache/incubator-airflow/releases/tag/1.8.0%2Bapache.incubating</a></p>
-
-<p><strong>Change log:</strong> 205 commits as shown in <a href="https://github.com/apache/incubator-airflow/blob/1.8.0%2Bapache.incubating/CHANGELOG.txt" target="_blank">Changelog (1.8.0-incubating)</a></p>
-
-<p><strong>PyPi:</strong> coming soon, check <a href="https://pypi.python.org/pypi/airflow" target="_blank">https://pypi.python.org/pypi/airflow</a></p>
-
+<p><strong>Git Tag:</strong> <a href="https://github.com/apache/incubator-airflow/releases/tag/1.8.0%2Bapache.incubating">https://github.com/apache/incubator-airflow/releases/tag/1.8.0%2Bapache.incubating</a></p>
+<p><strong>Change log:</strong> 205 commits as shown in <a href="https://github.com/apache/incubator-airflow/blob/1.8.0%2Bapache.incubating/CHANGELOG.txt">Changelog (1.8.0-incubating)</a></p>
+<p><strong>PyPi:</strong> coming soon, check <a href="https://pypi.python.org/pypi/airflow">https://pypi.python.org/pypi/airflow</a></p>
<ul>
-<li><p>Once available, you can install via :</p>
-
-<pre><code>pip install airflow --upgrade
-</code></pre></li>
+<li>
+<p>Once available, you can install via :</p>
+<pre><code> pip install airflow --upgrade
+</code></pre>
+</li>
</ul>
-
<p><strong>Note:</strong> remember to run <code>airflow upgradedb</code> after installation</p>
-
-<p>Thanks to everyone in the community that helped bring this about. A special thanks is due to Bolke de Bruin (<a href="https://github.com/bolkedebruin" target="_blank">@bolkedebruin</a>) for valiantly shepherding this release!</p>
-
+<p>Thanks to everyone in the community that helped bring this about. A special thanks is due to Bolke de Bruin (<a href="https://github.com/bolkedebruin">@bolkedebruin</a>) for valiantly shepherding this release!</p>
<h1 id="mar-14-2017">Mar 14, 2017</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Alex Guziel</strong> (<a href="https://github.com/saguziel" target="_blank">@saguziel</a>)</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Alex Guziel</strong> (<a href="https://github.com/saguziel">@saguziel</a>)</p>
<h1 id="feb-3-2017">Feb 3, 2017</h1>
-
-<p>We”re happy to welcome a new <a href="https://www.meetup.com/NYC-Apache-Airflow-incubating-Meetup/" target="_blank">Apache Meetup group in NYC</a></p>
-
+<p>We'’re happy to welcome a new <a href="https://www.meetup.com/NYC-Apache-Airflow-incubating-Meetup/">Apache Meetup group in NYC</a></p>
<h1 id="feb-2-2017">Feb 2, 2017</h1>
-
-<p>Next Bay Area Apache Airflow Meet-up (at PayPal) on Mar 14 : <a href="http://bit.ly/2knyNHv" target="_blank">http://bit.ly/2knyNHv</a></p>
-
+<p>Next Bay Area Apache Airflow Meet-up (at PayPal) on Mar 14 : <a href="http://bit.ly/2knyNHv">http://bit.ly/2knyNHv</a></p>
<h1 id="jan-30-2017">Jan 30, 2017</h1>
-
<p>Next New York Apache Airflow Meet-up (at Blue Apron) on Feb 1</p>
-
-<p><strong>Sign-up:</strong> <a href="https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf-u1uh3IleeHy8QTwaJ4BkkSkVM-X1E/edit#gid=0" target="_blank">https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf-u1uh3IleeHy8QTwaJ4BkkSkVM-X1E/edit#gid=0</a></p>
-
+<p><strong>Sign-up:</strong> <a href="https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf-u1uh3IleeHy8QTwaJ4BkkSkVM-X1E/edit#gid=0">https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf-u1uh3IleeHy8QTwaJ4BkkSkVM-X1E/edit#gid=0</a></p>
<p><strong>Time:</strong> 6:30 - 9pm EST</p>
-
<p><strong>Location:</strong> 40 W 23rd St. New York, NY 10010 (5th floor)</p>
-
<h1 id="nov-28-2016">Nov 28, 2016</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Alex Van Boxel</strong> (<a href="https://github.com/alexvanboxel" target="_blank">@alexvanboxel</a>)</p>
-
-<p>Video from the <a href="http://bit.ly/2dwKNls" target="_blank">Nov 16 meet-up</a> (at WePay) is now available at : <a href="https://wepayinc.app.box.com/s/1183ra3z8gxf8fridysu4wbjckg1s05v" target="_blank">https://wepayinc.app.box.com/s/1183ra3z8gxf8fridysu4wbjckg1s05v</a></p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Alex Van Boxel</strong> (<a href="https://github.com/alexvanboxel">@alexvanboxel</a>)</p>
+<p>Video from the <a href="http://bit.ly/2dwKNls">Nov 16 meet-up</a> (at WePay) is now available at : <a href="https://wepayinc.app.box.com/s/1183ra3z8gxf8fridysu4wbjckg1s05v">https://wepayinc.app.box.com/s/1183ra3z8gxf8fridysu4wbjckg1s05v</a></p>
<h1 id="oct-12-2016">Oct 12, 2016</h1>
-
-<p>New Apache Airflow Meet-up (at WePay) on Nov 16 : <a href="http://bit.ly/2dwKNls" target="_blank">http://bit.ly/2dwKNls</a></p>
-
+<p>New Apache Airflow Meet-up (at WePay) on Nov 16 : <a href="http://bit.ly/2dwKNls">http://bit.ly/2dwKNls</a></p>
<h1 id="sept-21-2016">Sept 21, 2016</h1>
-
-<p>New Apache Airflow Meet-up (at Stripe) on Sept 21 : <a href="http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/233316814/" target="_blank">http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/233316814/</a></p>
-
+<p>New Apache Airflow Meet-up (at Stripe) on Sept 21 : <a href="http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/233316814/">http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/233316814/</a></p>
<h1 id="sept-6-2016">Sept 6, 2016</h1>
-
-<p>New Warsaw (Poland) Hadoop User Group Meet-up on Sept 14 : <a href="http://www.meetup.com/warsaw-hug/events/233912332/" target="_blank">http://www.meetup.com/warsaw-hug/events/233912332/</a></p>
-
+<p>New Warsaw (Poland) Hadoop User Group Meet-up on Sept 14 : <a href="http://www.meetup.com/warsaw-hug/events/233912332/">http://www.meetup.com/warsaw-hug/events/233912332/</a></p>
<h1 id="august-19-2016">August 19, 2016</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Li Xuanji</strong> (<a href="https://github.com/zodiac" target="_blank">@zodiac</a>))</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Li Xuanji</strong> (<a href="https://github.com/zodiac">@zodiac</a>))</p>
<h1 id="august-16-2016">August 16, 2016</h1>
-
-<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Sumit Maheshwari</strong> (<a href="https://github.com/msumit" target="_blank">@msumit</a>)</p>
-
+<p>The Apache Airflow PPMC welcomes new committer and PPMC member, <strong>Sumit Maheshwari</strong> (<a href="https://github.com/msumit">@msumit</a>)</p>
<h1 id="june-15-2016">June 15, 2016</h1>
-
-<p>Video of yesterday’s Apache Airflow Meet-up talk featuring talks from WePay, Agari, and Yahoo is now available at <a href="http://bit.ly/1S69hOr" target="_blank">http://bit.ly/1S69hOr</a></p>
-
+<p>Video of yesterday’s Apache Airflow Meet-up talk featuring talks from WePay, Agari, and Yahoo is now available at <a href="http://bit.ly/1S69hOr">http://bit.ly/1S69hOr</a></p>
<h1 id="may-27-2016">May 27, 2016</h1>
-
<p>The first-ever Airflow Contributor Meeting will take place at 11AM PST on June 1st.</p>
-
-<p><a href="https://airbnb.webex.com/join/gurer.kiratli" target="_blank">https://airbnb.webex.com/join/gurer.kiratli</a></p>
-
+<p><a href="https://airbnb.webex.com/join/gurer.kiratli">https://airbnb.webex.com/join/gurer.kiratli</a></p>
<p><strong>Agenda</strong></p>
-
<ul>
<li>Ways to mitigate risks when rolling out new Airflow releases into production</li>
<li>Clarifying the process for working on large efforts</li>
<li>Refreshing the current roadmap</li>
<li>Communicate about how Airbnb will share information about sprints and internal roadmap moving forward</li>
</ul>
-
<h1 id="may-20-2016">May 20, 2016</h1>
-
-<p>Airflow <strong>1.7.1.2</strong> released: <a href="https://pypi.python.org/pypi/airflow/1.7.1.2" target="_blank">Pypi</a> or via <a href="https://github.com/apache/incubator-airflow/releases/tag/1.7.1.2" target="_blank">Git tag</a> โ 214 commits as per <a href="https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt" target="_blank">CHANGELOG.txt</a></p>
-
+<p>Airflow <strong>1.7.1.2</strong> released: <a href="https://pypi.python.org/pypi/airflow/1.7.1.2">Pypi</a> or via <a href="https://github.com/apache/incubator-airflow/releases/tag/1.7.1.2">Git tag</a> โ 214 commits as per <a href="https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt">CHANGELOG.txt</a></p>
<ul>
-<li><p>To install via Pypi,</p>
-
+<li>
+<p>To install via Pypi,</p>
<ul>
-<li><p>on a fresh install (e.g. fresh virtualenv install)</p>
-
-<pre><code>pip install airflow (or pip install airflow==1.7.1.2)
-</code></pre></li>
-
-<li><p>or upgrade via</p>
-
-<pre><code>pip install airflow --upgrade
-</code></pre></li>
-</ul></li>
+<li>
+<p>on a fresh install (e.g. fresh virtualenv install)</p>
+<pre><code> pip install airflow (or pip install airflow==1.7.1.2)
+</code></pre>
+</li>
+<li>
+<p>or upgrade via</p>
+<pre><code> pip install airflow --upgrade
+</code></pre>
+</li>
</ul>
-
-<p>Special thanks to Dan Davydov (<a href="https://github.com/aoen" target="_blank">@aoen</a>) for tirelessly shepherding this release!</p>
-
+</li>
+</ul>
+<p>Special thanks to Dan Davydov (<a href="https://github.com/aoen">@aoen</a>) for tirelessly shepherding this release!</p>
<h1 id="may-20-2016-1">May 20, 2016</h1>
-
-<p><strong>Steven Yvinec-Kruyk</strong> (<a href="https://github.com/syvineckruyk" target="_blank">@syvineckruyk</a>) joins the <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Committers" target="_blank">Apache Airflow Committer and PPMC group</a> today. Please give him a hearty welcome and of course.. ask him to review your PRs and answer any questions you may have :)</p>
-
+<p><strong>Steven Yvinec-Kruyk</strong> (<a href="https://github.com/syvineckruyk">@syvineckruyk</a>) joins the <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Committers">Apache Airflow Committer and PPMC group</a> today. Please give him a hearty welcome and of course.. ask him to review your PRs and answer any questions you may have :)</p>
<h1 id="april-29-2016">April 29, 2016</h1>
-
-<p>Want to keep abreast of new Apache Airflow updates (e.g. releases, meet-ups, new features, conference talks, etc… ), follow our newly-minted Twitter account : <a href="https://twitter.com/ApacheAirflow" target="_blank">@ApacheAirflow</a></p>
-
+<p>Want to keep abreast of new Apache Airflow updates (e.g. releases, meet-ups, new features, conference talks, etc… ), follow our newly-minted Twitter account : <a href="https://twitter.com/ApacheAirflow">@ApacheAirflow</a></p>
<h1 id="april-27-2016">April 27, 2016</h1>
-
<p>We are starting the migration to Apache Infrastructure (e.g. GitHub Issues –> Jira, Airbnb/Airflow GitHub to Apache/Airflow GitHub, Airbnb/Airflow GitHub Wiki to Apache Airflow Confluence Wiki)</p>
-
<ul>
-<li>The progress and migration status will be tracked on <a href="https://airflow.apache.org/announcements/" target="_blank">Migrating to Apache</a></li>
-<li>We expect this to take roughly 1 week. On and after May 4, we expect to be using Apache infrastructure exclusively. To prepare for that day, start using the new Apache infrastructure and follow instructions on this JIRA ticket <a href="https://issues.apache.org/jira/browse/AIRFLOW-11" target="_blank">AIRFLOW-11</a> to set up your accounts.</li>
+<li>The progress and migration status will be tracked on <a href="https://airflow.apache.org/announcements/">Migrating to Apache</a></li>
+<li>We expect this to take roughly 1 week. On and after May 4, we expect to be using Apache infrastructure exclusively. To prepare for that day, start using the new Apache infrastructure and follow instructions on this JIRA ticket <a href="https://issues.apache.org/jira/browse/AIRFLOW-11">AIRFLOW-11</a> to set up your accounts.</li>
</ul>
-
<h1 id="april-22-2016">April 22, 2016</h1>
-
-<p>Sid Anand (<a href="https://github.com/r39132" target="_blank">@r39132</a>) will be speaking on July 23 at <a href="http://datadayseattle.com/sessions#anand" target="_blank">Data Day Seattle</a></p>
-
+<p>Sid Anand (<a href="https://github.com/r39132">@r39132</a>) will be speaking on July 23 at <a href="http://datadayseattle.com/sessions#anand">Data Day Seattle</a></p>
<ul>
<li>If there is enough interest, I’d be happy to also speak at a meetup</li>
</ul>
-
<h1 id="april-12-2016">April 12, 2016</h1>
-
<p>A 1.7.1. lightweight tag was mistakenly pushed to master - we don’t
-yet have a viable 1.7.1 release candidate. On April 12, <a href="https://github.com/r39132" target="_blank">@r39132</a> deleted the tag on master</p>
-
+yet have a viable 1.7.1 release candidate. On April 12, <a href="https://github.com/r39132">@r39132</a> deleted the tag on master</p>
<h1 id="april-6-2016">April 6, 2016</h1>
-
<p>We now have an Apache Airflow meet-up group for folks in the Bay Area: Sign up today to get notified of upcoming meet-ups!</p>
-
<ul>
-<li><a href="http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/" target="_blank">Bay Area Apache Airflow (Incubating) Meetup</a></li>
+<li><a href="http://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/">Bay Area Apache Airflow (Incubating) Meetup</a></li>
</ul>
-
<h1 id="march-28-2016">March 28, 2016</h1>
-
-<p>Folks, A new release (1.7.0) is out via <a href="https://pypi.python.org/pypi/airflow/1.7.0" target="_blank">Pypi</a> and <a href="https://github.com/airbnb/airflow/releases/tag/1.7.0" target="_blank">git tag</a></p>
-
+<p>Folks, A new release (1.7.0) is out via <a href="https://pypi.python.org/pypi/airflow/1.7.0">Pypi</a> and <a href="https://github.com/airbnb/airflow/releases/tag/1.7.0">git tag</a></p>
<h1 id="march-24-2016">March 24, 2016</h1>
-
<p>Folks, A new release candidate version of Airflow is ready for the community to try: <strong>1.7.0rc1</strong></p>
-
<ul>
<li>Please check it out and report any issues that you see</li>
</ul>
-
<h1 id="march-19-2016">March 19, 2016</h1>
-
<p>This week, we applied for Airflow’s entry to the Apache Incubator</p>
-
<ul>
-<li><a href="https://wiki.apache.org/incubator/AirflowProposal" target="_blank">Airflow Proposal</a></li>
+<li><a href="https://wiki.apache.org/incubator/AirflowProposal">Airflow Proposal</a></li>
<li>We promoted several contributors to committers based on a proven track-record of contributions to the project and a strong commitment to improving the project going forward</li>
-<li>We published a <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62694601" target="_blank">WIP Roadmap [ARCHIVED]</a> and welcome comments and input from the community</li>
+<li>We published a <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62694601">WIP Roadmap [ARCHIVED]</a> and welcome comments and input from the community</li>
</ul>
-
<h1 id="march-1-2016">March 1, 2016</h1>
-
<p>Airbnb will be hosting the very first Airflow meetup at Airbnb HQ (888 Brannan, in SF) on March 28th.</p>
-
<p>We’re planning on doing this regularly, and want to put a good line up of food, drinks and talks for you all.</p>
-
<p>It should be pretty informal, but here’s a draft of the schedule for the night:</p>
-
<ul>
<li>meet, geet and eat</li>
<li>updates on the project, roadmap & upcoming Airbnb sprints</li>
@@ -1491,9 +1105,7 @@ yet have a viable 1.7.1 release candidate. On April 12, <a href="https://github.
<li>Community open mic, step up and make announcements if you’re recruiting, looking for help, planning on working on a feature, …</li>
<li>Define list of subjects of interest on a whiteboard for breaking into subject specific subgroups for discussion</li>
</ul>
-
-<p>Please RSVP here: <a href="https://www.airbnb.com/meetups/daywndmbd-airflow" target="_blank">https://www.airbnb.com/meetups/daywndmbd-airflow</a></p>
-
+<p>Please RSVP here: <a href="https://www.airbnb.com/meetups/daywndmbd-airflow">https://www.airbnb.com/meetups/daywndmbd-airflow</a></p>
<p>We’re planning on starting to have Airflow meetups regularly, so stay tuned!</p>
@@ -1664,7 +1276,7 @@ yet have a viable 1.7.1 release candidate. On April 12, <a href="https://github.
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-1.10.10/index.html b/blog/airflow-1.10.10/index.html
index f9898bc865..5b27065dd0 100644
--- a/blog/airflow-1.10.10/index.html
+++ b/blog/airflow-1.10.10/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Apache Airflow 1.10.10 | Apache Airflow</title><meta property="og:title" content="Apache Airflow 1.10.10" />
<meta property="og:description" content="We are happy to present Apache Airflow 1.10.10" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-1.10.10/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-1.10.10/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2020-04-09T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Apache Airflow 1.10.10">
-<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.10">
-
-
-<meta itemprop="datePublished" content="2020-04-09T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="1143">
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-<meta itemprop="keywords" content="release," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Apache Airflow 1.10.10">
+<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.10"><meta itemprop="datePublished" content="2020-04-09T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="1143"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="release," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Apache Airflow 1.10.10"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -495,51 +487,35 @@
<p class="blogpost-content__metadata--description">We are happy to present Apache Airflow 1.10.10</p>
</div>
<div class="markdown-content">
-
-
-<p>Airflow 1.10.10 contains 199 commits since 1.10.9 and includes 11 new features, 43 improvements, 44 bug fixes, and several doc changes.</p>
-
+ <p>Airflow 1.10.10 contains 199 commits since 1.10.9 and includes 11 new features, 43 improvements, 44 bug fixes, and several doc changes.</p>
<p><strong>Details</strong>:</p>
-
<ul>
-<li><strong>PyPI</strong>: <a href="https://pypi.org/project/apache-airflow/1.10.10/" target="_blank">https://pypi.org/project/apache-airflow/1.10.10/</a></li>
-<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.10/" target="_blank">https://airflow.apache.org/docs/1.10.10/</a></li>
-<li><strong>Changelog</strong>: <a href="http://airflow.apache.org/docs/1.10.10/changelog.html" target="_blank">http://airflow.apache.org/docs/1.10.10/changelog.html</a></li>
+<li><strong>PyPI</strong>: <a href="https://pypi.org/project/apache-airflow/1.10.10/">https://pypi.org/project/apache-airflow/1.10.10/</a></li>
+<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.10/">https://airflow.apache.org/docs/1.10.10/</a></li>
+<li><strong>Changelog</strong>: <a href="http://airflow.apache.org/docs/1.10.10/changelog.html">http://airflow.apache.org/docs/1.10.10/changelog.html</a></li>
</ul>
-
<p>Some of the noteworthy new features (user-facing) are:</p>
-
<ul>
-<li><a href="https://github.com/apache/airflow/pull/8046" target="_blank">Allow user to chose timezone to use in the RBAC UI</a></li>
-<li><a href="https://github.com/apache/airflow/pull/7832" target="_blank">Add Production Docker image support</a></li>
-<li><a href="http://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html" target="_blank">Allow Retrieving Airflow Connections & Variables from various Secrets backend</a></li>
-<li><a href="http://airflow.apache.org/docs/1.10.10/dag-serialization.html" target="_blank">Stateless Webserver using DAG Serialization</a></li>
-<li><a href="https://github.com/apache/airflow/pull/7880" target="_blank">Tasks with Dummy Operators are no longer sent to executor</a></li>
-<li><a href="https://github.com/apache/airflow/pull/7312" target="_blank">Allow passing DagRun conf when triggering dags via UI</a></li>
+<li><a href="https://github.com/apache/airflow/pull/8046">Allow user to chose timezone to use in the RBAC UI</a></li>
+<li><a href="https://github.com/apache/airflow/pull/7832">Add Production Docker image support</a></li>
+<li><a href="http://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html">Allow Retrieving Airflow Connections & Variables from various Secrets backend</a></li>
+<li><a href="http://airflow.apache.org/docs/1.10.10/dag-serialization.html">Stateless Webserver using DAG Serialization</a></li>
+<li><a href="https://github.com/apache/airflow/pull/7880">Tasks with Dummy Operators are no longer sent to executor</a></li>
+<li><a href="https://github.com/apache/airflow/pull/7312">Allow passing DagRun conf when triggering dags via UI</a></li>
</ul>
-
<h3 id="allow-user-to-chose-timezone-to-use-in-the-rbac-ui">Allow user to chose timezone to use in the RBAC UI</h3>
-
<p>By default the Web UI will show times in UTC. It is possible to change the timezone shown by using the menu in the top
- right (click on the clock to activate it):</p>
-
+right (click on the clock to activate it):</p>
<p><strong>Screenshot</strong>:
-<img src="rbac-ui-timezone.gif" alt="Allow user to chose timezone to use in the RBAC UI" /></p>
-
-<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/timezone.html#web-ui" target="_blank">https://airflow.apache.org/docs/1.10.10/timezone.html#web-ui</a></p>
-
+<img src="rbac-ui-timezone.gif" alt="Allow user to chose timezone to use in the RBAC UI"></p>
+<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/timezone.html#web-ui">https://airflow.apache.org/docs/1.10.10/timezone.html#web-ui</a></p>
<p><strong>Note</strong>: This feature is only available for the RBAC UI (enabled using <code>rbac=True</code> in <code>[webserver]</code> section in your <code>airflow.cfg</code>).</p>
-
<h3 id="add-production-docker-image-support">Add Production Docker image support</h3>
-
<p>There are brand new production images (alpha quality) available for Airflow 1.10.10. You can pull them from the
-<a href="https://hub.docker.com/r/apache/airflow" target="_blank">Apache Airflow Dockerhub</a> repository and start using it.</p>
-
-<p>More information about using production images can be found in <a href="https://github.com/apache/airflow/blob/master/IMAGES.rst#using-the-images" target="_blank">https://github.com/apache/airflow/blob/master/IMAGES.rst#using-the-images</a>. Soon it will be updated with
+<a href="https://hub.docker.com/r/apache/airflow">Apache Airflow Dockerhub</a> repository and start using it.</p>
+<p>More information about using production images can be found in <a href="https://github.com/apache/airflow/blob/master/IMAGES.rst#using-the-images">https://github.com/apache/airflow/blob/master/IMAGES.rst#using-the-images</a>. Soon it will be updated with
information how to use images using official helm chart.</p>
-
<p>To pull the images you can run one of the following commands:</p>
-
<ul>
<li><code>docker pull apache/airflow:1.10.10-python2.7</code></li>
<li><code>docker pull apache/airflow:1.10.10-python3.5</code></li>
@@ -547,145 +523,100 @@ information how to use images using official helm chart.</p>
<li><code>docker pull apache/airflow:1.10.10-python3.7</code></li>
<li><code>docker pull apache/airflow:1.10.10</code> (uses Python 3.6)</li>
</ul>
-
-<h3 id="allow-retrieving-airflow-connections-variables-from-various-secrets-backend">Allow Retrieving Airflow Connections & Variables from various Secrets backend</h3>
-
+<h3 id="allow-retrieving-airflow-connections--variables-from-various-secrets-backend">Allow Retrieving Airflow Connections & Variables from various Secrets backend</h3>
<p>From Airflow 1.10.10, users would be able to get Airflow Variables from Environment Variables.</p>
-
-<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/concepts.html#storing-variables-in-environment-variables" target="_blank">https://airflow.apache.org/docs/1.10.10/concepts.html#storing-variables-in-environment-variables</a></p>
-
+<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/concepts.html#storing-variables-in-environment-variables">https://airflow.apache.org/docs/1.10.10/concepts.html#storing-variables-in-environment-variables</a></p>
<p>A new concept of Secrets Backend has been introduced to retrieve Airflow Connections and Variables.</p>
-
<p>From Airflow 1.10.10, users can retrieve Connections & Variables using the same syntax (no DAG code change is required),
from a secret backend defined in <code>airflow.cfg</code>. If no backend is defined, Airflow falls-back to Environment Variables
and then Metadata DB.</p>
-
-<p>Check <a href="https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html#configuration" target="_blank">https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html#configuration</a> for details on how-to
+<p>Check <a href="https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html#configuration">https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html#configuration</a> for details on how-to
configure Secrets backend.</p>
-
<p>As of 1.10.10, Airflow supports the following Secret Backends:</p>
-
<ul>
<li>Hashicorp Vault</li>
<li>GCP Secrets Manager</li>
<li>AWS Parameters Store</li>
</ul>
-
-<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html" target="_blank">https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html</a></p>
-
+<p>Details: <a href="https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html">https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html</a></p>
<p>Example configuration to use Hashicorp Vault as the backend:</p>
-
-<pre><code class="language-ini">[secrets]
-backend = airflow.contrib.secrets.hashicorp_vault.VaultBackend
-backend_kwargs = {"url": "http://127.0.0.1:8200", "connections_path": "connections", "variables_path": "variables", "mount_point": "airflow"}
-</code></pre>
-
-<h3 id="stateless-webserver-using-dag-serialization">Stateless Webserver using DAG Serialization</h3>
-
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="k">[secrets]</span>
+<span class="na">backend</span> <span class="o">=</span> <span class="s">airflow.contrib.secrets.hashicorp_vault.VaultBackend</span>
+<span class="na">backend_kwargs</span> <span class="o">=</span> <span class="s">{"url": "http://127.0.0.1:8200", "connections_path": "connections", "variables_path": "variables", "mount_point": "airflow"}</span>
+</code></pre></div><h3 id="stateless-webserver-using-dag-serialization">Stateless Webserver using DAG Serialization</h3>
<p>The Webserver can now run without access to DAG Files when DAG Serialization is turned on.
The 2 limitations we had in 1.10.7-1.10.9 (
-<a href="https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations)" target="_blank">https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations)</a>
+<a href="https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations">https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations</a>)
have been resolved.</p>
-
<p>The main advantage of this would be reduction in Webserver startup time for large number of DAGs.
Without DAG Serialization all the DAGs are loaded in the DagBag during the
Webserver startup.</p>
-
<p>With DAG Serialization, an empty DagBag is created and
Dags are loaded from DB only when needed (i.e. when a particular DAG is
clicked on in the home page)</p>
-
-<p>Details: <a href="http://airflow.apache.org/docs/1.10.10/dag-serialization.html" target="_blank">http://airflow.apache.org/docs/1.10.10/dag-serialization.html</a></p>
-
+<p>Details: <a href="http://airflow.apache.org/docs/1.10.10/dag-serialization.html">http://airflow.apache.org/docs/1.10.10/dag-serialization.html</a></p>
<h3 id="tasks-using-dummy-operators-are-no-longer-sent-to-executor">Tasks using Dummy Operators are no longer sent to executor</h3>
-
<p>The Dummy operators does not actually do any work and are mostly used for organizing/grouping tasks along
with BranchPythonOperator.</p>
-
<p>Previously, when using Kubernetes Executor, the executor would spin up a whole worker pod to execute a dummy task.
With Airflow 1.10.10 tasks using Dummy Operators would be scheduled & evaluated by the Scheduler but not sent to the
Executor. This should significantly improve execution time and resource usage.</p>
-
<h3 id="allow-passing-dagrun-conf-when-triggering-dags-via-ui">Allow passing DagRun conf when triggering dags via UI</h3>
-
<p>When triggering a DAG from the CLI or the REST API, it s possible to pass configuration for the DAG run as a JSON blob.</p>
-
<p>From Airflow 1.10.10, when a user clicks on Trigger Dag button, a new screen confirming the trigger request, and allowing the user to pass a JSON configuration
blob would be show.</p>
-
<p><strong>Screenshot</strong>:
-<img src="trigger-dag-conf.png" alt="Allow passing DagRun conf when triggering dags via UI" /></p>
-
-<p>Details: <a href="https://github.com/apache/airflow/pull/7312" target="_blank">https://github.com/apache/airflow/pull/7312</a></p>
-
+<img src="trigger-dag-conf.png" alt="Allow passing DagRun conf when triggering dags via UI"></p>
+<p>Details: <a href="https://github.com/apache/airflow/pull/7312">https://github.com/apache/airflow/pull/7312</a></p>
<h2 id="updating-guide">Updating Guide</h2>
-
<p>If you are updating Apache Airflow from a previous version to <code>1.10.10</code>, please take a note of the following:</p>
-
<ul>
-<li><p>Run <code>airflow upgradedb</code> after <code>pip install -U apache-airflow==1.10.10</code> as <code>1.10.10</code> contains 3 database migrations.</p></li>
-
-<li><p>If you have used <code>none_failed</code> trigger rule in your DAG, change it to use the new <code>none_failed_or_skipped</code> trigger rule.
+<li>
+<p>Run <code>airflow upgradedb</code> after <code>pip install -U apache-airflow==1.10.10</code> as <code>1.10.10</code> contains 3 database migrations.</p>
+</li>
+<li>
+<p>If you have used <code>none_failed</code> trigger rule in your DAG, change it to use the new <code>none_failed_or_skipped</code> trigger rule.
As previously implemented, the actual behavior of <code>none_failed</code> trigger rule would skip the current task if all parents of the task
had also skipped. This was not in-line with what was documented about that trigger rule. We have changed the implementation to match
the documentation, hence if you need the old behavior use <code>none_failed_or_skipped</code>.</p>
-
-<p>More details in <a href="https://github.com/apache/airflow/pull/7464" target="_blank">https://github.com/apache/airflow/pull/7464</a>.</p></li>
-
-<li><p>Setting empty string to a Airflow Variable will now return an empty string, it previously returned <code>None</code>.</p>
-
+<p>More details in <a href="https://github.com/apache/airflow/pull/7464">https://github.com/apache/airflow/pull/7464</a>.</p>
+</li>
+<li>
+<p>Setting empty string to a Airflow Variable will now return an empty string, it previously returned <code>None</code>.</p>
<p>Example:</p>
-
<pre><code>>> Variable.set('test_key', '')
>> Variable.get('test_key')
</code></pre>
-
-<p>The above code returned <code>None</code> previously, now it will return “.</p></li>
-
-<li><p>When a task is marked as <code>success</code> by a user in Airflow UI, function defined in <code>on_success_callback</code> will be called.</p></li>
+<p>The above code returned <code>None</code> previously, now it will return ‘’.</p>
+</li>
+<li>
+<p>When a task is marked as <code>success</code> by a user in Airflow UI, function defined in <code>on_success_callback</code> will be called.</p>
+</li>
</ul>
-
-<h2 id="special-note-deprecations">Special Note / Deprecations</h2>
-
+<h2 id="special-note--deprecations">Special Note / Deprecations</h2>
<h3 id="python-2">Python 2</h3>
-
<p>Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2.
Airflow 1.10.* would be the last series to support Python 2.</p>
-
<p>We strongly recommend users to use Python >= 3.6</p>
-
<h3 id="use-airflow-rbac-ui">Use Airflow RBAC UI</h3>
-
<p>Airflow 1.10.10 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI.</p>
-
<p>The Flask-AppBuilder (FAB) based UI allows Role-based Access Control and has more advanced features compared to
the legacy Flask-admin based UI. This UI can be enabled by setting <code>rbac=True</code> in <code>[webserver]</code> section in your <code>airflow.cfg</code>.</p>
-
<p>Flask-admin based UI is deprecated and new features won’t be ported to it. This UI will still be the default
for 1.10.* series but would no longer be available from Airflow 2.0</p>
-
<h3 id="running-airflow-on-macos">Running Airflow on MacOS</h3>
-
<p>Run <code>export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES</code> in your scheduler environmentIf you are running Airflow on MacOS
and get the following error in the Scheduler logs:</p>
-
<pre><code>objc[1873]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[1873]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
</code></pre>
-
<p>This error occurs because of added security to restrict multiprocessing & multithreading in Mac OS High Sierra and above.</p>
-
<h3 id="we-have-moved-to-github-issues">We have moved to GitHub Issues</h3>
-
-<p>The Airflow Project has moved from <a href="https://issues.apache.org/jira/projects/AIRFLOW/issues" target="_blank">JIRA</a> to
-<a href="https://github.com/apache/airflow/issues" target="_blank">GitHub</a> for tracking issues.</p>
-
+<p>The Airflow Project has moved from <a href="https://issues.apache.org/jira/projects/AIRFLOW/issues">JIRA</a> to
+<a href="https://github.com/apache/airflow/issues">GitHub</a> for tracking issues.</p>
<p>So if you find any bugs in Airflow 1.10.10 please create a GitHub Issue for it.</p>
-
<h2 id="list-of-contributors">List of Contributors</h2>
-
<p>According to git shortlog, the following people contributed to the 1.10.10 release. Thank you to all contributors!</p>
-
<p>ANiteckiP, Alex Guziel, Alex Lue, Anita Fronczak, Ash Berlin-Taylor, Benji Visser, Bhavika Tekwani, Brad Dettmer, Chris McLennon, Cooper Gillan, Daniel Imberman, Daniel Standish, Felix Uellendall, Jarek Potiuk, Jiajie Zhong, Jithin Sukumar, Kamil Breguลa, Kaxil Naik, Kengo Seki, Kris, Kumpan Anton, Lokesh Lal, Louis Guitton, Louis Simoneau, Luyao Yang, Noรซl Bardelot, Omair Khan, Philipp Groรelfinger, Ping Zhang, RasPavel, Ray, Robin Edwards, Ry Walker, Saurabh, Sebastian Brandt, Tomek [...]
</div>
@@ -1014,7 +945,7 @@ objc[1873]: +[__NSPlaceholderDate initialize] may have been in progress in anoth
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-1.10.12/index.html b/blog/airflow-1.10.12/index.html
index 68284ab959..c58ffae651 100644
--- a/blog/airflow-1.10.12/index.html
+++ b/blog/airflow-1.10.12/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Apache Airflow 1.10.12 | Apache Airflow</title><meta property="og:title" content="Apache Airflow 1.10.12" />
<meta property="og:description" content="We are happy to present Apache Airflow 1.10.12" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-1.10.12/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-1.10.12/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2020-08-25T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Apache Airflow 1.10.12">
-<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.12">
-
-
-<meta itemprop="datePublished" content="2020-08-25T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="911">
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-
-<meta itemprop="keywords" content="release," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Apache Airflow 1.10.12">
+<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.12"><meta itemprop="datePublished" content="2020-08-25T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="911"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="release," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Apache Airflow 1.10.12"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -495,43 +487,30 @@
<p class="blogpost-content__metadata--description">We are happy to present Apache Airflow 1.10.12</p>
</div>
<div class="markdown-content">
-
-
-<p>Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new features, 23 improvements, 23 bug fixes,
+ <p>Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new features, 23 improvements, 23 bug fixes,
and several doc changes.</p>
-
<p><strong>Details</strong>:</p>
-
<ul>
-<li><strong>PyPI</strong>: <a href="https://pypi.org/project/apache-airflow/1.10.12/" target="_blank">https://pypi.org/project/apache-airflow/1.10.12/</a></li>
-<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.12/" target="_blank">https://airflow.apache.org/docs/1.10.12/</a></li>
-<li><strong>Changelog</strong>: <a href="http://airflow.apache.org/docs/1.10.12/changelog.html" target="_blank">http://airflow.apache.org/docs/1.10.12/changelog.html</a></li>
+<li><strong>PyPI</strong>: <a href="https://pypi.org/project/apache-airflow/1.10.12/">https://pypi.org/project/apache-airflow/1.10.12/</a></li>
+<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.12/">https://airflow.apache.org/docs/1.10.12/</a></li>
+<li><strong>Changelog</strong>: <a href="http://airflow.apache.org/docs/1.10.12/changelog.html">http://airflow.apache.org/docs/1.10.12/changelog.html</a></li>
</ul>
-
<p><strong>Airflow 1.10.11 has breaking changes with respect to
KubernetesExecutor & KubernetesPodOperator so I recommend users to directly upgrade to Airflow 1.10.12 instead</strong>.</p>
-
<p>Some of the noteworthy new features (user-facing) are:</p>
-
<ul>
-<li><a href="https://github.com/apache/airflow/pull/8560" target="_blank">Allow defining custom XCom class</a></li>
-<li><a href="https://github.com/apache/airflow/pull/9645" target="_blank">Get Airflow configs with sensitive data from Secret Backends</a></li>
-<li><a href="https://github.com/apache/airflow/pull/10282" target="_blank">Add AirflowClusterPolicyViolation support to Airflow local settings</a></li>
+<li><a href="https://github.com/apache/airflow/pull/8560">Allow defining custom XCom class</a></li>
+<li><a href="https://github.com/apache/airflow/pull/9645">Get Airflow configs with sensitive data from Secret Backends</a></li>
+<li><a href="https://github.com/apache/airflow/pull/10282">Add AirflowClusterPolicyViolation support to Airflow local settings</a></li>
</ul>
-
<h3 id="allow-defining-custom-xcom-class">Allow defining Custom XCom class</h3>
-
<p>Until Airflow 1.10.11, the XCom data was only stored in Airflow Metadatabase. From Airflow 1.10.12, users
would be able to define custom XCom classes. This will allow users to transfer larger data between tasks.
An example here would be to store XCom in S3 or GCS Bucket if the size of data that needs to be stored is larger
than <code>XCom.MAX_XCOM_SIZE</code> (48 KB).</p>
-
-<p><strong>PR</strong>: <a href="https://github.com/apache/airflow/pull/8560" target="_blank">https://github.com/apache/airflow/pull/8560</a></p>
-
+<p><strong>PR</strong>: <a href="https://github.com/apache/airflow/pull/8560">https://github.com/apache/airflow/pull/8560</a></p>
<h3 id="get-airflow-configs-with-sensitive-data-from-secret-backends">Get Airflow configs with sensitive data from Secret Backends</h3>
-
<p>Users would be able to get the following Airflow configs from Secrets Backend like Hashicorp Vault:</p>
-
<ul>
<li><code>sql_alchemy_conn</code> in [core] section</li>
<li><code>fernet_key</code> in [core] section</li>
@@ -543,110 +522,75 @@ than <code>XCom.MAX_XCOM_SIZE</code> (48 KB).</p>
<li><code>bind_password</code> in [ldap] section</li>
<li><code>git_password</code> in [kubernetes] section</li>
</ul>
-
<p>Further improving Airflow’s Secret Management story, from Airflow 1.10.12, users don’t need to hardcode
the <strong>sensitive</strong> config value in airflow.cfg nor then need to use an Environment variable to set this config.</p>
-
<p>For example, the metadata database connection string can either be set in airflow.cfg like this:</p>
-
-<pre><code class="language-ini">[core]
-sql_alchemy_conn_secret = sql_alchemy_conn
-</code></pre>
-
-<p>This will retrieve config option from the set Secret Backends.</p>
-
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="k">[core]</span>
+<span class="na">sql_alchemy_conn_secret</span> <span class="o">=</span> <span class="s">sql_alchemy_conn</span>
+</code></pre></div><p>This will retrieve config option from the set Secret Backends.</p>
<p>As you can see you just need to add a <code>_secret</code> suffix at the end of the actual config option
and the value needs to be the <strong>key</strong> which the Secrets backend will look for.</p>
-
<p>Similarly, <code>_secret</code> config options can also be set using a corresponding environment variable. For example:</p>
-
-<pre><code>export AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
-</code></pre>
-
-<p>More details: <a href="http://airflow.apache.org/docs/1.10.12/howto/set-config.html" target="_blank">http://airflow.apache.org/docs/1.10.12/howto/set-config.html</a></p>
-
-<h3 id="add-airflowclusterpolicyviolation-support-to-airflow-local-settings-py">Add AirflowClusterPolicyViolation support to airflow_local_settings.py</h3>
-
+<pre tabindex="0"><code>export AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
+</code></pre><p>More details: <a href="http://airflow.apache.org/docs/1.10.12/howto/set-config.html">http://airflow.apache.org/docs/1.10.12/howto/set-config.html</a></p>
+<h3 id="add-airflowclusterpolicyviolation-support-to-airflow_local_settingspy">Add AirflowClusterPolicyViolation support to airflow_local_settings.py</h3>
<p>Users can use Cluster Policies to apply cluster-wide checks on Airflow
-tasks. You can raise <a href="http://airflow.apache.org/docs/1.10.12/_api/airflow/exceptions/index.html#airflow.exceptions.AirflowClusterPolicyViolation" target="_blank">AirflowClusterPolicyViolation</a>
+tasks. You can raise <a href="http://airflow.apache.org/docs/1.10.12/_api/airflow/exceptions/index.html#airflow.exceptions.AirflowClusterPolicyViolation">AirflowClusterPolicyViolation</a>
in a policy or task mutation hook to prevent a DAG from being
imported or prevent a task from being executed if the task is not compliant with
your check.</p>
-
<p>These checks are intended to help teams using Airflow to protect against common
beginner errors that may get past a code reviewer, rather than as technical
security controls.</p>
-
<p>For example, don’t run tasks without <code>airflow</code> owners:</p>
-
-<pre><code class="language-python">def task_must_have_owners(task):
- if not task.owner or task.owner.lower() == conf.get('operators', 'default_owner'):
- raise AirflowClusterPolicyViolation(
- 'Task must have non-None non-default owner. Current value: {}'.format(task.owner))
-</code></pre>
-
-<p>More details: <a href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks" target="_blank">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
-
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">task_must_have_owners</span><span class="p">(</span><span class="n">task</span><span class="p">):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">task</span><span class="o">.</span><span class="n">owner</span> <span class="ow">or</span> <span class="n">task</span><span class="o">.</span><span class="n">owner</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="n">conf</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'operators'</span> [...]
+ <span class="k">raise</span> <span class="n">AirflowClusterPolicyViolation</span><span class="p">(</span>
+ <span class="s1">'Task must have non-None non-default owner. Current value: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">task</span><span class="o">.</span><span class="n">owner</span><span class="p">))</span>
+</code></pre></div><p>More details: <a href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
<h3 id="launch-pods-via-yaml-files-when-using-kubernetesexecutor-and-kubernetespodoperator">Launch Pods via YAML files when using KubernetesExecutor and KubernetesPodOperator</h3>
-
<p>As of 1.10.12, users can launch pods via YAML files instead of passing various configurations.</p>
-
<p>To allow greater flexibility we have deprecated Airflow’s Pod class and instead now use classes and
objects from the official Kubernetes API. The POD class will still work but raise a deprecation
warning. This feature involved a pretty extensive rewrite of all of our pod creation code.</p>
-
<p>Initially, we were going to hold off on these features until Airflow 2.0. However, we soon
realized that exposing these features in 1.10.x is crucial in preparing users for the 2.0 release to come.</p>
-
-<p>Details: <a href="https://github.com/apache/airflow/pull/6230" target="_blank">https://github.com/apache/airflow/pull/6230</a> (<a href="https://github.com/apache/airflow/commit/7aa0f472b57985a952a3e3d0a38f1b2535d93413" target="_blank">Backport commit</a>)</p>
-
+<p>Details: <a href="https://github.com/apache/airflow/pull/6230">https://github.com/apache/airflow/pull/6230</a> (<a href="https://github.com/apache/airflow/commit/7aa0f472b57985a952a3e3d0a38f1b2535d93413">Backport commit</a>)</p>
<h2 id="updating-guide">Updating Guide</h2>
-
<p>If you are updating Apache Airflow from a previous version to <code>1.10.12</code>, please take a note of the following:</p>
-
<ul>
-<li><p>Run <code>airflow upgradedb</code> after <code>pip install -U apache-airflow==1.10.12</code> as <code>1.10.12</code> contains 1 database migration.</p></li>
-
-<li><p>As of airflow 1.10.12, using the <code>airflow.contrib.kubernetes.Pod</code> class in the <code>pod_mutation_hook</code> is now
+<li>
+<p>Run <code>airflow upgradedb</code> after <code>pip install -U apache-airflow==1.10.12</code> as <code>1.10.12</code> contains 1 database migration.</p>
+</li>
+<li>
+<p>As of airflow 1.10.12, using the <code>airflow.contrib.kubernetes.Pod</code> class in the <code>pod_mutation_hook</code> is now
deprecated. Instead we recommend that users treat the pod parameter as a <code>kubernetes.client.models.V1Pod</code> object.
-This means that users now have access to the full Kubernetes API when modifying airflow pods for mutating POD.</p></li>
-
-<li><p>Previously, when tasks skipped by SkipMixin (such as <code>BranchPythonOperator</code>, <code>BaseBranchOperator</code> and
+This means that users now have access to the full Kubernetes API when modifying airflow pods for mutating POD.</p>
+</li>
+<li>
+<p>Previously, when tasks skipped by SkipMixin (such as <code>BranchPythonOperator</code>, <code>BaseBranchOperator</code> and
<code>ShortCircuitOperator</code>) are cleared, they execute. Since 1.10.12, when such skipped tasks are cleared,
-they will be skipped again by the newly introduced <code>NotPreviouslySkippedDep</code>.</p></li>
+they will be skipped again by the newly introduced <code>NotPreviouslySkippedDep</code>.</p>
+</li>
</ul>
-
<h2 id="special-note">Special Note</h2>
-
<h3 id="python-2">Python 2</h3>
-
<p>Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2.
Airflow 1.10.* would be the last series to support Python 2.</p>
-
<p>We strongly recommend users to use Python >= 3.6</p>
-
<h3 id="use-airflow-rbac-ui">Use Airflow RBAC UI</h3>
-
<p>Airflow 1.10.12 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI.</p>
-
<p>The Flask-AppBuilder (FAB) based UI allows Role-based Access Control and has more advanced features compared to
the legacy Flask-admin based UI. This UI can be enabled by setting <code>rbac=True</code> in <code>[webserver]</code> section in
your <code>airflow.cfg</code>.</p>
-
<p>Flask-admin based UI is deprecated and new features won’t be ported to it. This UI will still be the default
for 1.10.* series but would no longer be available from Airflow 2.0</p>
-
<h3 id="we-have-moved-to-github-issues">We have moved to GitHub Issues</h3>
-
-<p>The Airflow Project has moved from <a href="https://issues.apache.org/jira/projects/AIRFLOW/issues" target="_blank">JIRA</a> to
-<a href="https://github.com/apache/airflow/issues" target="_blank">GitHub</a> for tracking issues.</p>
-
+<p>The Airflow Project has moved from <a href="https://issues.apache.org/jira/projects/AIRFLOW/issues">JIRA</a> to
+<a href="https://github.com/apache/airflow/issues">GitHub</a> for tracking issues.</p>
<p>So if you find any bugs in Airflow 1.10.12 please create a GitHub Issue for it.</p>
-
<h2 id="list-of-contributors">List of Contributors</h2>
-
<p>According to git shortlog, the following people contributed to the 1.10.12 release. Thank you to all contributors!</p>
-
<p>Alexander Sutcliffe, Andy, Aneesh Joseph, Ash Berlin-Taylor, Aviral Agrawal, BaoshanGu, Beni Ben zikry,
Daniel Imberman, Daniel Standish, Danylo Baibak, Ephraim Anierobi, Felix Uellendall, Greg Neiheisel,
Hartorn, Jacob Ferriero, Jannik F, Jarek Potiuk, Jinhui Zhang, Kamil Breguลa, Kaxil Naik, Kurganov,
@@ -979,7 +923,7 @@ William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314, retornam,
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-1.10.8-1.10.9/index.html b/blog/airflow-1.10.8-1.10.9/index.html
index c070b7258a..3960191c04 100644
--- a/blog/airflow-1.10.8-1.10.9/index.html
+++ b/blog/airflow-1.10.8-1.10.9/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Apache Airflow 1.10.8 & 1.10.9 | Apache Airflow</title><meta property="og:title" content="Apache Airflow 1.10.8 & 1.10.9" />
<meta property="og:description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-1.10.8-1.10.9/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-1.10.8-1.10.9/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2020-02-23T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Apache Airflow 1.10.8 & 1.10.9">
-<meta itemprop="description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow.">
-
-
-<meta itemprop="datePublished" content="2020-02-23T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="437">
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-
-<meta itemprop="keywords" content="release," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Apache Airflow 1.10.8 & 1.10.9">
+<meta itemprop="description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow."><meta itemprop="datePublished" content="2020-02-23T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="437"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="release," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Apache Airflow 1.10.8 & 1.10.9"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -495,74 +487,48 @@
<p class="blogpost-content__metadata--description">We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow.</p>
</div>
<div class="markdown-content">
-
-
-<p>Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes.</p>
-
+ <p>Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes.</p>
<p>We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8.</p>
-
<p><strong>Details</strong>:</p>
-
<ul>
-<li><strong>PyPI</strong>: <a href="https://pypi.org/project/apache-airflow/1.10.9/" target="_blank">https://pypi.org/project/apache-airflow/1.10.9/</a></li>
-<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.9/" target="_blank">https://airflow.apache.org/docs/1.10.9/</a></li>
-<li><strong>Changelog (1.10.8)</strong>: <a href="http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07" target="_blank">http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07</a></li>
-<li><strong>Changelog (1.10.9)</strong>: <a href="http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10" target="_blank">http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10</a></li>
+<li><strong>PyPI</strong>: <a href="https://pypi.org/project/apache-airflow/1.10.9/">https://pypi.org/project/apache-airflow/1.10.9/</a></li>
+<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.9/">https://airflow.apache.org/docs/1.10.9/</a></li>
+<li><strong>Changelog (1.10.8)</strong>: <a href="http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07">http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07</a></li>
+<li><strong>Changelog (1.10.9)</strong>: <a href="http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10">http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10</a></li>
</ul>
-
<p>Some of the noteworthy new features (user-facing) are:</p>
-
<ul>
-<li><a href="https://github.com/apache/airflow/pull/6489" target="_blank">Add tags to DAGs and use it for filtering in the UI (RBAC only)</a></li>
-<li><a href="http://airflow.apache.org/docs/1.10.9/executor/debug.html" target="_blank">New Executor: DebugExecutor for Local debugging from your IDE</a></li>
-<li><a href="https://github.com/apache/airflow/pull/7281" target="_blank">Allow passing conf in “Add DAG Run” (Triggered Dags) view</a></li>
-<li><a href="https://github.com/apache/airflow/pull/7038" target="_blank">Allow dags to run for future execution dates for manually triggered DAGs (only if <code>schedule_interval=None</code>)</a></li>
-<li><a href="https://airflow.apache.org/docs/1.10.9/configurations-ref.html" target="_blank">Dedicated page in documentation for all configs in airflow.cfg</a></li>
+<li><a href="https://github.com/apache/airflow/pull/6489">Add tags to DAGs and use it for filtering in the UI (RBAC only)</a></li>
+<li><a href="http://airflow.apache.org/docs/1.10.9/executor/debug.html">New Executor: DebugExecutor for Local debugging from your IDE</a></li>
+<li><a href="https://github.com/apache/airflow/pull/7281">Allow passing conf in “Add DAG Run” (Triggered Dags) view</a></li>
+<li><a href="https://github.com/apache/airflow/pull/7038">Allow dags to run for future execution dates for manually triggered DAGs (only if <code>schedule_interval=None</code>)</a></li>
+<li><a href="https://airflow.apache.org/docs/1.10.9/configurations-ref.html">Dedicated page in documentation for all configs in airflow.cfg</a></li>
</ul>
-
<h3 id="add-tags-to-dags-and-use-it-for-filtering-in-the-ui">Add tags to DAGs and use it for filtering in the UI</h3>
-
<p>In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button.</p>
-
<p>For example:</p>
-
<p>In your Dag file, pass a list of tags you want to add to DAG object:</p>
-
-<pre><code class="language-python">dag = DAG(
- dag_id='example_dag_tag',
- schedule_interval='0 0 * * *',
- tags=['example']
-)
-</code></pre>
-
-<p><strong>Screenshot</strong>:
-<img src="airflow-dag-tags.png" alt="Add filter by DAG tags" /></p>
-
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
+ <span class="n">dag_id</span><span class="o">=</span><span class="s1">'example_dag_tag'</span><span class="p">,</span>
+ <span class="n">schedule_interval</span><span class="o">=</span><span class="s1">'0 0 * * *'</span><span class="p">,</span>
+ <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s1">'example'</span><span class="p">]</span>
+<span class="p">)</span>
+</code></pre></div><p><strong>Screenshot</strong>:
+<img src="airflow-dag-tags.png" alt="Add filter by DAG tags"></p>
<p><strong>Note</strong>: This feature is only available for the RBAC UI (enabled using <code>rbac=True</code> in <code>[webserver]</code> section in your <code>airflow.cfg</code>).</p>
-
-<h2 id="special-note-deprecations">Special Note / Deprecations</h2>
-
+<h2 id="special-note--deprecations">Special Note / Deprecations</h2>
<h3 id="python-2">Python 2</h3>
-
<p>Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2.
Airflow 1.10.* would be the last series to support Python 2.</p>
-
<p>We strongly recommend users to use Python >= 3.6</p>
-
<h3 id="use-airflow-rbac-ui">Use Airflow RBAC UI</h3>
-
<p>Airflow 1.10.9 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI.</p>
-
<p>The Flask-AppBuilder (FAB) based UI is allows Role-based Access Control and has more advanced features compared to
the legacy Flask-admin based UI. This UI can be enabled by setting <code>rbac=True</code> in <code>[webserver]</code> section in your <code>airflow.cfg</code>.</p>
-
<p>Flask-admin based UI is deprecated and new features won’t be ported to it. This UI will still be the default
for 1.10.* series but would no longer be available from Airflow 2.0</p>
-
<h2 id="list-of-contributors">List of Contributors</h2>
-
<p>According to git shortlog, the following people contributed to the 1.10.8 and 1.10.9 release. Thank you to all contributors!</p>
-
<p>Anita Fronczak, Ash Berlin-Taylor, BasPH, Bharat Kashyap, Bharath Palaksha, Bhavika Tekwani, Bjorn Olsen, Brian Phillips, Cooper Gillan, Daniel Cohen, Daniel Imberman, Daniel Standish, Gabriel Eckers, Hossein Torabi, Igor Khrol, Jacob, Jarek Potiuk, Jay, Jiajie Zhong, Jithin Sukumar, Kamil Breguลa, Kaxil Naik, Kousuke Saruta, Mustafa Gรถk, Noรซl Bardelot, Oluwafemi Sule, Pete DeJoy, QP Hou, Qian Yu, Robin Edwards, Ry Walker, Steven van Rossum, Tomek Urbaszek, Xinbin Huang, Yuen-Kuei Hsu [...]
</div>
@@ -891,7 +857,7 @@ for 1.10.* series but would no longer be available from Airflow 2.0</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-2.2.0/index.html b/blog/airflow-2.2.0/index.html
index aa0ae83cb3..59c69bbeed 100644
--- a/blog/airflow-2.2.0/index.html
+++ b/blog/airflow-2.2.0/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>What's new in Apache Airflow 2.2.0 | Apache Airflow</title><meta property="og:title" content="What's new in Apache Airflow 2.2.0" />
<meta property="og:description" content="We're proud to announce that Apache Airflow 2.2.0 has been released." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.2.0/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-2.2.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2021-10-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="What's new in Apache Airflow 2.2.0">
-<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.2.0 has been released.">
-
-
-<meta itemprop="datePublished" content="2021-10-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="658">
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-<meta itemprop="keywords" content="Release," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="What's new in Apache Airflow 2.2.0">
+<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.2.0 has been released."><meta itemprop="datePublished" content="2021-10-11T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="658"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Release," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="What's new in Apache Airflow 2.2.0"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -480,81 +472,49 @@
<p class="blogpost-content__metadata--description">We're proud to announce that Apache Airflow 2.2.0 has been released.</p>
</div>
<div class="markdown-content">
-
-
-<p>Iโm proud to announce that Apache Airflow 2.2.0 has been released. It contains over 600 commits since 2.1.4 and includes 30 new features, 84 improvements, 85 bug fixes, and many internal and doc changes.</p>
-
+ <p>Iโm proud to announce that Apache Airflow 2.2.0 has been released. It contains over 600 commits since 2.1.4 and includes 30 new features, 84 improvements, 85 bug fixes, and many internal and doc changes.</p>
<p><strong>Details</strong>:</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.0/" target="_blank">https://pypi.org/project/apache-airflow/2.2.0/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.0/</a> <br />
-๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html</a> <br />
-๐ณ Docker Image: docker pull apache/airflow:2.2.0 <br />
-๐ Constraints: <a href="https://github.com/apache/airflow/tree/constraints-2.2.0" target="_blank">https://github.com/apache/airflow/tree/constraints-2.2.0</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.2.0/">https://pypi.org/project/apache-airflow/2.2.0/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/">https://airflow.apache.org/docs/apache-airflow/2.2.0/</a> <br>
+๐ ๏ธ Changelog: <a href="https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html">https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html</a> <br>
+๐ณ Docker Image: docker pull apache/airflow:2.2.0 <br>
+๐ Constraints: <a href="https://github.com/apache/airflow/tree/constraints-2.2.0">https://github.com/apache/airflow/tree/constraints-2.2.0</a></p>
<p>As the changelog is quite large, the following are some notable new features that shipped in this release.</p>
-
<h2 id="custom-timetables-aip-39">Custom Timetables (AIP-39)</h2>
-
<p>Airflow has historically used cron expressions and timedeltas to represent when a DAG should run. This worked for a lot of use cases, but not all. For example, running daily on Monday-Friday, but not on weekends wasnโt possible.</p>
-
<p>To provide more scheduling flexibility, determining when a DAG should run is now done with Timetables. Of course, backwards compatibility has been maintained - cron expressions and timedeltas are still fully supported, however, timetables are pluggable so you can add your own custom timetable to fit your needs! For example, you could write a timetable to schedule a DagRun</p>
-
<p><code>execution_date</code> has long been confusing to new Airflowers, so as part of this change a new concept has been added to Airflow to replace it named <code>data_interval</code>, which is the period of data that a task should operate on. The following are now available:</p>
-
<ul>
<li><code>logical_date</code> (aka <code>execution_date</code>)</li>
<li><code>data_interval_start</code> (same value as <code>execution_date</code> for cron)</li>
<li><code>data_interval_end</code> (aka <code>next_execution_date</code>)</li>
</ul>
-
<p>If you write your own timetables, keep in mind they should be idempotent and fast as they are used in the scheduler to create DagRuns.</p>
-
-<p>More information can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/timetable.html" target="_blank">Customizing DAG Scheduling with Timetables</a></p>
-
+<p>More information can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/timetable.html">Customizing DAG Scheduling with Timetables</a></p>
<h2 id="deferrable-tasks-aip-40">Deferrable Tasks (AIP-40)</h2>
-
<p>Deferrable tasks allows operators or sensors to defer themselves until a light-weight async check passes, at which point they can resume executing. Most importantly, this results in the worker slot, and most notably any resources used by it, to be returned to Airflow. This allows simple things like monitoring a job in an external system or watching for an event to be much cheaper.</p>
-
<p>To support this feature, a new component has been added to Airflow, the triggerer, which is the daemon process that runs the asyncio event loop.</p>
-
<p>Airflow 2.2.0 ships with 2 deferrable sensors, <code>DateTimeSensorAsync</code> and <code>TimeDeltaSensorAsync</code>, both of which are drop-in replacements for the existing corresponding sensor.</p>
-
<p>More information can be found at:</p>
-
-<p><a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html" target="_blank">Deferrable Operators & Triggers</a></p>
-
-<h2 id="custom-task-decorators-and-task-docker">Custom <code>@task</code> decorators and <code>@task.docker</code></h2>
-
+<p><a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html">Deferrable Operators & Triggers</a></p>
+<h2 id="custom-task-decorators-and-taskdocker">Custom <code>@task</code> decorators and <code>@task.docker</code></h2>
<p>Airflow 2.2.0 allows providers to create custom <code>@task</code> decorators in the TaskFlow interface.</p>
-
<p>The <code>@task.docker</code> decorator is one such decorator that allows you to run a function in a docker container. Airflow handles getting the code into the container and returning xcom - you just worry about your function. This is particularly useful when you have conflicting dependencies between Airflow itself and tasks you need to run.</p>
-
-<p>More information on creating custom <code>@task</code> decorators can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/create-custom-decorator.html" target="_blank">Creating Custom @task Decorators</a></p>
-
-<p>More information on the <code>@task.docker</code> decorator can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments" target="_blank">Using the Taskflow API with Docker or Virtual Environments</a></p>
-
+<p>More information on creating custom <code>@task</code> decorators can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/create-custom-decorator.html">Creating Custom @task Decorators</a></p>
+<p>More information on the <code>@task.docker</code> decorator can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments">Using the Taskflow API with Docker or Virtual Environments</a></p>
<h2 id="validation-of-dag-params">Validation of DAG params</h2>
-
-<p>You can now apply validation on DAG params by passing a <code>Param</code> object for each param. The <code>Param</code> object supports the full <a href="https://json-schema.org/draft/2020-12/json-schema-validation.html" target="_blank">json-schema validation specifications</a>.</p>
-
+<p>You can now apply validation on DAG params by passing a <code>Param</code> object for each param. The <code>Param</code> object supports the full <a href="https://json-schema.org/draft/2020-12/json-schema-validation.html">json-schema validation specifications</a>.</p>
<p>Currently this only functions with manually triggered DAGs, but it does set the stage for future params related functionality.</p>
-
-<p>More information can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html" target="_blank">Params</a></p>
-
+<p>More information can be found at: <a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html">Params</a></p>
<h2 id="other-small-features">Other small features</h2>
-
<p>This isnโt a comprehensive list, but some noteworthy or interesting small features include:</p>
-
<ul>
<li>Testing Connections from the UI - test the credentials for your Connection actually work</li>
<li>Duplication Connections from the UI</li>
<li>DAGs โNext runโ info is shown in the UI, including when the run will actually start</li>
<li><code>airflow standalone</code> command runs all of the Airflow components directly without docker - great for local development</li>
</ul>
-
<h2 id="contributors">Contributors</h2>
-
<p>Thanks to everyone who contributed to this release: Andrew Godwin, Ash Berlin-Taylor, Brent Bovenzi, Elad Kalif, Ephraim Anierobi, James Timmins, Jarek Potiuk, Jed Cunningham, Josh Fell, Kamil Breguลa, Kaxil Naik, Malthe Borch, Sam Wheating, Sumit Maheshwari, Tzu-ping Chung and many others</p>
</div>
@@ -883,7 +843,7 @@
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-2.3.0/index.html b/blog/airflow-2.3.0/index.html
index 6132ad189d..1235f4028b 100644
--- a/blog/airflow-2.3.0/index.html
+++ b/blog/airflow-2.3.0/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Apache Airflow 2.3.0 is here | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.3.0 is here" />
<meta property="og:description" content="We're proud to announce that Apache Airflow 2.3.0 has been released." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-2.3.0/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-2.3.0/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2022-04-30T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Apache Airflow 2.3.0 is here">
-<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.3.0 has been released.">
-
-
-<meta itemprop="datePublished" content="2022-04-30T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="708">
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-
-<meta itemprop="keywords" content="Release," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Apache Airflow 2.3.0 is here">
+<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.3.0 has been released."><meta itemprop="datePublished" content="2022-04-30T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="708"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Release," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Apache Airflow 2.3.0 is here"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -480,132 +472,87 @@
<p class="blogpost-content__metadata--description">We're proud to announce that Apache Airflow 2.3.0 has been released.</p>
</div>
<div class="markdown-content">
-
-
-<p>Apache Airflow 2.3.0 contains over 700 commits since 2.2.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.</p>
-
+ <p>Apache Airflow 2.3.0 contains over 700 commits since 2.2.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.</p>
<p><strong>Details</strong>:</p>
-
-<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.0/" target="_blank">https://pypi.org/project/apache-airflow/2.3.0/</a> <br />
-๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.0/</a> <br />
-๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html" target="_blank">https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html</a> <br />
-๐ณ Docker Image: docker pull apache/airflow:2.3.0 <br />
-๐ Constraints: <a href="https://github.com/apache/airflow/tree/constraints-2.3.0" target="_blank">https://github.com/apache/airflow/tree/constraints-2.3.0</a></p>
-
+<p>๐ฆ PyPI: <a href="https://pypi.org/project/apache-airflow/2.3.0/">https://pypi.org/project/apache-airflow/2.3.0/</a> <br>
+๐ Docs: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/">https://airflow.apache.org/docs/apache-airflow/2.3.0/</a> <br>
+๐ ๏ธ Release Notes: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html">https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html</a> <br>
+๐ณ Docker Image: docker pull apache/airflow:2.3.0 <br>
+๐ Constraints: <a href="https://github.com/apache/airflow/tree/constraints-2.3.0">https://github.com/apache/airflow/tree/constraints-2.3.0</a></p>
<p>As the changelog is quite large, the following are some notable new features that shipped in this release.</p>
-
-<h2 id="dynamic-task-mapping-aip-42">Dynamic Task Mapping(AIP-42)</h2>
-
+<h2 id="dynamic-task-mappingaip-42">Dynamic Task Mapping(AIP-42)</h2>
<p>There’s now first-class support for dynamic tasks in Airflow. What this means is that you can generate tasks dynamically at runtime. Much like using a <code>for</code> loop
to create a list of tasks, here you can create the same tasks without having to know the exact number of tasks ahead of time.</p>
-
<p>You can have a <code>task</code> generate the list to iterate over, which is not possible with a <code>for</code> loop.</p>
-
<p>Here is an example:</p>
-
-<pre><code class="language-python">@task
-def make_list():
- # This can also be from an API call, checking a database, -- almost anything you like, as long as the
- # resulting list/dictionary can be stored in the current XCom backend.
- return [1, 2, {"a": "b"}, "str"]
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="nd">@task</span>
+<span class="k">def</span> <span class="nf">make_list</span><span class="p">():</span>
+ <span class="c1"># This can also be from an API call, checking a database, -- almost anything you like, as long as the</span>
+ <span class="c1"># resulting list/dictionary can be stored in the current XCom backend.</span>
+ <span class="k">return</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="p">{</span><span class="s2">"a"</span><span class="p">:</span> <span class="s2">"b"</span><span class="p">},</span> <span class="s2">"str"</span><span class="p">]</span>
-@task
-def consumer(arg):
- print(list(arg))
+<span class="nd">@task</span>
+<span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
+ <span class="nb">print</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">arg</span><span class="p">))</span>
-with DAG(dag_id="dynamic-map", start_date=datetime(2022, 4, 2)) as dag:
- consumer.expand(arg=make_list())
-</code></pre>
-
-<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html" target="_blank">Dynamic Task Mapping</a></p>
-
+<span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="s2">"dynamic-map"</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2022</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span> <span class="k">as</span> [...]
+ <span class="n">consumer</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="n">arg</span><span class="o">=</span><span class="n">make_list</span><span class="p">())</span>
+</code></pre></div><p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html">Dynamic Task Mapping</a></p>
<h2 id="grid-view-replaces-tree-view">Grid View replaces Tree View</h2>
-
<p>Grid view replaces tree view in Airflow 2.3.0.</p>
-
<p><strong>Screenshots</strong>:
-<img src="grid-view.png" alt="The new grid view" /></p>
-
+<img src="grid-view.png" alt="The new grid view"></p>
<h2 id="purge-history-from-metadata-database">Purge history from metadata database</h2>
-
<p>Airflow 2.3.0 introduces a new <code>airflow db clean</code> command that can be used to purge old data from the metadata database.</p>
-
<p>You would want to use this command if you want to reduce the size of the metadata database.</p>
-
-<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/usage-cli.html#purge-history-from-metadata-database" target="_blank">Purge history from metadata database</a></p>
-
+<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/usage-cli.html#purge-history-from-metadata-database">Purge history from metadata database</a></p>
<h2 id="localkubernetesexecutor">LocalKubernetesExecutor</h2>
-
<p>There is a new executor named LocalKubernetesExecutor. This executor helps you run some tasks using LocalExecutor and run another set of tasks using the KubernetesExecutor in the same deployment based on the task’s queue.</p>
-
-<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/executor/local_kubernetes.html" target="_blank">LocalKubernetesExecutor</a></p>
-
+<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/executor/local_kubernetes.html">LocalKubernetesExecutor</a></p>
<h2 id="dagprocessormanager-as-standalone-process-aip-43">DagProcessorManager as standalone process (AIP-43)</h2>
-
<p>As of 2.3.0, you can run the DagProcessorManager as a standalone process. Because DagProcessorManager runs user code, separating it from the scheduler process and running it as an independent process in a different host is a good idea.</p>
-
-<p>The <code>airflow dag-processor</code> cli command will start a new process that will run the DagProcessorManager in a separate process. Before you can run the DagProcessorManager as a standalone process, you need to set the <a href="https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#standalone_dag_processor" target="_blank">[scheduler] standalone_dag_processor</a> to <code>True</code>.</p>
-
-<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/cli-and-env-variables-ref.html#dag-processor" target="_blank">dag-processor CLI command</a></p>
-
+<p>The <code>airflow dag-processor</code> cli command will start a new process that will run the DagProcessorManager in a separate process. Before you can run the DagProcessorManager as a standalone process, you need to set the <a href="https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#standalone_dag_processor">[scheduler] standalone_dag_processor</a> to <code>True</code>.</p>
+<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/cli-and-env-variables-ref.html#dag-processor">dag-processor CLI command</a></p>
<h2 id="json-serialization-for-connections">JSON serialization for connections</h2>
-
<p>You can now create connections using the <code>json</code> serialization format.</p>
-
-<pre><code class="language-bash">airflow connections add 'my_prod_db' \
- --conn-json '{
- "conn_type": "my-conn-type",
- "login": "my-login",
- "password": "my-password",
- "host": "my-host",
- "port": 1234,
- "schema": "my-schema",
- "extra": {
- "param1": "val1",
- "param2": "val2"
- }
- }'
-</code></pre>
-
-<p>You can also use <code>json</code> serialization format when setting the connection in environment variables.</p>
-
-<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/howto/connection.html" target="_blank">JSON serialization for connections</a></p>
-
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash">airflow connections add <span class="s1">'my_prod_db'</span> <span class="se">\
+</span><span class="se"></span> --conn-json <span class="s1">'{
+</span><span class="s1"> "conn_type": "my-conn-type",
+</span><span class="s1"> "login": "my-login",
+</span><span class="s1"> "password": "my-password",
+</span><span class="s1"> "host": "my-host",
+</span><span class="s1"> "port": 1234,
+</span><span class="s1"> "schema": "my-schema",
+</span><span class="s1"> "extra": {
+</span><span class="s1"> "param1": "val1",
+</span><span class="s1"> "param2": "val2"
+</span><span class="s1"> }
+</span><span class="s1"> }'</span>
+</code></pre></div><p>You can also use <code>json</code> serialization format when setting the connection in environment variables.</p>
+<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/howto/connection.html">JSON serialization for connections</a></p>
<h2 id="airflow-db-downgrade-and-offline-generation-of-sql-scripts">Airflow <code>db downgrade</code> and Offline generation of SQL scripts</h2>
-
<p>Airflow 2.3.0 introduced a new command <code>airflow db downgrade</code> that will downgrade the database to your chosen version.</p>
-
<p>You can also generate the downgrade/upgrade SQL scripts for your database and manually run it against your database or just view the SQL queries that would be run by the downgrade/upgrade command.</p>
-
-<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/usage-cli.html#downgrading-airflow" target="_blank">Airflow <code>db downgrade</code> and Offline generation of SQL scripts</a></p>
-
+<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/usage-cli.html#downgrading-airflow">Airflow <code>db downgrade</code> and Offline generation of SQL scripts</a></p>
<h2 id="reuse-of-decorated-tasks">Reuse of decorated tasks</h2>
-
<p>You can now reuse decorated tasks across your dag files. A decorated task has an <code>override</code> method that allows you to override it’s arguments.</p>
-
<p>Here’s an example:</p>
-
-<pre><code class="language-python">@task
-def add_task(x, y):
- print(f"Task args: x={x}, y={y}")
- return x + y
-
-
-@dag(start_date=datetime(2022, 1, 1))
-def mydag():
- start = add_task.override(task_id="start")(1, 2)
- for i in range(3):
- start >> add_task.override(task_id=f"add_start_{i}")(start, i)
-</code></pre>
-
-<p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/tutorial_taskflow_api.html#reusing-a-decorated-task" target="_blank">Reuse of decorated DAGs</a></p>
-
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="nd">@task</span>
+<span class="k">def</span> <span class="nf">add_task</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
+ <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Task args: x=</span><span class="si">{</span><span class="n">x</span><span class="si">}</span><span class="s2">, y=</span><span class="si">{</span><span class="n">y</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
+
+
+<span class="nd">@dag</span><span class="p">(</span><span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2022</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
+<span class="k">def</span> <span class="nf">mydag</span><span class="p">():</span>
+ <span class="n">start</span> <span class="o">=</span> <span class="n">add_task</span><span class="o">.</span><span class="n">override</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s2">"start"</span><span class="p">)(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
+ <span class="n">start</span> <span class="o">>></span> <span class="n">add_task</span><span class="o">.</span><span class="n">override</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="sa">f</span><span class="s2">"add_start_</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">)(</span><span class="n">start</span><span class="p">,</span> <span class="n [...]
+</code></pre></div><p>More information can be found here: <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/tutorial_taskflow_api.html#reusing-a-decorated-task">Reuse of decorated DAGs</a></p>
<h2 id="other-small-features">Other small features</h2>
-
<p>This isnโt a comprehensive list, but some noteworthy or interesting small features include:</p>
-
<ul>
<li>Support different timeout value for dag file parsing</li>
<li><code>airflow dags reserialize</code> command to reserialize dags</li>
@@ -613,9 +560,7 @@ def mydag():
<li>SmoothOperator - Operator that does literally nothing except logging a YouTube link to
Sade’s “Smooth Operator”. Enjoy!</li>
</ul>
-
<h2 id="contributors">Contributors</h2>
-
<p>Thanks to everyone who contributed to this release: Ash Berlin-Taylor, Brent Bovenzi, Daniel Standish, Elad, Ephraim Anierobi, Jarek Potiuk, Jed Cunningham, Josh Fell, Kamil Breguลa, Kanthi, Kaxil Naik, Khalid Mammadov, Malthe Borch, Ping Zhang, Tzu-ping Chung and many others who keep making Airflow better for everyone.</p>
</div>
@@ -944,7 +889,7 @@ Sade’s “Smooth Operator”. Enjoy!</li>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-survey-2020/index.html b/blog/airflow-survey-2020/index.html
index 3c6d0fd453..254ea1e5e3 100644
--- a/blog/airflow-survey-2020/index.html
+++ b/blog/airflow-survey-2020/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Airflow Survey 2020 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2020" />
<meta property="og:description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey-2020/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-survey-2020/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2021-03-09T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Airflow Survey 2020">
-<meta itemprop="description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.">
-
-
-<meta itemprop="datePublished" content="2021-03-09T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="2136">
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-<meta itemprop="keywords" content="community,survey,users," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Airflow Survey 2020">
+<meta itemprop="description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance."><meta itemprop="datePublished" content="2021-03-09T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="2136"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="community,survey,users," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Airflow Survey 2020"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -503,24 +495,16 @@
<p class="blogpost-content__metadata--description">We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.</p>
</div>
<div class="markdown-content">
-
-
-<h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
-
+ <h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
<p>World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
an amount of active contributors. So listening and understanding our community is of high importance.</p>
-
<p>It’s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
is true, we will learn next year!</p>
-
<h2 id="overview-of-the-user">Overview of the user</h2>
-
-<p><img src="What_best_describes_your_current_occupation.png" alt="" /></p>
-
+<p><img src="What_best_describes_your_current_occupation.png" alt=""></p>
<p><strong>What best describes your current occupation? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -529,50 +513,42 @@ is true, we will learn next year!</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Data Engineer</td>
<td>115</td>
<td>56.65</td>
</tr>
-
<tr>
<td>Developer</td>
<td>28</td>
<td>13.79</td>
</tr>
-
<tr>
<td>DevOps</td>
<td>17</td>
<td>8.37</td>
</tr>
-
<tr>
<td>Solutions Architect</td>
<td>14</td>
<td>6.9</td>
</tr>
-
<tr>
<td>Data Scientist</td>
<td>12</td>
<td>5.91</td>
</tr>
-
<tr>
<td>Other</td>
<td>10</td>
<td>4.93</td>
</tr>
-
<tr>
<td>Data Analyst</td>
<td>4</td>
<td>1.97</td>
</tr>
-
<tr>
<td>Support Engineer</td>
<td>3</td>
@@ -580,15 +556,11 @@ is true, we will learn next year!</p>
</tr>
</tbody>
</table>
-
<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
-our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> with
+our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/">those from 2019</a> with
visible slight increase in ML use cases.</p>
-
<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
-
<p><strong>How many people work in your company? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -597,26 +569,22 @@ visible slight increase in ML use cases.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>200+</td>
<td>107</td>
<td>52.71</td>
</tr>
-
<tr>
<td>51-200</td>
<td>44</td>
<td>21.67</td>
</tr>
-
<tr>
<td>11-50</td>
<td>37</td>
<td>18.23</td>
</tr>
-
<tr>
<td>1-10</td>
<td>15</td>
@@ -624,9 +592,7 @@ visible slight increase in ML use cases.</p>
</tr>
</tbody>
</table>
-
<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -635,26 +601,22 @@ visible slight increase in ML use cases.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>1-5</td>
<td>84</td>
<td>41.38</td>
</tr>
-
<tr>
<td>6-20</td>
<td>75</td>
<td>36.95</td>
</tr>
-
<tr>
<td>21-50</td>
<td>23</td>
<td>11.33</td>
</tr>
-
<tr>
<td>50+</td>
<td>21</td>
@@ -662,12 +624,9 @@ visible slight increase in ML use cases.</p>
</tr>
</tbody>
</table>
-
<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
-
<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -676,14 +635,12 @@ fine for teams of different sizes. However, in some cases users may use multiple
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>No, we are happy with Airflow</td>
<td>174</td>
<td>85.71</td>
</tr>
-
<tr>
<td>Yes</td>
<td>29</td>
@@ -691,19 +648,14 @@ fine for teams of different sizes. However, in some cases users may use multiple
</tr>
</tbody>
</table>
-
<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
(8.37% both).</p>
-
-<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api" target="_blank">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
+<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
in the new major version. And the early feedback from 2.0 users seems to be confirming it.</p>
-
<p>The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
Luigi, Kubeflow or custom solutions.</p>
-
<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -712,20 +664,17 @@ Luigi, Kubeflow or custom solutions.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>I wish we could</td>
<td>99</td>
<td>48.77</td>
</tr>
-
<tr>
<td>No</td>
<td>59</td>
<td>29.06</td>
</tr>
-
<tr>
<td>Yes</td>
<td>45</td>
@@ -733,15 +682,12 @@ Luigi, Kubeflow or custom solutions.</p>
</tr>
</tbody>
</table>
-
<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
But it would be good to learn if there’s something else than time that is stopping people who wish to contribute
from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
That said - if you know something we can improve please reach out via Slack, dev list or Github
discussions.</p>
-
<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -751,7 +697,6 @@ discussions.</p>
<th>2019 %</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Very Likely</td>
@@ -759,28 +704,24 @@ discussions.</p>
<td>61.58</td>
<td>45.45%</td>
</tr>
-
<tr>
<td>Likely</td>
<td>62</td>
<td>30.54</td>
<td>40.26%</td>
</tr>
-
<tr>
<td>Neutral</td>
<td>11</td>
<td>5.42</td>
<td>10.71%</td>
</tr>
-
<tr>
<td>Unlikely</td>
<td>3</td>
<td>1.48</td>
<td>2.60%</td>
</tr>
-
<tr>
<td>Very unlikely</td>
<td>2</td>
@@ -789,11 +730,8 @@ discussions.</p>
</tr>
</tbody>
</table>
-
<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
-
<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -802,56 +740,47 @@ discussions.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Documentation</td>
<td>154</td>
<td>75.86</td>
</tr>
-
<tr>
<td>Airflow website</td>
<td>139</td>
<td>68.47</td>
</tr>
-
<tr>
<td>Slack</td>
<td>128</td>
<td>63.05</td>
</tr>
-
<tr>
<td>Github</td>
<td>127</td>
<td>62.56</td>
</tr>
-
<tr>
<td>Stack Overflow</td>
<td>72</td>
<td>35.47</td>
</tr>
-
<tr>
<td>Airflow Summit Videos</td>
<td>44</td>
<td>21.67</td>
</tr>
-
<tr>
<td>The dev mailing list</td>
<td>33</td>
<td>16.26</td>
</tr>
-
<tr>
<td>Awesome Apache Airflow repository</td>
<td>21</td>
<td>10.34</td>
</tr>
-
<tr>
<td>Other</td>
<td>15</td>
@@ -859,16 +788,11 @@ discussions.</p>
</tr>
</tbody>
</table>
-
<p>Here we see that Airflow documentation is the crucial source of information. What’s interesting is that more
than 60% of users are getting information from Github and Slack channels.</p>
-
-<p><img src="Where_are_you_based.png" alt="" /></p>
-
+<p><img src="Where_are_you_based.png" alt=""></p>
<h2 id="airflow-uses-cases">Airflow uses cases</h2>
-
<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -877,20 +801,17 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>No, we use vanilla Airflow</td>
<td>154</td>
<td>75.86</td>
</tr>
-
<tr>
<td>Yes, we have small patches (no fork)</td>
<td>34</td>
<td>16.75</td>
</tr>
-
<tr>
<td>Yes, we have separate fork</td>
<td>15</td>
@@ -898,9 +819,7 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -909,44 +828,37 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>No guide on best practises on developing DAGs</td>
<td>102</td>
<td>50.25</td>
</tr>
-
<tr>
<td>There is no easy option to launch Airflow</td>
<td>64</td>
<td>31.53</td>
</tr>
-
<tr>
<td>Small number of tutorials on different aspects of using Airflow</td>
<td>57</td>
<td>28.08</td>
</tr>
-
<tr>
<td>Documentation is not clear enough</td>
<td>53</td>
<td>26.11</td>
</tr>
-
<tr>
<td>There is no easy option to deploy DAGs to an Airflow instance</td>
<td>52</td>
<td>25.62</td>
</tr>
-
<tr>
<td>No problems</td>
<td>34</td>
<td>16.75</td>
</tr>
-
<tr>
<td>Small number of blogs regarding Airflow</td>
<td>30</td>
@@ -954,9 +866,7 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -965,32 +875,27 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Original Airflow Graphical User Interface</td>
<td>199</td>
<td>98.03</td>
</tr>
-
<tr>
<td>CLI</td>
<td>88</td>
<td>43.35</td>
</tr>
-
<tr>
<td>API</td>
<td>48</td>
<td>23.65</td>
</tr>
-
<tr>
<td>Custom (own created) Airflow Graphical User Interface</td>
<td>12</td>
<td>5.91</td>
</tr>
-
<tr>
<td>Other</td>
<td>3</td>
@@ -998,9 +903,7 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1009,26 +912,22 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Yes, by triggering another DAG</td>
<td>87</td>
<td>42.86</td>
</tr>
-
<tr>
<td>No, I don’t combine multiple DAGs</td>
<td>79</td>
<td>38.92</td>
</tr>
-
<tr>
<td>Yes, through SubDAG</td>
<td>40</td>
<td>19.7</td>
</tr>
-
<tr>
<td>Other</td>
<td>18</td>
@@ -1036,9 +935,7 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1047,26 +944,22 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Using existing dedicated operators / hooks</td>
<td>147</td>
<td>72.41</td>
</tr>
-
<tr>
<td>Using Bash / Python operator</td>
<td>140</td>
<td>68.97</td>
</tr>
-
<tr>
<td>Using own custom operators / hooks</td>
<td>138</td>
<td>67.98</td>
</tr>
-
<tr>
<td>Other</td>
<td>12</td>
@@ -1074,9 +967,7 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1085,44 +976,37 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Amazon Web Services</td>
<td>121</td>
<td>59.61</td>
</tr>
-
<tr>
<td>Internal company systems</td>
<td>113</td>
<td>55.67</td>
</tr>
-
<tr>
<td>Google Cloud Platform / Google APIs</td>
<td>97</td>
<td>47.78</td>
</tr>
-
<tr>
<td>Hadoop / Spark / Flink / Other Apache software</td>
<td>72</td>
<td>35.47</td>
</tr>
-
<tr>
<td>Microsoft Azure</td>
<td>21</td>
<td>10.34</td>
</tr>
-
<tr>
<td>Other</td>
<td>19</td>
<td>9.36</td>
</tr>
-
<tr>
<td>I do not use external services in my Airflow DAGs</td>
<td>5</td>
@@ -1130,11 +1014,8 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
-<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt="" /></p>
-
+<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt=""></p>
<p><strong>Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1143,32 +1024,27 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Adding new operators/sensors and hooks</td>
<td>119</td>
<td>58.62</td>
</tr>
-
<tr>
<td>I don’t use Airflow plugins</td>
<td>69</td>
<td>33.99</td>
</tr>
-
<tr>
<td>Adding AppBuilder views & menu items</td>
<td>27</td>
<td>13.3</td>
</tr>
-
<tr>
<td>Adding new executors</td>
<td>17</td>
<td>8.37</td>
</tr>
-
<tr>
<td>Adding OperatorExtraLinks</td>
<td>13</td>
@@ -1176,11 +1052,8 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
<p>| Other</p>
-
<p><strong>Do you use Airflow’s data lineage feature? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1189,32 +1062,27 @@ than 60% of users are getting information from Github and Slack channels.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>No, I will use such feature if fully supported in Airflow</td>
<td>105</td>
<td>51.72</td>
</tr>
-
<tr>
<td>No, data lineage isnโt a concern for my usage.</td>
<td>68</td>
<td>33.5</td>
</tr>
-
<tr>
<td>Yes, I use another data lineage product</td>
<td>24</td>
<td>11.82</td>
</tr>
-
<tr>
<td>Yes, I use custom implementation</td>
<td>5</td>
<td>2.46</td>
</tr>
-
<tr>
<td>Yes, I use Airflow’s experimental data lineage feature</td>
<td>1</td>
@@ -1222,14 +1090,10 @@ than 60% of users are getting information from Github and Slack channels.</p>
</tr>
</tbody>
</table>
-
<p>When asked what lineage product users use, the answers were varying from custom tools
to known product like Amundsen, Atlas or dbt.</p>
-
<h2 id="deployment">Deployment</h2>
-
<p><strong>How many active DAGs do you have in your largest Airflow instance? (open question)</strong></p>
-
<table>
<thead>
<tr>
@@ -1238,50 +1102,42 @@ to known product like Amundsen, Atlas or dbt.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>< 20</td>
<td>64</td>
<td>32</td>
</tr>
-
<tr>
<td>21-40</td>
<td>33</td>
<td>16</td>
</tr>
-
<tr>
<td>41-60</td>
<td>13</td>
<td>6</td>
</tr>
-
<tr>
<td>61-100</td>
<td>32</td>
<td>16</td>
</tr>
-
<tr>
<td>101-200</td>
<td>31</td>
<td>15</td>
</tr>
-
<tr>
<td>201-300</td>
<td>8</td>
<td>4</td>
</tr>
-
<tr>
<td>301-999</td>
<td>12</td>
<td>6</td>
</tr>
-
<tr>
<td>1000+</td>
<td>10</td>
@@ -1289,9 +1145,7 @@ to known product like Amundsen, Atlas or dbt.</p>
</tr>
</tbody>
</table>
-
<p><strong>What is the maximum number of tasks that you have used in one DAG? (open question)</strong></p>
-
<table>
<thead>
<tr>
@@ -1300,56 +1154,47 @@ to known product like Amundsen, Atlas or dbt.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>< 10</td>
<td>42</td>
<td>21</td>
</tr>
-
<tr>
<td>11-20</td>
<td>31</td>
<td>15</td>
</tr>
-
<tr>
<td>21-30</td>
<td>15</td>
<td>7</td>
</tr>
-
<tr>
<td>31-40</td>
<td>11</td>
<td>5</td>
</tr>
-
<tr>
<td>41-50</td>
<td>22</td>
<td>11</td>
</tr>
-
<tr>
<td>51-100</td>
<td>39</td>
<td>19</td>
</tr>
-
<tr>
<td>101-200</td>
<td>16</td>
<td>8</td>
</tr>
-
<tr>
<td>201-500</td>
<td>16</td>
<td>8</td>
</tr>
-
<tr>
<td>501+</td>
<td>11</td>
@@ -1357,9 +1202,7 @@ to known product like Amundsen, Atlas or dbt.</p>
</tr>
</tbody>
</table>
-
<p><strong>Which version of Airflow do you use currently? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1368,68 +1211,57 @@ to known product like Amundsen, Atlas or dbt.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>1.10.14</td>
<td>55</td>
<td>27.09</td>
</tr>
-
<tr>
<td>2.0.0+</td>
<td>45</td>
<td>22.17</td>
</tr>
-
<tr>
<td>1.10.12</td>
<td>27</td>
<td>13.3</td>
</tr>
-
<tr>
<td>1.10.10</td>
<td>26</td>
<td>12.81</td>
</tr>
-
<tr>
<td>1.10.11</td>
<td>14</td>
<td>6.9</td>
</tr>
-
<tr>
<td>1.10.5 or older</td>
<td>10</td>
<td>4.93</td>
</tr>
-
<tr>
<td>1.10.9</td>
<td>8</td>
<td>3.94</td>
</tr>
-
<tr>
<td>1.10.13</td>
<td>7</td>
<td>3.45</td>
</tr>
-
<tr>
<td>1.10.6</td>
<td>4</td>
<td>1.97</td>
</tr>
-
<tr>
<td>1.10.7</td>
<td>4</td>
<td>1.97</td>
</tr>
-
<tr>
<td>1.10.8</td>
<td>3</td>
@@ -1437,15 +1269,11 @@ to known product like Amundsen, Atlas or dbt.</p>
</tr>
</tbody>
</table>
-
<p>This was probably one of the most important questions in the survey. While it’s good to see
that more than 60% of users use one of three latest Airflow versions, it’s worrying that the rest
are using versions that are old or have known security vulnerabilities.</p>
-
<p>Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.</p>
-
<p><strong>What meta-database do you use? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1454,50 +1282,42 @@ are using versions that are old or have known security vulnerabilities.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Postgres 12</td>
<td>36</td>
<td>17.73</td>
</tr>
-
<tr>
<td>Postgres 9.6</td>
<td>33</td>
<td>16.26</td>
</tr>
-
<tr>
<td>Postgres 11</td>
<td>31</td>
<td>15.27</td>
</tr>
-
<tr>
<td>MySQL 5.7</td>
<td>27</td>
<td>13.3</td>
</tr>
-
<tr>
<td>MySQL 8.0</td>
<td>20</td>
<td>9.85</td>
</tr>
-
<tr>
<td>Postgres 10</td>
<td>20</td>
<td>9.85</td>
</tr>
-
<tr>
<td>Other</td>
<td>19</td>
<td>9.36</td>
</tr>
-
<tr>
<td>Postgres 13</td>
<td>18</td>
@@ -1505,17 +1325,12 @@ are using versions that are old or have known security vulnerabilities.</p>
</tr>
</tbody>
</table>
-
<p>This means that more about 69% of users decide to use Postgres as their meta-database.
MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.</p>
-
<p>It’s good to know that users rather avoid using SQLite in production deployments!</p>
-
<p><strong>What executor type do you use? (single choice)</strong></p>
-
-<p><img src="What_executor_type_do_you_use.png" alt="" /></p>
-
+<p><img src="What_executor_type_do_you_use.png" alt=""></p>
<table>
<thead>
<tr>
@@ -1525,7 +1340,6 @@ like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) o
<th>2019</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Celery</td>
@@ -1533,28 +1347,24 @@ like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) o
<td>49.26%</td>
<td>44.81%</td>
</tr>
-
<tr>
<td>Kubernetes</td>
<td>48</td>
<td>23.65%</td>
<td>16.88%</td>
</tr>
-
<tr>
<td>Local</td>
<td>40</td>
<td>19.7%</td>
<td>27.60%</td>
</tr>
-
<tr>
<td>Sequential</td>
<td>10</td>
<td>4.93%</td>
<td>7.14%</td>
</tr>
-
<tr>
<td>Other</td>
<td>5</td>
@@ -1563,16 +1373,12 @@ like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) o
</tr>
</tbody>
</table>
-
<p>In comparison to previous year it seems that more users use currently Celery and
Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
-suggest that users’ deployments are growing, and they need more scalable solutions.</p>
-
+suggest that users' deployments are growing, and they need more scalable solutions.</p>
<p>Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
is using other brokers or is not sure what is used in their deployments.</p>
-
<p><strong>What metrics do you use to monitor Airflow? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1581,32 +1387,27 @@ is using other brokers or is not sure what is used in their deployments.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>I do not use monitoring</td>
<td>65</td>
<td>32.02</td>
</tr>
-
<tr>
<td>External monitoring service</td>
<td>60</td>
<td>29.56</td>
</tr>
-
<tr>
<td>Information from metadatabase</td>
<td>51</td>
<td>25.12</td>
</tr>
-
<tr>
<td>Statsd</td>
<td>49</td>
<td>24.14</td>
</tr>
-
<tr>
<td>Other</td>
<td>31</td>
@@ -1614,12 +1415,9 @@ is using other brokers or is not sure what is used in their deployments.</p>
</tr>
</tbody>
</table>
-
<p>The other responses included mostly information about tools used by users
including DataDog and Prometheus exporter.</p>
-
<p><strong>How do you deploy Airflow? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1628,44 +1426,37 @@ including DataDog and Prometheus exporter.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>On virtual machines (for example using AWS EC2)</td>
<td>64</td>
<td>31.53</td>
</tr>
-
<tr>
<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
<td>35</td>
<td>17.24</td>
</tr>
-
<tr>
<td>On Kubernetes (using custom deployments)</td>
<td>29</td>
<td>14.29</td>
</tr>
-
<tr>
<td>On premises</td>
<td>28</td>
<td>13.79</td>
</tr>
-
<tr>
<td>On Kubernetes (using another helm chart)</td>
<td>20</td>
<td>9.85</td>
</tr>
-
<tr>
<td>On Kubernetes (using Apache Airflow’s helm chart)</td>
<td>17</td>
<td>8.37</td>
</tr>
-
<tr>
<td>Other</td>
<td>12</td>
@@ -1673,13 +1464,10 @@ including DataDog and Prometheus exporter.</p>
</tr>
</tbody>
</table>
-
<p>Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
10 percent more than in 2019. There’s slightly increase in usage of Airflow via
managed services (14.61% in 2019).</p>
-
<p><strong>Do you use containerisation for deployment? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1688,26 +1476,22 @@ managed services (14.61% in 2019).</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Yes, using helm chart / kubernetes</td>
<td>58</td>
<td>28.57</td>
</tr>
-
<tr>
<td>No, I donโt use containerisation</td>
<td>57</td>
<td>28.08</td>
</tr>
-
<tr>
<td>Yes, single docker image</td>
<td>49</td>
<td>24.14</td>
</tr>
-
<tr>
<td>Yes, using docker compose</td>
<td>39</td>
@@ -1715,12 +1499,9 @@ managed services (14.61% in 2019).</p>
</tr>
</tbody>
</table>
-
<p>Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
42% of those users use docker-compose for deployments.</p>
-
<p><strong>How do you distribute your DAGs? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1729,32 +1510,27 @@ managed services (14.61% in 2019).</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
<td>79</td>
<td>38.92</td>
</tr>
-
<tr>
<td>Bake them into the docker image</td>
<td>56</td>
<td>27.59</td>
</tr>
-
<tr>
<td>Shared files system</td>
<td>34</td>
<td>16.75</td>
</tr>
-
<tr>
<td>Other</td>
<td>20</td>
<td>9.85</td>
</tr>
-
<tr>
<td>I donโt know</td>
<td>14</td>
@@ -1762,14 +1538,10 @@ managed services (14.61% in 2019).</p>
</tr>
</tbody>
</table>
-
<p>The most popular way of distributing DAGs seems to be using a synchronizing process. About
40% of users use this process together with Kubernetes deployments.</p>
-
<h2 id="future-of-airflow">Future of Airflow</h2>
-
<p><strong>In your opinion, what could be improved in Airflow? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1778,92 +1550,77 @@ managed services (14.61% in 2019).</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Web UI</td>
<td>100</td>
<td>49.26</td>
</tr>
-
<tr>
<td>Examples, how-to, onboarding documentation</td>
<td>90</td>
<td>44.33</td>
</tr>
-
<tr>
<td>Logging, monitoring and alerting</td>
<td>90</td>
<td>44.33</td>
</tr>
-
<tr>
<td>Technical documentation</td>
<td>90</td>
<td>44.33</td>
</tr>
-
<tr>
<td>Scheduler performance</td>
<td>83</td>
<td>40.89</td>
</tr>
-
<tr>
<td>DAG authoring</td>
<td>64</td>
<td>31.53</td>
</tr>
-
<tr>
<td>Authentication and authorization</td>
<td>58</td>
<td>28.57</td>
</tr>
-
<tr>
<td>REST API</td>
<td>51</td>
<td>25.12</td>
</tr>
-
<tr>
<td>Other</td>
<td>44</td>
<td>21.67</td>
</tr>
-
<tr>
<td>Reliability</td>
<td>41</td>
<td>20.2</td>
</tr>
-
<tr>
<td>External integration e.g. AWS, GCP, Apache products</td>
<td>36</td>
<td>17.73</td>
</tr>
-
<tr>
<td>Security</td>
<td>28</td>
<td>13.79</td>
</tr>
-
<tr>
<td>CLI</td>
<td>20</td>
<td>9.85</td>
</tr>
-
<tr>
<td>Everything work fine for me</td>
<td>14</td>
<td>6.9</td>
</tr>
-
<tr>
<td>I donโt know</td>
<td>4</td>
@@ -1871,9 +1628,7 @@ managed services (14.61% in 2019).</p>
</tr>
</tbody>
</table>
-
<p><strong>Which features would most interest you? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1882,74 +1637,62 @@ managed services (14.61% in 2019).</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>DAG versioning</td>
<td>109</td>
<td>53.69</td>
</tr>
-
<tr>
<td>Builtin statistics</td>
<td>71</td>
<td>34.98</td>
</tr>
-
<tr>
<td>Improved data lineage</td>
<td>65</td>
<td>32.02</td>
</tr>
-
<tr>
<td>Scheduling at the start of the interval</td>
<td>63</td>
<td>31.03</td>
</tr>
-
<tr>
<td>Stateless workers</td>
<td>59</td>
<td>29.06</td>
</tr>
-
<tr>
<td>More option to configure schedules (time units, increments)</td>
<td>57</td>
<td>28.08</td>
</tr>
-
<tr>
<td>Multi-tenant deployment</td>
<td>49</td>
<td>24.14</td>
</tr>
-
<tr>
<td>DAG fetcher (AIP-5)</td>
<td>39</td>
<td>19.21</td>
</tr>
-
<tr>
<td>Generic transfer operator</td>
<td>34</td>
<td>16.75</td>
</tr>
-
<tr>
<td>Other</td>
<td>33</td>
<td>16.26</td>
</tr>
-
<tr>
<td>I have everything I need</td>
<td>11</td>
<td>5.42</td>
</tr>
-
<tr>
<td>Nothing</td>
<td>11</td>
@@ -1957,9 +1700,7 @@ managed services (14.61% in 2019).</p>
</tr>
</tbody>
</table>
-
<p><strong>Will you consider migrating to Airflow 2.0? (single choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -1968,32 +1709,27 @@ managed services (14.61% in 2019).</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Yes, as soon as possible</td>
<td>81</td>
<td>39.9</td>
</tr>
-
<tr>
<td>Yes, once itโs mature (for example after 2.1)</td>
<td>72</td>
<td>35.47</td>
</tr>
-
<tr>
<td>I am already using Airflow 2.0+</td>
<td>39</td>
<td>19.21</td>
</tr>
-
<tr>
<td>I don’t know yet</td>
<td>8</td>
<td>3.94</td>
</tr>
-
<tr>
<td>No, I do not plan to migrate</td>
<td>3</td>
@@ -2001,9 +1737,7 @@ managed services (14.61% in 2019).</p>
</tr>
</tbody>
</table>
-
<p><strong>What are the features of Airflow 2.0 you are most excited about? (multiple choice)</strong></p>
-
<table>
<thead>
<tr>
@@ -2012,62 +1746,52 @@ managed services (14.61% in 2019).</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>General performance improvements</td>
<td>133</td>
<td>65.52</td>
</tr>
-
<tr>
<td>Refreshed WebUI</td>
<td>102</td>
<td>50.25</td>
</tr>
-
<tr>
<td>Scheduler HA</td>
<td>99</td>
<td>48.77</td>
</tr>
-
<tr>
<td>Official docker image</td>
<td>84</td>
<td>41.38</td>
</tr>
-
<tr>
<td>@task decorator</td>
<td>56</td>
<td>27.59</td>
</tr>
-
<tr>
<td>Official helm chart</td>
<td>51</td>
<td>25.12</td>
</tr>
-
<tr>
<td>Providers packages</td>
<td>41</td>
<td>20.2</td>
</tr>
-
<tr>
<td>Configurable XCom backends</td>
<td>33</td>
<td>16.26</td>
</tr>
-
<tr>
<td>CeleryKubernetesExecutor</td>
<td>31</td>
<td>15.27</td>
</tr>
-
<tr>
<td>Other</td>
<td>12</td>
@@ -2075,19 +1799,14 @@ managed services (14.61% in 2019).</p>
</tr>
</tbody>
</table>
-
<h2 id="summary">Summary</h2>
-
<p>From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and
are willing to upgrade to new ones.</p>
-
<p>Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow
deployments. However, we hope that with the increase of adoption there will be an increase in people willing
to share their experience and tools.</p>
-
<h2 id="data">Data</h2>
-
<p>If you think I missed something or you simply want to look for insights on your own, the data is available for you here: (Airflow User Survey 2020.csv)[/data/survey-responses/airflow-user-survey-responses-2020.csv.zip]</p>
</div>
@@ -2432,7 +2151,7 @@ to share their experience and tools.</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-survey-2022/index.html b/blog/airflow-survey-2022/index.html
index 223f7de61c..7f801ba41c 100644
--- a/blog/airflow-survey-2022/index.html
+++ b/blog/airflow-survey-2022/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Airflow Survey 2022 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2022" />
<meta property="og:description" content="2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey-2022/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-survey-2022/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2022-06-17T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Airflow Survey 2022">
-<meta itemprop="description" content="2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward.">
-
-
-<meta itemprop="datePublished" content="2022-06-17T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="3865">
-
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="keywords" content="community,survey,users," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Airflow Survey 2022">
+<meta itemprop="description" content="2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward."><meta itemprop="datePublished" content="2022-06-17T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="3865"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="community,survey,users," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Airflow Survey 2022"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -488,63 +480,44 @@
<p class="blogpost-content__metadata--description">2021 saw rapid adoption of Airflow 2, and continued growth of the community. This annual survey helps us understand how people are using Airflow, and where we can best focus our efforts going forward.</p>
</div>
<div class="markdown-content">
-
-
-<h1 id="airflow-user-survey-2022">Airflow User Survey 2022</h1>
-
+ <h1 id="airflow-user-survey-2022">Airflow User Survey 2022</h1>
<p>This yearโs survey has come and gone, and with it weโve got a new batch of data for everyone! We collected 210 responses over two weeks. We continue to see growth in both contributions and downloads over the last two years, and expect that trend will continue through 2022.</p>
-
-<p>The raw response data will be made available here soon, in the meantime, feel free to email john.thomas@astronomer.io for a copy.</p>
-
-<h2 id="tl-dr">TL;DR</h2>
-
+<p>The raw response data will be made available here soon, in the meantime, feel free to email <a href="mailto:john.thomas@astronomer.io">john.thomas@astronomer.io</a> for a copy.</p>
+<h2 id="tldr">TL;DR</h2>
<h3 id="overview-of-the-user">Overview of the user</h3>
-
<ul>
-<li>Like previous years, more than half of the Airflow users are Data Engineers (54%). Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%) are also active Airflow users! There was a slight increase in the representation of Solutions Architect roles compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank"> [...]
-<li>Airflow is used and popular in bigger companies, 64% of Airflow users work for companies with 200+ employees which is an 11 percent increase compared to <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>.</li>
+<li>Like previous years, more than half of the Airflow users are Data Engineers (54%). Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%) are also active Airflow users! There was a slight increase in the representation of Solutions Architect roles compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a> .</li>
+<li>Airflow is used and popular in bigger companies, 64% of Airflow users work for companies with 200+ employees which is an 11 percent increase compared to <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>.</li>
<li>62% of the survey participants have more than 6 Airflow users in their company.</li>
-<li>More Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow ( 85.7% in <a href="https://airflow.apache.org/blog/ai [...]
-<li>Airflow documentation is a critical source of information, with more than 90% (15% increase compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>) of survey participants using the documentation. Airflow documentation is also one of the top areas to improve! Whatโs interesting, also Stack Overflow usage is critical, with about 60% users declaring to use it as a source of information (24% increase compared [...]
+<li>More Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow ( 85.7% in <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a> and 92% [...]
+<li>Airflow documentation is a critical source of information, with more than 90% (15% increase compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>) of survey participants using the documentation. Airflow documentation is also one of the top areas to improve! Whatโs interesting, also Stack Overflow usage is critical, with about 60% users declaring to use it as a source of information (24% increase compared to results from [...]
</ul>
-
<h3 id="deployments">Deployments</h3>
-
<ul>
<li>85% of the Airflow users have between 1 to 7 active Airflow instances. 62.5% of the Airflow users have between 11 to 250 DAGs in their largest Airflow instance. 75% of the surveyed Airflow users have between 1 to 100 tasks per DAG.</li>
<li>Close to 85% of users use one of the Airflow 2 versions, 9.2% users still use 1.10.15, while the remaining 6.3% are still using olderAirflow 1 versions. The good news is that the majority of users on Airflow 1 are planning migration to Airflow 2 quite soon, with resources and capacity being the main blockers.</li>
-<li>In comparison to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>, more users were interested in monitoring in general and specifically in using tools such as external monitoring services (40.7%, up from 29.6%) and information from metabase (35.7%, up from 25.1%).</li>
+<li>In comparison to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>, more users were interested in monitoring in general and specifically in using tools such as external monitoring services (40.7%, up from 29.6%) and information from metabase (35.7%, up from 25.1%).</li>
<li>Celery (52.7%) and Kubernetes (39.4%) are the most common executors used.</li>
</ul>
-
<h3 id="usage">Usage</h3>
-
<ul>
<li>81.3% of Airflow users who responded to the survey donโt have any customisation of Airflow.</li>
<li>Xcom (69.8%) is the most popular method to pass inputs and outputs between tasks, however Saving and Retrieving Inputs and Outputs from Storage still plays an important role (49%).</li>
<li>Lineage itself is a quite new topic for Airflow users, most of them donโt use lineage solutions but might be interested if supported by Airflow (47.5%), are not familiar with data lineage (29%) or that data lineage is not their concern (13%).</li>
-<li>The Airflow web UI is used heavily for Monitoring Runs (95.9%), Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%). The top 3 views used are: List of DAGs, Task Logs and DAG Runs, which is very similar to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">2019</a>.</li>
+<li>The Airflow web UI is used heavily for Monitoring Runs (95.9%), Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%). The top 3 views used are: List of DAGs, Task Logs and DAG Runs, which is very similar to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>.</li>
</ul>
-
<h3 id="community-and-contribution">Community and contribution</h3>
-
<ul>
<li>Most Airflow users (57.1%) are aware they could contribute but do not, and an additional 21.7% contribute very rarely. 14.8% of users were not aware they could contribute. There is much more to be done to engage our community to be more active contributors and raise the current 6.4% of users who actively contribute, especially considering that one important blocker for contribution is lack of knowledge on how to start (37.7%).</li>
</ul>
-
<h3 id="the-future-of-airflow">The future of Airflow</h3>
-
<ul>
<li>The top area for improvement is still the Airflow web UI (49.5%), closely followed by more telemetry for logging, monitoring and alerting purposes (48%). However all those efforts should go in line with improved documentation (36.6.%) and resources about using the Airflow, especially when we take into account the need of onboarding new users (36.6%).</li>
<li>DAG Versioning(66.2%) is a winner for new features in Airflow, and itโs not a surprise as this feature may positively impact daily work of Airflow users. It is followed by three other ideas: Dependency management and Data-driven scheduling (42.6%), More dynamic task structure (42.1%) and Multi-Tenancy (37.9%).</li>
</ul>
-
<h2 id="overview-of-the-user-1">Overview of the user</h2>
-
<h3 id="what-best-describes-your-current-occupation-single-choice">What best describes your current occupation? (single choice)</h3>
-
-<p><img src="images/image1.png" alt="alt_text" title="user_occupations" /></p>
-
+<p><img src="images/image1.png" alt="alt_text" title="user_occupations"></p>
<table>
<thead>
<tr>
@@ -553,62 +526,52 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Data Engineer</td>
<td>114</td>
<td>54%</td>
</tr>
-
<tr>
<td>Solutions Architect</td>
<td>27</td>
<td>13%</td>
</tr>
-
<tr>
<td>Developer</td>
<td>25</td>
<td>12%</td>
</tr>
-
<tr>
<td>DevOps</td>
<td>12</td>
<td>6%</td>
</tr>
-
<tr>
<td>Data Scientist</td>
<td>8</td>
<td>4%</td>
</tr>
-
<tr>
<td>Support Engineer</td>
<td>5</td>
<td>2%</td>
</tr>
-
<tr>
<td>Data Analyst</td>
<td>3</td>
<td>1%</td>
</tr>
-
<tr>
<td>Business Analyst</td>
<td>2</td>
<td>1%</td>
</tr>
-
<tr>
<td>Other</td>
<td>14</td>
@@ -616,13 +579,9 @@
</tr>
</tbody>
</table>
-
-<p>According to the survey, more than half of Airflow users are Data Engineers (54%). Roles of the remaining Airflow users might be broken down into Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%). The 2022 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> and <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> with a slight increa [...]
-
+<p>According to the survey, more than half of Airflow users are Data Engineers (54%). Roles of the remaining Airflow users might be broken down into Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%). The 2022 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/">those from 2019</a> and <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> with a slight increase in the representation of Solu [...]
<h3 id="how-often-do-you-interact-with-airflow-single-choice">How often do you interact with Airflow? (single choice)</h3>
-
-<p><img src="images/image2.png" alt="alt_text" title="interaction_frequency" /></p>
-
+<p><img src="images/image2.png" alt="alt_text" title="interaction_frequency"></p>
<table>
<thead>
<tr>
@@ -631,32 +590,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Every day</td>
<td>154</td>
<td>73%</td>
</tr>
-
<tr>
<td>At least once per week</td>
<td>36</td>
<td>17%</td>
</tr>
-
<tr>
<td>At least once per month</td>
<td>11</td>
<td>5%</td>
</tr>
-
<tr>
<td>Less than once per month</td>
<td>9</td>
@@ -664,13 +618,9 @@
</tr>
</tbody>
</table>
-
<p>Users who took the survey are actively using Airflow as part of their current role. 73% of Airflow users who responded use it on a daily basis, 17% weekly.</p>
-
<h3 id="how-many-people-work-at-your-company-single-choice">How many people work at your company? (single choice)</h3>
-
-<p><img src="images/image3.png" alt="alt_text" title="company_size" /></p>
-
+<p><img src="images/image3.png" alt="alt_text" title="company_size"></p>
<table>
<thead>
<tr>
@@ -679,38 +629,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>201-5000</td>
<td>85</td>
<td>41%</td>
</tr>
-
<tr>
<td>5000+</td>
<td>49</td>
<td>23%</td>
</tr>
-
<tr>
<td>51-200</td>
<td>46</td>
<td>22%</td>
</tr>
-
<tr>
<td>11-50</td>
<td>20</td>
<td>10%</td>
</tr>
-
<tr>
<td>1-10</td>
<td>9</td>
@@ -718,13 +662,9 @@
</tr>
</tbody>
</table>
-
-<p>Airflow is a framework that is used and popular in bigger companies, 64% of Airflow users who responded (compared to 52.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>) work for companies bigger than 200 employees (41% in companies size 201-5000 and 23% in companies size 5000+).</p>
-
+<p>Airflow is a framework that is used and popular in bigger companies, 64% of Airflow users who responded (compared to 52.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>) work for companies bigger than 200 employees (41% in companies size 201-5000 and 23% in companies size 5000+).</p>
<h3 id="how-many-people-at-your-company-use-airflow-single-choice">How many people at your company use Airflow? (single choice)</h3>
-
-<p><img src="images/image4.png" alt="alt_text" title="airflow_usage" /></p>
-
+<p><img src="images/image4.png" alt="alt_text" title="airflow_usage"></p>
<table>
<thead>
<tr>
@@ -733,32 +673,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>6-20</td>
<td>80</td>
<td>38%</td>
</tr>
-
<tr>
<td>1-5</td>
<td>61</td>
<td>29%</td>
</tr>
-
<tr>
<td>51-200</td>
<td>49</td>
<td>24%</td>
</tr>
-
<tr>
<td>200+</td>
<td>18</td>
@@ -766,11 +701,8 @@
</tr>
</tbody>
</table>
-
<p>Airflow is generally used by small to medium-sized teams. 62% of the survey participants have more than 6 Airflow users in their company (38% have between 6 and 200 users, 24% between 51-200 users).</p>
-
<h3 id="how-likely-are-you-to-recommend-apache-airflow-single-choice">How likely are you to recommend Apache Airflow? (single choice)</h3>
-
<table>
<thead>
<tr>
@@ -780,7 +712,6 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
@@ -788,35 +719,30 @@
<td>% 2020</td>
<td>% 2022</td>
</tr>
-
<tr>
<td>Very Likely</td>
<td>45.4%</td>
<td>61.6%</td>
<td>65.9%</td>
</tr>
-
<tr>
<td>Likely</td>
<td>40.3%</td>
<td>30.4%</td>
<td>26.9%</td>
</tr>
-
<tr>
<td>Neutral</td>
<td>10.7%</td>
<td>5.4%</td>
<td>6.3%</td>
</tr>
-
<tr>
<td>Unlikely</td>
<td>2.6%</td>
<td>1.5%</td>
<td>0.5%</td>
</tr>
-
<tr>
<td>Very Unlikely</td>
<td>1%</td>
@@ -825,11 +751,8 @@
</tr>
</tbody>
</table>
-
-<p>According to the survey, more Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow (92% in <a href="https://airfl [...]
-
+<p>According to the survey, more Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow (92% in <a href="https://airflow.apache.org/blog/airflow-surve [...]
<h3 id="what-is-your-source-of-information-about-airflow-multiple-choice">What is your source of information about Airflow? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -838,68 +761,57 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Documentation</td>
<td>189</td>
<td>90.4%</td>
</tr>
-
<tr>
<td>Airflow website (Blog, etc.)</td>
<td>142</td>
<td>67.9%</td>
</tr>
-
<tr>
<td>Stack Overflow</td>
<td>126</td>
<td>60.3%</td>
</tr>
-
<tr>
<td>Github Issues</td>
<td>104</td>
<td>49.8%</td>
</tr>
-
<tr>
<td>Slack</td>
<td>96</td>
<td>45.9%</td>
</tr>
-
<tr>
<td>Airflow Summit Videos</td>
<td>88</td>
<td>42.1%</td>
</tr>
-
<tr>
<td>GitHub Discussions</td>
<td>76</td>
<td>36.4%</td>
</tr>
-
<tr>
<td>Airflow Community Webinars</td>
<td>41</td>
<td>19.6%</td>
</tr>
-
<tr>
<td>Astronomer Registry</td>
<td>51</td>
<td>24.4%</td>
</tr>
-
<tr>
<td>Airflow Mailing List</td>
<td>34</td>
@@ -907,15 +819,10 @@
</tr>
</tbody>
</table>
-
-<p>Airflow documentation is a critical source of information, with more than 90% of survey participants using the documentation. It is of increasing importance compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> where documentation was at about 75% level. Moreover, more than 60% of users are getting information from the Airflow website (67.9% ) and Stack Overflow (60.3%) which is also a big increase compare [...]
-
+<p>Airflow documentation is a critical source of information, with more than 90% of survey participants using the documentation. It is of increasing importance compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> where documentation was at about 75% level. Moreover, more than 60% of users are getting information from the Airflow website (67.9% ) and Stack Overflow (60.3%) which is also a big increase compared to 36% level i [...]
<h2 id="deployments-1">Deployments</h2>
-
<h3 id="how-many-active-dags-do-you-have-in-your-largest-airflow-instance-single-choice">How many active DAGs do you have in your largest Airflow instance? (single choice)</h3>
-
-<p><img src="images/image5.png" alt="alt_text" title="active_dags" /></p>
-
+<p><img src="images/image5.png" alt="alt_text" title="active_dags"></p>
<table>
<thead>
<tr>
@@ -924,50 +831,42 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>51-250</td>
<td>66</td>
<td>31.7%</td>
</tr>
-
<tr>
<td>11-50</td>
<td>64</td>
<td>30.8%</td>
</tr>
-
<tr>
<td>5-10</td>
<td>25</td>
<td>12.0%</td>
</tr>
-
<tr>
<td>251-500</td>
<td>20</td>
<td>9.6%</td>
</tr>
-
<tr>
<td><5</td>
<td>14</td>
<td>6.7%</td>
</tr>
-
<tr>
<td>1000+</td>
<td>10</td>
<td>4.8%</td>
</tr>
-
<tr>
<td>501-1000</td>
<td>9</td>
@@ -975,13 +874,9 @@
</tr>
</tbody>
</table>
-
<p>62.5% of the Airflow users surveyed have between 11 to 250 DAGs in their largest Airflow instance.</p>
-
<h3 id="how-many-active-airflow-instances-do-you-have-single-choice">How many active Airflow instances do you have? (single choice)</h3>
-
-<p><img src="images/image6.png" alt="alt_text" title="image_tooltip" /></p>
-
+<p><img src="images/image6.png" alt="alt_text" title="image_tooltip"></p>
<table>
<thead>
<tr>
@@ -990,50 +885,42 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>1</td>
<td>52</td>
<td>25.2%</td>
</tr>
-
<tr>
<td>2</td>
<td>46</td>
<td>22.3%</td>
</tr>
-
<tr>
<td>4-7</td>
<td>40</td>
<td>19.4%</td>
</tr>
-
<tr>
<td>3</td>
<td>37</td>
<td>18.0%</td>
</tr>
-
<tr>
<td>20+</td>
<td>19</td>
<td>9.2%</td>
</tr>
-
<tr>
<td>8-10</td>
<td>7</td>
<td>3.4%</td>
</tr>
-
<tr>
<td>11-20</td>
<td>5</td>
@@ -1041,13 +928,9 @@
</tr>
</tbody>
</table>
-
<p>85% of the Airflow users surveyed have between 1 and 7 active Airflow instances, and nearly 50% have only 1 or 2.</p>
-
-<h3 id="what-is-the-maximum-number-of-tasks-that-you-have-used-in-a-single-dag-single-choice">What is the maximum number of tasks that you have used in a single DAG?(single choice)</h3>
-
-<p><img src="images/image7.png" alt="alt_text" title="maximum tasks" /></p>
-
+<h3 id="what-is-the-maximum-number-of-tasks-that-you-have-used-in-a-single-dagsingle-choice">What is the maximum number of tasks that you have used in a single DAG?(single choice)</h3>
+<p><img src="images/image7.png" alt="alt_text" title="maximum tasks"></p>
<table>
<thead>
<tr>
@@ -1056,62 +939,52 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>11-25</td>
<td>51</td>
<td>24.5%</td>
</tr>
-
<tr>
<td>26-50</td>
<td>41</td>
<td>19.7%</td>
</tr>
-
<tr>
<td>51-100</td>
<td>35</td>
<td>16.8%</td>
</tr>
-
<tr>
<td><10</td>
<td>29</td>
<td>13.9%</td>
</tr>
-
<tr>
<td>101-250</td>
<td>23</td>
<td>11.1%</td>
</tr>
-
<tr>
<td>501-1000</td>
<td>9</td>
<td>4.3%</td>
</tr>
-
<tr>
<td>1000-2500</td>
<td>8</td>
<td>3.8%</td>
</tr>
-
<tr>
<td>251-500</td>
<td>8</td>
<td>3.8%</td>
</tr>
-
<tr>
<td>2500-5000</td>
<td>4</td>
@@ -1119,13 +992,9 @@
</tr>
</tbody>
</table>
-
<p>75% of the surveyed Airflow users have between 1 and 100 tasks per DAG.</p>
-
<h3 id="how-many-schedulers-do-you-have-in-your-largest-airflow-instance-single-choice">How many schedulers do you have in your largest Airflow instance? (single choice)</h3>
-
-<p><img src="images/image8.png" alt="alt_text" title="max_schedulers" /></p>
-
+<p><img src="images/image8.png" alt="alt_text" title="max_schedulers"></p>
<table>
<thead>
<tr>
@@ -1134,32 +1003,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>1</td>
<td>113</td>
<td>55.1%</td>
</tr>
-
<tr>
<td>2</td>
<td>61</td>
<td>29.8%</td>
</tr>
-
<tr>
<td>3</td>
<td>18</td>
<td>8.8%</td>
</tr>
-
<tr>
<td>4+</td>
<td>13</td>
@@ -1167,11 +1031,8 @@
</tr>
</tbody>
</table>
-
<p>More than half of Airflow users who responded to the survey have 1 scheduler in their largest Airflow instance, however itโs important to notice that the second half of Airflow users decided to have 2 schedulers and more.</p>
-
<h3 id="what-executor-type-do-you-use-multiple-choice">What executor type do you use? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1180,38 +1041,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Celery</td>
<td>107</td>
<td>52.7 %</td>
</tr>
-
<tr>
<td>Kubernetes</td>
<td>80</td>
<td>39.4%</td>
</tr>
-
<tr>
<td>Local</td>
<td>49</td>
<td>24.1%</td>
</tr>
-
<tr>
<td>Sequential</td>
<td>21</td>
<td>10.3%</td>
</tr>
-
<tr>
<td>CeleryKubernetes</td>
<td>14</td>
@@ -1219,13 +1074,9 @@
</tr>
</tbody>
</table>
-
<p>Celery (52.7%) and Kubernetes (39.4%) are the most common executors used. CeleryKubernetes (6.9%) executor also started to be noticed and used by Airflow users.</p>
-
<h3 id="if-you-use-the-celery-executor-how-many-workers-do-you-have-in-your-largest-airflow-instance-single-choice">If you use the Celery executor, how many workers do you have in your largest Airflow instance? (single choice)</h3>
-
-<p><img src="images/image9.png" alt="alt_text" title="max_workers" /></p>
-
+<p><img src="images/image9.png" alt="alt_text" title="max_workers"></p>
<table>
<thead>
<tr>
@@ -1234,32 +1085,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>2-5</td>
<td>64</td>
<td>44.8%</td>
</tr>
-
<tr>
<td>10+</td>
<td>28</td>
<td>19.6%</td>
</tr>
-
<tr>
<td>1</td>
<td>26</td>
<td>18.2%</td>
</tr>
-
<tr>
<td>6-10</td>
<td>25</td>
@@ -1267,13 +1113,9 @@
</tr>
</tbody>
</table>
-
<p>Amongst Celery executor users who responded to the survey, close to half the number (44.8%) have between 2 to 5 workers in their largest Airflow instance. Itโs notable that nearly a fifth (19.6%) have more than 10 workers.</p>
-
<h3 id="which-version-of-airflow-do-you-currently-use-single-choice">Which version of Airflow do you currently use? (single choice)</h3>
-
-<p><img src="images/image10.png" alt="alt_text" title="airflow_version" /></p>
-
+<p><img src="images/image10.png" alt="alt_text" title="airflow_version"></p>
<table>
<thead>
<tr>
@@ -1282,44 +1124,37 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>1.10.14 or older</td>
<td>13</td>
<td>6.3%</td>
</tr>
-
<tr>
<td>1.10.15</td>
<td>19</td>
<td>9.2%</td>
</tr>
-
<tr>
<td>2.0.x</td>
<td>23</td>
<td>11.1%</td>
</tr>
-
<tr>
<td>2.1.x</td>
<td>24</td>
<td>11.6%</td>
</tr>
-
<tr>
<td>2.2.x</td>
<td>79</td>
<td>38.2%</td>
</tr>
-
<tr>
<td>2.3.x</td>
<td>49</td>
@@ -1327,15 +1162,10 @@
</tr>
</tbody>
</table>
-
<p>It’s good to see that close to 85% of users who responded to the survey use one of the Airflow 2 versions, 9.2% users still use 1.10.15, while the remaining 6.3% are still using older Airflow 1.10 versions.</p>
-
<p>The good news is that the majority of users on Airflow 1 are planning migration to Airflow 2 quite soon, as for now they have capacity constraints to undertake such a significant effort in their opinion. However, it can also be noticed in the surveyโs comments that some users are generally skeptical towards migration to Airflow 2, they have negative opinions about the new scheduler or compatibility with the helm chart.</p>
-
<p>As to plans about migration to the newest version of Airflow 2, users who responded to the survey are committed and waiting especially for the features related to dynamic DAGs. However, some users also reported that they are waiting to solve some dependencies they have or they prefer to wait a little bit more for the community to test the new version before they decide to move on.</p>
-
<h3 id="what-metrics-do-you-use-to-monitor-airflow-multiple-choice">What metrics do you use to monitor Airflow? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1344,38 +1174,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>External monitoring service</td>
<td>81</td>
<td>40.7%</td>
</tr>
-
<tr>
<td>Information from metadatabase</td>
<td>71</td>
<td>35.7%</td>
</tr>
-
<tr>
<td>Statsd</td>
<td>54</td>
<td>27.1%</td>
</tr>
-
<tr>
<td>I do not use monitoring</td>
<td>47</td>
<td>23.6%</td>
</tr>
-
<tr>
<td>Other</td>
<td>14</td>
@@ -1383,11 +1207,8 @@
</tr>
</tbody>
</table>
-
-<p>In comparison to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>, more users are monitoring airflow in some way. External monitoring services (40.7%) and information from metabase (35.7%) started to play a more important role in Airflow monitoring.</p>
-
+<p>In comparison to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>, more users are monitoring airflow in some way. External monitoring services (40.7%) and information from metabase (35.7%) started to play a more important role in Airflow monitoring.</p>
<h3 id="how-do-you-deploy-airflow-multiple-choice">How do you deploy Airflow? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1396,50 +1217,42 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>On virtual machines (for example using AWS EC2)</td>
<td>63</td>
<td>30.6 %</td>
</tr>
-
<tr>
<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
<td>54</td>
<td>26.2 %</td>
</tr>
-
<tr>
<td>On Kubernetes (using Apache Airflowโs helm chart)</td>
<td>46</td>
<td>22.3%</td>
</tr>
-
<tr>
<td>On premises</td>
<td>43</td>
<td>20.9%</td>
</tr>
-
<tr>
<td>On Kubernetes (using custom deployments)</td>
<td>39</td>
<td>18.9%</td>
</tr>
-
<tr>
<td>On Kubernetes (using another helm chart)</td>
<td>21</td>
<td>10.2%</td>
</tr>
-
<tr>
<td>Other</td>
<td>13</td>
@@ -1447,11 +1260,8 @@
</tr>
</tbody>
</table>
-
-<p>More than half of Airflow users who responded (51.4%) deploy Airflow on Kubernetes. This is about 20 percent more than in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>. The remaining top deployment methods are on virtual machines (30.6%) and via managed services (26.2%).</p>
-
+<p>More than half of Airflow users who responded (51.4%) deploy Airflow on Kubernetes. This is about 20 percent more than in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>. The remaining top deployment methods are on virtual machines (30.6%) and via managed services (26.2%).</p>
<h3 id="how-do-you-distribute-your-dags-from-your-developer-environment-to-the-cloud-single-choice">How do you distribute your DAGs from your developer environment to the cloud? (single choice)</h3>
-
<table>
<thead>
<tr>
@@ -1460,38 +1270,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
<td>100</td>
<td>49%</td>
</tr>
-
<tr>
<td>Bake them into the docker image</td>
<td>51</td>
<td>25%</td>
</tr>
-
<tr>
<td>Shared files system</td>
<td>30</td>
<td>14.7%</td>
</tr>
-
<tr>
<td>Other</td>
<td>16</td>
<td>7.9%</td>
</tr>
-
<tr>
<td>I donโt know</td>
<td>7</td>
@@ -1499,15 +1303,10 @@
</tr>
</tbody>
</table>
-
<p>According to the survey responses, the most popular way of distributing DAGs is a synchronizing process, about half of Airflow users (49%) use this process to distribute DAGs from developer environments to the cloud.</p>
-
<h2 id="usage-1">Usage</h2>
-
<h3 id="do-you-have-any-customisation-of-airflow-single-choice">Do you have any customisation of Airflow? (single choice)</h3>
-
-<p><img src="images/image11.png" alt="alt_text" title="customization" /></p>
-
+<p><img src="images/image11.png" alt="alt_text" title="customization"></p>
<table>
<thead>
<tr>
@@ -1516,32 +1315,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>No, we use vanilla airflow</td>
<td>165</td>
<td>81.3%</td>
</tr>
-
<tr>
<td>Yes, we have a separate fork</td>
<td>13</td>
<td>6.4%</td>
</tr>
-
<tr>
<td>Yes, we use a 3rd-party fork</td>
<td>12</td>
<td>5.9%</td>
</tr>
-
<tr>
<td>Yes, weโve backpropagated bug fixes to an older version</td>
<td>13</td>
@@ -1549,13 +1343,9 @@
</tr>
</tbody>
</table>
-
-<p>More Airflow users (81.3%) donโt have any customisation of Airflow (compared to 75.9% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>). Those Airflow users who have customisations (18.7%) decided to introduce them mainly to separate development and production workflows, to backport bug fixes, due to security fixes or to run a backfill command on Kubernetes pod.</p>
-
+<p>More Airflow users (81.3%) donโt have any customisation of Airflow (compared to 75.9% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>). Those Airflow users who have customisations (18.7%) decided to introduce them mainly to separate development and production workflows, to backport bug fixes, due to security fixes or to run a backfill command on Kubernetes pod.</p>
<h3 id="which-metadata-database-do-you-use-single-choice">Which Metadata Database do you use? (single choice)</h3>
-
-<p><img src="images/image12.png" alt="alt_text" title="database" /></p>
-
+<p><img src="images/image12.png" alt="alt_text" title="database"></p>
<table>
<thead>
<tr>
@@ -1564,44 +1354,37 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%I</td>
</tr>
-
<tr>
<td>PostgreSQL 13</td>
<td>86</td>
<td>43.9%</td>
</tr>
-
<tr>
<td>PostgreSQL 12</td>
<td>74</td>
<td>37.8%</td>
</tr>
-
<tr>
<td>MySQL 8</td>
<td>22</td>
<td>11.2%</td>
</tr>
-
<tr>
<td>MySQL 5</td>
<td>9</td>
<td>4.6%</td>
</tr>
-
<tr>
<td>MariaDB</td>
<td>4</td>
<td>2.0%</td>
</tr>
-
<tr>
<td>MsSQL</td>
<td>1</td>
@@ -1609,13 +1392,9 @@
</tr>
</tbody>
</table>
-
<p>According to the survey responses, the most popular metadata databases are PostgreSQL 13 (43.9%) and PostgreSQL 12 (37.8%). This represents a sharp increase from 2020, up from 68.9% to 81.7% total on PostgreSQL, with a corresponding decrease in MySQL, down from 23% to 15%. This is an interesting result taking into account community discussion about not adding support for more database backend or even deciding on single database support.</p>
-
-<h3 id="what-s-the-primary-method-by-which-you-integrate-with-providers-and-external-services-in-your-airflow-dags-single-choice">What’s the primary method by which you integrate with providers and external services in your Airflow DAGs? (single choice)</h3>
-
-<p><img src="images/image13.png" alt="alt_text" title="providers_interface" /></p>
-
+<h3 id="whats-the-primary-method-by-which-you-integrate-with-providers-and-external-services-in-your-airflow-dags-single-choice">What’s the primary method by which you integrate with providers and external services in your Airflow DAGs? (single choice)</h3>
+<p><img src="images/image13.png" alt="alt_text" title="providers_interface"></p>
<table>
<thead>
<tr>
@@ -1624,32 +1403,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Using existing dedicated operators / hooks</td>
<td>70</td>
<td>34.5%</td>
</tr>
-
<tr>
<td>Using Bash/Python operators</td>
<td>58</td>
<td>28.6%</td>
</tr>
-
<tr>
<td>Using custom operators / hooks</td>
<td>50</td>
<td>24.6%</td>
</tr>
-
<tr>
<td>Using KubernetesPodOperator</td>
<td>25</td>
@@ -1657,11 +1431,8 @@
</tr>
</tbody>
</table>
-
-<p>According to the survey responses, the following ways of using Airflow to connect to external services are the most popular: Using existing dedicated operators / hooks (34.5%), Using Bash/Python operators (28.6%), Using custom operators / hooks (24.6%). Using KubernetesPodOperator (12.3%) is less popular regarding the survey responses. The integration with providers and external services methods ranking is similar to the one from <a href="https://airflow.apache.org/blog/airflow-survey [...]
-
+<p>According to the survey responses, the following ways of using Airflow to connect to external services are the most popular: Using existing dedicated operators / hooks (34.5%), Using Bash/Python operators (28.6%), Using custom operators / hooks (24.6%). Using KubernetesPodOperator (12.3%) is less popular regarding the survey responses. The integration with providers and external services methods ranking is similar to the one from <a href="https://airflow.apache.org/blog/airflow-survey [...]
<h3 id="what-providers-do-you-use-in-your-airflow-dags-multiple-choice">What providers do you use in your Airflow DAGs? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1670,50 +1441,42 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Amazon Web Services</td>
<td>112</td>
<td>55.4%</td>
</tr>
-
<tr>
<td>Google Cloud Platform / Google APIs</td>
<td>79</td>
<td>39.1%</td>
</tr>
-
<tr>
<td>Internal company systems</td>
<td>75</td>
<td>37.1%</td>
</tr>
-
<tr>
<td>Hadoop / Spark / Flink / Other Apache software</td>
<td>57</td>
<td>28.2%</td>
</tr>
-
<tr>
<td>Microsoft Azure</td>
<td>17</td>
<td>8.4%</td>
</tr>
-
<tr>
<td>Other</td>
<td>21</td>
<td>10.5%</td>
</tr>
-
<tr>
<td>I do not use external services in my Airflow DAGs</td>
<td>14</td>
@@ -1721,13 +1484,9 @@
</tr>
</tbody>
</table>
-
-<p>Itโs not surprising that Amazon Web Services (55.4% vs 59.6% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/" target="_blank">2020</a>), on the next three positions Google Cloud Platform (39.1% vs 47.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/" target="_blank">2020</a> ), Internal company systems (37.1% vs 55.6% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/" target="_blank">2020</a>), and other Apache products (28.2% vs 35.47 [...]
-
+<p>Itโs not surprising that Amazon Web Services (55.4% vs 59.6% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/">2020</a>), on the next three positions Google Cloud Platform (39.1% vs 47.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/">2020</a> ), Internal company systems (37.1% vs 55.6% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/">2020</a>), and other Apache products (28.2% vs 35.47% in <a href="https://airflow.apache.org/blog/ai [...]
<h3 id="how-frequently-do-you-upgrade-airflow-environments-single-choice">How frequently do you upgrade Airflow environments? (single choice)</h3>
-
-<p><img src="images/image14.png" alt="alt_text" title="upgrade_frequency" /></p>
-
+<p><img src="images/image14.png" alt="alt_text" title="upgrade_frequency"></p>
<table>
<thead>
<tr>
@@ -1736,32 +1495,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>every 12 months</td>
<td>46</td>
<td>22.9%</td>
</tr>
-
<tr>
<td>every 6 months</td>
<td>49</td>
<td>24.4%</td>
</tr>
-
<tr>
<td>once a quarter</td>
<td>47</td>
<td>23.4%</td>
</tr>
-
<tr>
<td>Whenever there is a newer version</td>
<td>59</td>
@@ -1769,13 +1523,9 @@
</tr>
</tbody>
</table>
-
<p>Different frequencies of Airflow environments upgrades are almost equally popular amongst Airflow users who responded to the survey.</p>
-
<h3 id="do-you-upgrade-providers-separately-from-the-core-single-choice">Do you upgrade providers separately from the core? (single choice)</h3>
-
-<p><img src="images/image15.png" alt="alt_text" title="providers_upgrade" /></p>
-
+<p><img src="images/image15.png" alt="alt_text" title="providers_upgrade"></p>
<table>
<thead>
<tr>
@@ -1784,32 +1534,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>When I need it</td>
<td>83</td>
<td>42.8%</td>
</tr>
-
<tr>
<td>Never - always use the providers that come with Airflow</td>
<td>68</td>
<td>35.1%</td>
</tr>
-
<tr>
<td>I did not know I can upgrade providers separately</td>
<td>32</td>
<td>16.5%</td>
</tr>
-
<tr>
<td>I upgrade providers when they are released</td>
<td>11</td>
@@ -1817,11 +1562,8 @@
</tr>
</tbody>
</table>
-
<p>According to the survey responses, Airflow users most often upgrade providers when they need it (42.8%) or prefer to stay with providers that come with Airflow (35.1%). Itโs surprising that 16.5% of Airflow users who responded to the survey were not aware that they can upgrade their providers separately from the core Airflow.</p>
-
<h3 id="how-do-you-pass-inputs-and-outputs-between-tasks-multiple-choice">How do you pass inputs and outputs between tasks? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1830,38 +1572,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Xcom</td>
<td>141</td>
<td>69.8%</td>
</tr>
-
<tr>
<td>Saving and retrieving from Storage</td>
<td>99</td>
<td>49%</td>
</tr>
-
<tr>
<td>TaskFlow</td>
<td>37</td>
<td>18.3%</td>
</tr>
-
<tr>
<td>Other</td>
<td>5</td>
<td>2.5%</td>
</tr>
-
<tr>
<td>We donโt</td>
<td>29</td>
@@ -1869,11 +1605,8 @@
</tr>
</tbody>
</table>
-
<p>According to the survey responses, Xcom (69.8%) is the most popular method to pass inputs and outputs between tasks, however Saving and Retrieving Inputs and Outputs from Storage still plays an important role (49%). Itโs interesting that close to 15% of Airflow users who responded to the survey declare to not pass any outputs or inputs between tasks.</p>
-
<h3 id="do-you-use-a-data-lineage-backend-multiple-choice">Do you use a data lineage backend? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1882,44 +1615,37 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>No, but I will use such feature if fully supported in Airflow</td>
<td>95</td>
<td>47.5%</td>
</tr>
-
<tr>
<td>Iโm not familiar with data lineage</td>
<td>58</td>
<td>29%</td>
</tr>
-
<tr>
<td>No, data lineage isnโt a concern for my usage</td>
<td>26</td>
<td>13%</td>
</tr>
-
<tr>
<td>Yes, I send lineage to an Open Source lineage repository</td>
<td>15</td>
<td>7.5%</td>
</tr>
-
<tr>
<td>Yes, I send lineage to an Enterprise lineage repository</td>
<td>7</td>
<td>3.5%</td>
</tr>
-
<tr>
<td>Yes, I send lineage to a custom internal lineage repository</td>
<td>9</td>
@@ -1927,11 +1653,8 @@
</tr>
</tbody>
</table>
-
<p>When asked what lineage backend Airflow users use, the answers indicated that, while lineage itself is a quite new topic, there is interest in the feature as a whole. Most Airflow users responded that they donโt use lineage solutions currently but might be interested in the future if supported by Airflow (47.5%), are not familiar with data lineage (29%) or that data lineage is not their concern (13%).</p>
-
<h3 id="which-interfaces-of-airflow-do-you-use-as-part-of-your-current-role-multiple-choice">Which interfaces of Airflow do you use as part of your current role? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1940,38 +1663,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Original Airflow Graphical User Interface</td>
<td>189</td>
<td>94%</td>
</tr>
-
<tr>
<td>CLI</td>
<td>98</td>
<td>48.8%</td>
</tr>
-
<tr>
<td>API</td>
<td>80</td>
<td>39.8%</td>
</tr>
-
<tr>
<td>Custom (own created) Airflow Graphical User Interface</td>
<td>12</td>
<td>6%</td>
</tr>
-
<tr>
<td>GCP Composer</td>
<td>1</td>
@@ -1979,11 +1696,8 @@
</tr>
</tbody>
</table>
-
<p>Itโs clear that usage of Airflow web UI is important as 94% of users who responded to the survey declare to use it as a part of their current role. Usage of CLI (48.8%) and API (39.8%) goes in pairs but are not so common compared to Airflow web UI usage.</p>
-
<h3 id="if-gui-marked-what-do-you-use-the-gui-for-multiple-choice">(If GUI Marked) What do you use the GUI for? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -1992,44 +1706,37 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Monitoring Runs</td>
<td>188</td>
<td>95.9%</td>
</tr>
-
<tr>
<td>Accessing Task Logs</td>
<td>176</td>
<td>89.8%</td>
</tr>
-
<tr>
<td>Manually triggering DAGs</td>
<td>167</td>
<td>85.2%</td>
</tr>
-
<tr>
<td>Clearing Tasks</td>
<td>162</td>
<td>82.7%</td>
</tr>
-
<tr>
<td>Marking Tasks as successful</td>
<td>119</td>
<td>60.7%</td>
</tr>
-
<tr>
<td>Other</td>
<td>6</td>
@@ -2037,11 +1744,8 @@
</tr>
</tbody>
</table>
-
<p>Airflow web UI is used heavily for monitoring: Monitoring Runs (95.9%) and troubleshooting: Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%).</p>
-
<h3 id="if-cli-marked-what-do-you-use-the-cli-for-multiple-choice">(if CLI Marked) What do you use the CLI For? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -2050,50 +1754,42 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Backfilling</td>
<td>63</td>
<td>56.8%</td>
</tr>
-
<tr>
<td>Manually triggering DAGs</td>
<td>52</td>
<td>46.8%</td>
</tr>
-
<tr>
<td>Clearing Tasks</td>
<td>26</td>
<td>23.4%</td>
</tr>
-
<tr>
<td>Monitoring Runs</td>
<td>25</td>
<td>22.5%</td>
</tr>
-
<tr>
<td>Accessing Task Logs</td>
<td>21</td>
<td>18.9%</td>
</tr>
-
<tr>
<td>Marking Tasks as successful</td>
<td>11</td>
<td>9.9%</td>
</tr>
-
<tr>
<td>Other</td>
<td>17</td>
@@ -2101,11 +1797,8 @@
</tr>
</tbody>
</table>
-
<p>Compared to Airflow web UI, Airflow CLI is used mainly for Backfilling (56.8%) and Manually triggering DAGs (46.8%).</p>
-
<h3 id="in-airflow-which-ui-views-are-important-for-you-multiple-choice">In Airflow, which UI views are important for you? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -2114,92 +1807,77 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>List of DAGs</td>
<td>178</td>
<td>89.4%</td>
</tr>
-
<tr>
<td>Task Logs</td>
<td>162</td>
<td>81.4%</td>
</tr>
-
<tr>
<td>DAG Runs</td>
<td>160</td>
<td>80.4%</td>
</tr>
-
<tr>
<td>Graph view</td>
<td>147</td>
<td>73.9%</td>
</tr>
-
<tr>
<td>Grid/Tree View</td>
<td>138</td>
<td>69.3%</td>
</tr>
-
<tr>
<td>Run Details</td>
<td>117</td>
<td>58.8%</td>
</tr>
-
<tr>
<td>DAG details</td>
<td>111</td>
<td>55.8%</td>
</tr>
-
<tr>
<td>Task Instances</td>
<td>102</td>
<td>51.3%</td>
</tr>
-
<tr>
<td>Task Duration</td>
<td>91</td>
<td>45.7%</td>
</tr>
-
<tr>
<td>Code</td>
<td>90</td>
<td>45.2%</td>
</tr>
-
<tr>
<td>Task Tries</td>
<td>60</td>
<td>30.2%</td>
</tr>
-
<tr>
<td>Gantt</td>
<td>48</td>
<td>21.4%</td>
</tr>
-
<tr>
<td>Landing Times</td>
<td>27</td>
<td>13.6%</td>
</tr>
-
<tr>
<td>Other</td>
<td>4</td>
@@ -2207,15 +1885,10 @@
</tr>
</tbody>
</table>
-
-<p>UI views importance ranking shows that the majority Airflow users use Web UI mostly for monitoring and/or troubleshooting purposes, where the top 3 views are List of DAGs (89.4%), Task Logs (81.4%) and DAG Runs (80.4%). The results are very similar to those from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">2019</a>.</p>
-
+<p>UI views importance ranking shows that the majority Airflow users use Web UI mostly for monitoring and/or troubleshooting purposes, where the top 3 views are List of DAGs (89.4%), Task Logs (81.4%) and DAG Runs (80.4%). The results are very similar to those from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>.</p>
<h2 id="community-and-contribution-1">Community and contribution</h2>
-
<h3 id="are-you-participating-in-the-airflow-community-discussions-single-choice">Are you participating in the Airflow community discussions? (single choice)</h3>
-
-<p><img src="images/image16.png" alt="alt_text" title="discussions_engagement" /></p>
-
+<p><img src="images/image16.png" alt="alt_text" title="discussions_engagement"></p>
<table>
<thead>
<tr>
@@ -2224,62 +1897,52 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>I see them from time to time</td>
<td>99</td>
<td>48.3%</td>
</tr>
-
<tr>
<td>I regularly follow what’s being discussed but don’t participate</td>
<td>53</td>
<td>25.9%</td>
</tr>
-
<tr>
<td>I didn’t know I could</td>
<td>41</td>
<td>20.0%</td>
</tr>
-
<tr>
<td>I actively participate in the discussions</td>
<td>12</td>
<td>5.9%</td>
</tr>
-
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>I know I can but I do not contribute</td>
<td>116</td>
<td>57.1%</td>
</tr>
-
<tr>
<td>Very rarely when it relates to what I need</td>
<td>44</td>
<td>21.7%</td>
</tr>
-
<tr>
<td>I do not know I could</td>
<td>30</td>
<td>14.8%</td>
</tr>
-
<tr>
<td>I regularly contribute by discussing, reviewing and submitting PR</td>
<td>13</td>
@@ -2287,13 +1950,9 @@
</tr>
</tbody>
</table>
-
<p>Results related to the Airflow contribution are very similar to those about participating in the Airflow community discussions. Most of the Airflow users (57.1%) who responded to the survey are aware but do not contribute or contribute very rarely (21.7%). 14.8% of users were not aware they could contribute. Once again, itโs a clear indicator that there is much more to be done to engage our community to be more active contributors and raise the current 6.4% of users who actively contr [...]
-
-<h3 id="if-you-do-not-contribute-why">If you do not contribute - why?</h3>
-
-<p><img src="images/image18.png" alt="alt_text" title="contribution_reasons" /></p>
-
+<h3 id="if-you-do-not-contribute---why">If you do not contribute - why?</h3>
+<p><img src="images/image18.png" alt="alt_text" title="contribution_reasons"></p>
<table>
<thead>
<tr>
@@ -2302,38 +1961,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>I have no time to contribute even if would like to</td>
<td>65</td>
<td>38.9%</td>
</tr>
-
<tr>
<td>I donโt know how to start</td>
<td>63</td>
<td>37.7%</td>
</tr>
-
<tr>
<td>I donโt have a need to contribute</td>
<td>19</td>
<td>11.4%</td>
</tr>
-
<tr>
<td>I didnโt know I could</td>
<td>12</td>
<td>7.2%</td>
</tr>
-
<tr>
<td>My employer has policy that makes it difficult to contribute</td>
<td>8</td>
@@ -2341,13 +1994,9 @@
</tr>
</tbody>
</table>
-
<p>According to the survey results, the most important blocker for the Airflow contribution is limited time (38.9%), but surprisingly interesting and important blocker is also lack of knowledge on how to start (37.7%), followed by lack of knowledge that itโs possible to contribute (7.2%).</p>
-
<h2 id="the-future-of-airflow-1">The future of Airflow</h2>
-
<h3 id="in-your-opinion-what-could-be-improved-in-airflow-multiple-choice">In your opinion, what could be improved in Airflow? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -2356,86 +2005,72 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Web UI</td>
<td>100</td>
<td>49.5%</td>
</tr>
-
<tr>
<td>Logging, monitoring and alerting</td>
<td>97</td>
<td>48.0%</td>
</tr>
-
<tr>
<td>Examples, how-to, onboarding documentation</td>
<td>74</td>
<td>36.6%</td>
</tr>
-
<tr>
<td>Technical documentation</td>
<td>74</td>
<td>36.6%</td>
</tr>
-
<tr>
<td>Scheduler performance</td>
<td>56</td>
<td>27.7%</td>
</tr>
-
<tr>
<td>Reliability</td>
<td>52</td>
<td>25.7%</td>
</tr>
-
<tr>
<td>DAG authoring</td>
<td>48</td>
<td>23.8%</td>
</tr>
-
<tr>
<td>REST API</td>
<td>43</td>
<td>21.3%</td>
</tr>
-
<tr>
<td>Authentication and authorization</td>
<td>41</td>
<td>20.3%</td>
</tr>
-
<tr>
<td>External integration e.g. AWS, GCP, Apache products</td>
<td>41</td>
<td>20.3%</td>
</tr>
-
<tr>
<td>Better support for various deployments (Docker-compose/Nomad/Others)</td>
<td>39</td>
<td>19.3%</td>
</tr>
-
<tr>
<td>Everything works fine for me</td>
<td>19</td>
<td>9.4%</td>
</tr>
-
<tr>
<td>I donโt know</td>
<td>4</td>
@@ -2443,11 +2078,8 @@
</tr>
</tbody>
</table>
-
<p>The results are quite self-explanatory. According to the survey results, the top area for improvement is still the Airflow web UI (49.5%), closely followed by more telemetry for logging, monitoring and alerting purposes (48%). However all those efforts should go in line with improved documentation (36.6.%) and resources about using the Airflow, especially when we take into account the need of onboarding new users (36.6%).</p>
-
<h3 id="which-features-would-you-like-to-see-in-airflow">Which features would you like to see in Airflow?</h3>
-
<table>
<thead>
<tr>
@@ -2456,80 +2088,67 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>DAG Versioning</td>
<td>129</td>
<td>66.2%</td>
</tr>
-
<tr>
<td>Dependency management and Data-driven scheduling</td>
<td>83</td>
<td>42.6%</td>
</tr>
-
<tr>
<td>More dynamic task structure</td>
<td>82</td>
<td>42.1%</td>
</tr>
-
<tr>
<td>Multi-Tenancy</td>
<td>74</td>
<td>37.9%</td>
</tr>
-
<tr>
<td>Signal-based scheduling</td>
<td>67</td>
<td>34.4%</td>
</tr>
-
<tr>
<td>Better Security (Isolation)</td>
<td>65</td>
<td>33.3%</td>
</tr>
-
<tr>
<td>Submitting new DAGs externally via API</td>
<td>53</td>
<td>27.2%</td>
</tr>
-
<tr>
<td>Composable Operators</td>
<td>46</td>
<td>23.6%</td>
</tr>
-
<tr>
<td>Support for native cloud executors (AWS/GCP/Azure etc.)</td>
<td>44</td>
<td>22.6%</td>
</tr>
-
<tr>
<td>Better support for Machine Learning</td>
<td>38</td>
<td>19.5%</td>
</tr>
-
<tr>
<td>Remote CLI</td>
<td>36</td>
<td>18.5%</td>
</tr>
-
<tr>
<td>Support for hybrid executors</td>
<td>22</td>
@@ -2537,11 +2156,8 @@
</tr>
</tbody>
</table>
-
<p>According to the survey results, DAG Versioning is a winner for new features in Airflow, and itโs not a surprise as this feature may positively impact daily work of Airflow users. It is followed by three other ideas: Dependency management and Data-driven scheduling (42.6%), More dynamic task structure (42.1%) and Multi-Tenancy (37.9%). Another interesting point from that question is that only 11.3% think that support for hybrid executors is needed in Airflow.</p>
-
<h2 id="data">Data</h2>
-
<p>If you’re interested in taking a look at the raw data yourself, it’s available here: (Airflow User Survey 2022.csv)[/data/survey-responses/airflow-user-survey-responses-2022.csv.zip]</p>
</div>
@@ -2886,7 +2502,7 @@
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-survey/index.html b/blog/airflow-survey/index.html
index eb72519dac..2409905f5d 100644
--- a/blog/airflow-survey/index.html
+++ b/blog/airflow-survey/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Airflow Survey 2019 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2019" />
<meta property="og:description" content="Receiving and adjusting to our usersโ feedback is a must. Letโs see who Airflow users are, how they play with it, and what they miss." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-survey/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Airflow Survey 2019">
-<meta itemprop="description" content="Receiving and adjusting to our usersโ feedback is a must. Letโs see who Airflow users are, how they play with it, and what they miss.">
-
-
-<meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="1775">
-
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="keywords" content="community,survey,users," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Airflow Survey 2019">
+<meta itemprop="description" content="Receiving and adjusting to our usersโ feedback is a must. Letโs see who Airflow users are, how they play with it, and what they miss."><meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="1775"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="community,survey,users," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Airflow Survey 2019"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -503,19 +495,13 @@
<p class="blogpost-content__metadata--description">Receiving and adjusting to our usersโ feedback is a must. Letโs see who Airflow users are, how they play with it, and what they miss.</p>
</div>
<div class="markdown-content">
-
-
-<h1 id="apache-airflow-survey-2019">Apache Airflow Survey 2019</h1>
-
-<p>Apache Airflow is <a href="https://www.astronomer.io/blog/why-airflow/" target="_blank">growing faster than ever</a>.
+ <h1 id="apache-airflow-survey-2019">Apache Airflow Survey 2019</h1>
+<p>Apache Airflow is <a href="https://www.astronomer.io/blog/why-airflow/">growing faster than ever</a>.
Thus, receiving and adjusting to our usersโ feedback is a must. We created
-<a href="https://forms.gle/XAzR1pQBZiftvPQM7" target="_blank">survey</a> and we got <strong>308</strong> responses.
+<a href="https://forms.gle/XAzR1pQBZiftvPQM7">survey</a> and we got <strong>308</strong> responses.
Letโs see who Airflow users are, how they play with it, and what they miss.</p>
-
<h1 id="overview-of-the-user">Overview of the user</h1>
-
<p><strong>What best describes your current occupation?</strong></p>
-
<table>
<thead>
<tr>
@@ -524,92 +510,77 @@ Letโs see who Airflow users are, how they play with it, and what they miss.</p
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Data Engineer</td>
<td>194</td>
<td>62.99%</td>
</tr>
-
<tr>
<td>Developer</td>
<td>34</td>
<td>11.04%</td>
</tr>
-
<tr>
<td>Architect</td>
<td>23</td>
<td>7.47%</td>
</tr>
-
<tr>
<td>Data Scientist</td>
<td>19</td>
<td>6.17%</td>
</tr>
-
<tr>
<td>Data Analyst</td>
<td>13</td>
<td>4.22%</td>
</tr>
-
<tr>
<td>DevOps</td>
<td>13</td>
<td>4.22%</td>
</tr>
-
<tr>
<td>IT Administrator</td>
<td>2</td>
<td>0.65%</td>
</tr>
-
<tr>
<td>Machine Learning Engineer</td>
<td>2</td>
<td>0.65%</td>
</tr>
-
<tr>
<td>Manager</td>
<td>2</td>
<td>0.65%</td>
</tr>
-
<tr>
<td>Operations</td>
<td>2</td>
<td>0.65%</td>
</tr>
-
<tr>
<td>Chief Data Officer</td>
<td>1</td>
<td>0.32%</td>
</tr>
-
<tr>
<td>Engineering Manager</td>
<td>1</td>
<td>0.32%</td>
</tr>
-
<tr>
<td>Intern</td>
<td>1</td>
<td>0.32%</td>
</tr>
-
<tr>
<td>Product owner</td>
<td>1</td>
<td>0.32%</td>
</tr>
-
<tr>
<td>Quant</td>
<td>1</td>
@@ -617,9 +588,7 @@ Letโs see who Airflow users are, how they play with it, and what they miss.</p
</tr>
</tbody>
</table>
-
<p><strong>In your day to day job, what do you use Airflow for?</strong></p>
-
<table>
<thead>
<tr>
@@ -628,20 +597,17 @@ Letโs see who Airflow users are, how they play with it, and what they miss.</p
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Data processing (ETL)</td>
<td>298</td>
<td>96.75%</td>
</tr>
-
<tr>
<td>Artificial Intelligence and Machine Learning Pipelines</td>
<td>90</td>
<td>29.22%</td>
</tr>
-
<tr>
<td>Automating DevOps operations</td>
<td>64</td>
@@ -649,15 +615,12 @@ Letโs see who Airflow users are, how they play with it, and what they miss.</p
</tr>
</tbody>
</table>
-
<p>According to the survey, most of the Airflow users are the โdataโ people. Moreover,
28.57% uses Airflow to both ETL and ML pipelines meaning that those two fields
are somehow connected. Only five respondents use Airflow for DevOps operations only,
That means that other 59 people who use Airflow for DevOps stuff use it also for
ETL / ML purposes.</p>
-
<p><strong>How many active DAGs do you have in your largest Airflow instance?</strong></p>
-
<table>
<thead>
<tr>
@@ -666,50 +629,42 @@ ETL / ML purposes.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>0-20</td>
<td>115</td>
<td>37.34%</td>
</tr>
-
<tr>
<td>21-40</td>
<td>65</td>
<td>21.10%</td>
</tr>
-
<tr>
<td>41-60</td>
<td>44</td>
<td>14.29%</td>
</tr>
-
<tr>
<td>61-100</td>
<td>28</td>
<td>9.09%</td>
</tr>
-
<tr>
<td>101-200</td>
<td>28</td>
<td>9.09%</td>
</tr>
-
<tr>
<td>201-300</td>
<td>7</td>
<td>2.27%</td>
</tr>
-
<tr>
<td>301-999</td>
<td>8</td>
<td>2.60%</td>
</tr>
-
<tr>
<td>1000+</td>
<td>13</td>
@@ -717,12 +672,9 @@ ETL / ML purposes.</p>
</tr>
</tbody>
</table>
-
<p>The majority of users do not exceed 100 active DAGs per Airflow instance. However,
as we can see there are users who exceed thousands of DAGs with a maximum number 5000.</p>
-
<p><strong>What is the maximum number of tasks that you have used in one DAG?</strong></p>
-
<table>
<thead>
<tr>
@@ -731,56 +683,47 @@ as we can see there are users who exceed thousands of DAGs with a maximum number
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>0-10</td>
<td>61</td>
<td>19.81%</td>
</tr>
-
<tr>
<td>11-20</td>
<td>60</td>
<td>19.48%</td>
</tr>
-
<tr>
<td>21-30</td>
<td>31</td>
<td>10.06%</td>
</tr>
-
<tr>
<td>31-40</td>
<td>21</td>
<td>6.82%</td>
</tr>
-
<tr>
<td>41-50</td>
<td>26</td>
<td>8.44%</td>
</tr>
-
<tr>
<td>51-100</td>
<td>36</td>
<td>11.69%</td>
</tr>
-
<tr>
<td>101-200</td>
<td>28</td>
<td>9.09%</td>
</tr>
-
<tr>
<td>201-500</td>
<td>21</td>
<td>6.82%</td>
</tr>
-
<tr>
<td>501+</td>
<td>24</td>
@@ -788,13 +731,10 @@ as we can see there are users who exceed thousands of DAGs with a maximum number
</tr>
</tbody>
</table>
-
<p>The given maximum number of tasks in a single DAG was 10 000 (!). The number of tasks
depends on the purposes of a DAG, so itโs rather hard to say if users have โsimpleโ
or โcomplicatedโ workflows.</p>
-
<p><strong>When onboarding new members to Airflow, what is the biggest problem?</strong></p>
-
<table>
<thead>
<tr>
@@ -803,32 +743,27 @@ or โcomplicatedโ workflows.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>No guide on best practises on developing DAGs</td>
<td>160</td>
<td>51.95%</td>
</tr>
-
<tr>
<td>Small number of tutorials on different aspects of using Airflow</td>
<td>57</td>
<td>18.51%</td>
</tr>
-
<tr>
<td>Documentation is not clear enough</td>
<td>42</td>
<td>13.64%</td>
</tr>
-
<tr>
<td>Small number of blogs regarding Airflow</td>
<td>6</td>
<td>1.95%</td>
</tr>
-
<tr>
<td>Other</td>
<td>43</td>
@@ -836,19 +771,15 @@ or โcomplicatedโ workflows.</p>
</tr>
</tbody>
</table>
-
<p>This is an important result. Using Airflow is all about writing and scheduling DAGs.
No guide or any other complete resource on best practices for developing Dags is a big
problem. Diving deep in the โotherโ answers, we can find that:</p>
-
<ul>
<li>Airflowโs โmagicโ (scheduler, executors, schedule times) is hard to understand</li>
<li>DAG testing is not easy to do and to explain</li>
<li>Airflow UI needs some love.</li>
</ul>
-
<p><strong>How likely are you to recommend Apache Airflow?</strong></p>
-
<table>
<thead>
<tr>
@@ -857,32 +788,27 @@ problem. Diving deep in the โotherโ answers, we can find that:</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Very Likely</td>
<td>140</td>
<td>45.45%</td>
</tr>
-
<tr>
<td>Likely</td>
<td>124</td>
<td>40.26%</td>
</tr>
-
<tr>
<td>Neutral</td>
<td>33</td>
<td>10.71%</td>
</tr>
-
<tr>
<td>Unlikely</td>
<td>8</td>
<td>2.60%</td>
</tr>
-
<tr>
<td>Very unlikely</td>
<td>3</td>
@@ -890,16 +816,12 @@ problem. Diving deep in the โotherโ answers, we can find that:</p>
</tr>
</tbody>
</table>
-
<p>This means that more than 85% of people who use Airflow like it. It seems Airflow does
its job nicely. However, we have to remember that this survey is likely biased - itโs
more likely that you respond to the survey if you like the tool you use. Should we
focus then on those 11 people who did not like Airflow? Itโs a good question.</p>
-
<h2 id="airflow-usage">Airflow usage</h2>
-
<p><strong>Which interface(s) of Airflow do you use as part of your current role?</strong></p>
-
<table>
<thead>
<tr>
@@ -908,38 +830,32 @@ focus then on those 11 people who did not like Airflow? Itโs a good question.<
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Original Airflow Graphical User Interface</td>
<td>297</td>
<td>96.43%</td>
</tr>
-
<tr>
<td>CLI</td>
<td>126</td>
<td>40.91%</td>
</tr>
-
<tr>
<td>Original Airflow Graphical User Interface, CLI</td>
<td>117</td>
<td>37.99%</td>
</tr>
-
<tr>
<td>API</td>
<td>60</td>
<td>19.48%</td>
</tr>
-
<tr>
<td>Original Airflow Graphical User Interface, CLI, API</td>
<td>32</td>
<td>10.39%</td>
</tr>
-
<tr>
<td>Custom (own created) Airflow Graphical User Interface</td>
<td>25</td>
@@ -947,35 +863,23 @@ focus then on those 11 people who did not like Airflow? Itโs a good question.<
</tr>
</tbody>
</table>
-
<p>Itโs visible that usage of CLI goes in pair with using Airflow web UI. Our
survey included some UX related questions to allow us to understand how users
use Airflow webserver.</p>
-
<p><strong>What do you use the Graphical User Interface for?</strong></p>
-
-<p><img src="plot1.png" alt="" /></p>
-
+<p><img src="plot1.png" alt=""></p>
<p><strong>What do you use CLI for?</strong></p>
-
-<p><img src="plot2.png" alt="" /></p>
-
+<p><img src="plot2.png" alt=""></p>
<p><strong>In Airflow, which UI view(s) are important for you?</strong></p>
-
-<p><img src="plot3.png" alt="" /></p>
-
+<p><img src="plot3.png" alt=""></p>
<p>Here we see that the majority uses Web UI mostly for monitoring purposes:</p>
-
<ul>
<li>Monitoring DAGs</li>
<li>Accessing logs</li>
</ul>
-
<p>An interesting result is that many people seem not to use backfilling as
thereโs no other way than to do it by CLI.</p>
-
<p><strong>What executor type do you use?</strong></p>
-
<table>
<thead>
<tr>
@@ -984,32 +888,27 @@ thereโs no other way than to do it by CLI.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Celery</td>
<td>138</td>
<td>44.81%</td>
</tr>
-
<tr>
<td>Local</td>
<td>85</td>
<td>27.60%</td>
</tr>
-
<tr>
<td>Kubernetes</td>
<td>52</td>
<td>16.88%</td>
</tr>
-
<tr>
<td>Sequential</td>
<td>22</td>
<td>7.14%</td>
</tr>
-
<tr>
<td>Other</td>
<td>11</td>
@@ -1017,14 +916,11 @@ thereโs no other way than to do it by CLI.</p>
</tr>
</tbody>
</table>
-
<p>The other option mostly consisted of information that someone uses a few types or is
migrating from one executor to another. What can be observed is an increase in usage
-of Local and Kubernetes executors when compared to results from an earlier <a href="https://ash.berlintaylor.com/writings/2019/02/airflow-user-survey-2019/" target="_blank">survey done
+of Local and Kubernetes executors when compared to results from an earlier <a href="https://ash.berlintaylor.com/writings/2019/02/airflow-user-survey-2019/">survey done
by Ash</a>.</p>
-
<p><strong>Do you use Kubernetes-based deployments for Airflow?</strong></p>
-
<table>
<thead>
<tr>
@@ -1033,38 +929,32 @@ by Ash</a>.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>No - we do not plan to use Kubernetes near term</td>
<td>88</td>
<td>28.57%</td>
</tr>
-
<tr>
<td>Yes - setup on our own via Helm Chart or similar</td>
<td>65</td>
<td>21.10%</td>
</tr>
-
<tr>
<td>Not yet - but we use Kubernetes in our organization and we could move</td>
<td>61</td>
<td>19.81%</td>
</tr>
-
<tr>
<td>Yes - via managed service in the cloud (Composer / Astronomer etc.)</td>
<td>45</td>
<td>14.61%</td>
</tr>
-
<tr>
<td>Not yet - but we plan to deploy Kubernetes in our organization soon</td>
<td>42</td>
<td>13.64%</td>
</tr>
-
<tr>
<td>Other</td>
<td>7</td>
@@ -1072,14 +962,11 @@ by Ash</a>.</p>
</tr>
</tbody>
</table>
-
<p>The most interesting thing is that thereโs nearly 30% of users who do not use Kubernetes,
and they are not going to move. This means we should keep other deployment options in
mind when working on Airflow 2.0. On the other hand, almost 70% of the users already
use Kubernetes, or itโs a viable option for them.</p>
-
<p><strong>Do you combine multiple DAGs?</strong></p>
-
<table>
<thead>
<tr>
@@ -1088,26 +975,22 @@ use Kubernetes, or itโs a viable option for them.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>No, I don’t combine multiple DAGs</td>
<td>127</td>
<td>41.23%</td>
</tr>
-
<tr>
<td>Yes, through SubDAG</td>
<td>73</td>
<td>23.70%</td>
</tr>
-
<tr>
<td>Yes, by triggering another DAG</td>
<td>72</td>
<td>23.38%</td>
</tr>
-
<tr>
<td>Other</td>
<td>36</td>
@@ -1115,12 +998,9 @@ use Kubernetes, or itโs a viable option for them.</p>
</tr>
</tbody>
</table>
-
<p>In the other category, 9 people explicitly mentioned using <code>ExternalTaskSensor</code>,
and I think it could be treated as running subDAGs by triggering other DAGs.</p>
-
<p><strong>Do you use Airflow Plugins? If yes, what do you use it for?</strong></p>
-
<table>
<thead>
<tr>
@@ -1129,32 +1009,27 @@ and I think it could be treated as running subDAGs by triggering other DAGs.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Adding new operators/sensors and hooks</td>
<td>187</td>
<td>60.71%</td>
</tr>
-
<tr>
<td>I don’t use Airflow plugins</td>
<td>109</td>
<td>35.39%</td>
</tr>
-
<tr>
<td>Adding AppBuilder views & menu items</td>
<td>31</td>
<td>10.06%</td>
</tr>
-
<tr>
<td>Adding new executor</td>
<td>18</td>
<td>5.84%</td>
</tr>
-
<tr>
<td>Adding OperatorExtraLinks</td>
<td>7</td>
@@ -1162,31 +1037,23 @@ and I think it could be treated as running subDAGs by triggering other DAGs.</p>
</tr>
</tbody>
</table>
-
<p>The high percentage - 60% for โAdding new operators/sensors and hooksโ is quite a
surprising result for some of us - especially that you do not actually need to use the
plugin mechanism to add any of those. Those are standard python objects, and you can
simply drop your hooks/operators/sensors code to <code>PYTHONPATH</code> environment variable and
they will work. It seems that this may be a result of a lack of best practices guide.</p>
-
<p>Plugins are more useful for adding views and menu items - yet only 10%.
OperatorExtraLinks are even more useful (though relatively new) feature, so itโs not
entirely surprising they are hardly used.</p>
-
<p>It was also kind of surprising that someone at all uses plugins to use their own
executors. We considered removing that option recently - but now we have to rethink
our approach.</p>
-
<p><strong>What metrics do you use to monitor Airflow?</strong></p>
-
<p>There were a lot of different responses. Some use Prometheus and other services,
others do not use any monitoring. One of the interesting responses linked to this
-solution for <a href="https://github.com/mastak/airflow_operators_metrics" target="_blank">airflow_operators_metrics</a>.</p>
-
+solution for <a href="https://github.com/mastak/airflow_operators_metrics">airflow_operators_metrics</a>.</p>
<h2 id="external-services">External services</h2>
-
<p><strong>What external services do you use in your Airflow DAGs?</strong></p>
-
<table>
<thead>
<tr>
@@ -1195,38 +1062,32 @@ solution for <a href="https://github.com/mastak/airflow_operators_metrics" targe
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Amazon Web Services</td>
<td>160</td>
<td>51.95%</td>
</tr>
-
<tr>
<td>Internal company systems</td>
<td>150</td>
<td>48.7%</td>
</tr>
-
<tr>
<td>Hadoop / Spark / Flink / Other Apache software</td>
<td>119</td>
<td>38.64%</td>
</tr>
-
<tr>
<td>Google Cloud Platform / Google APIs</td>
<td>112</td>
<td>36.36%</td>
</tr>
-
<tr>
<td>Microsoft Azure</td>
<td>28</td>
<td>9.09%</td>
</tr>
-
<tr>
<td>I do not use external services in my Airflow DAGs</td>
<td>18</td>
@@ -1234,13 +1095,10 @@ solution for <a href="https://github.com/mastak/airflow_operators_metrics" targe
</tr>
</tbody>
</table>
-
<p>Itโs not surprising that Amazon Web Services is leading the way as they are considered the most mature
cloud provider. Internal system and other Apache products on the next two positions are
quite understandable if we take into account that the majority uses Airflow for ETL processes.</p>
-
<p><strong>What external services do you use in your Airflow DAGs? (Mixed providers)</strong></p>
-
<table>
<thead>
<tr>
@@ -1249,20 +1107,17 @@ quite understandable if we take into account that the majority uses Airflow for
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Google Cloud Platform / Google APIs, Amazon Web Services</td>
<td>44</td>
<td>14.29%</td>
</tr>
-
<tr>
<td>Amazon Web Services, Microsoft Azure</td>
<td>5</td>
<td>1.62%</td>
</tr>
-
<tr>
<td>Google Cloud Platform / Google APIs, Microsoft Azure</td>
<td>4</td>
@@ -1270,12 +1125,9 @@ quite understandable if we take into account that the majority uses Airflow for
</tr>
</tbody>
</table>
-
<p>This result is not surprising because companies usually prefer to stick with one cloud
provider.</p>
-
<p><strong>How do you integrate with external services?</strong></p>
-
<table>
<thead>
<tr>
@@ -1284,20 +1136,17 @@ provider.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Using Bash / Python operator</td>
<td>220</td>
<td>71.43%</td>
</tr>
-
<tr>
<td>Using existing, dedicated operators / hooks</td>
<td>217</td>
<td>70.45%</td>
</tr>
-
<tr>
<td>Using own, custom operators / hooks</td>
<td>216</td>
@@ -1305,15 +1154,11 @@ provider.</p>
</tr>
</tbody>
</table>
-
<p>We had some anecdotal evidence that people use more Python/Bash operators than the
dedicated ones - but it looks like all ways of using Airflow to connect to external
services are equally popular.</p>
-
<h2 id="what-can-be-improved">What can be improved</h2>
-
<p><strong>In your opinion, what could be improved in Airflow?</strong></p>
-
<table>
<thead>
<tr>
@@ -1322,68 +1167,57 @@ services are equally popular.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Scheduler performance</td>
<td>189</td>
<td>61.36%</td>
</tr>
-
<tr>
<td>Web UI</td>
<td>180</td>
<td>58.44%</td>
</tr>
-
<tr>
<td>Logging, monitoring and alerting</td>
<td>145</td>
<td>47.08%</td>
</tr>
-
<tr>
<td>Examples, how-to, onboarding documentation</td>
<td>143</td>
<td>46.43%</td>
</tr>
-
<tr>
<td>Technical documentation</td>
<td>137</td>
<td>44.48%</td>
</tr>
-
<tr>
<td>Reliability</td>
<td>112</td>
<td>36.36%</td>
</tr>
-
<tr>
<td>REST API</td>
<td>96</td>
<td>31.17%</td>
</tr>
-
<tr>
<td>Authentication and authorization</td>
<td>89</td>
<td>28.9%</td>
</tr>
-
<tr>
<td>External integration e.g. AWS, GCP, Apache product</td>
<td>49</td>
<td>15.91%</td>
</tr>
-
<tr>
<td>CLI</td>
<td>41</td>
<td>13.31%</td>
</tr>
-
<tr>
<td>I donโt know</td>
<td>5</td>
@@ -1391,18 +1225,14 @@ services are equally popular.</p>
</tr>
</tbody>
</table>
-
<p>The results are rather quite self-explaining. Improved performance of Airflow, better
UI, and more telemetry are desirable. But this should go in pair with improved
documentation and resources about using the Airflow, especially when we
take into account the problem of onboarding new users.</p>
-
<p>Another interesting point from that question is that only 16% think that operators
should be extended and improved. This suggests that we should focus on improving
Airflow core instead of adding more and more integrations.</p>
-
<p><strong>What would be the most interesting feature for you?</strong></p>
-
<table>
<thead>
<tr>
@@ -1411,44 +1241,37 @@ Airflow core instead of adding more and more integrations.</p>
<th>%</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Production-ready Airflow docker image</td>
<td>175</td>
<td>56.82%</td>
</tr>
-
<tr>
<td>Declarative way of writing DAGs / automated DAGs generation</td>
<td>155</td>
<td>50.32%</td>
</tr>
-
<tr>
<td>Horizontal Autoscaling</td>
<td>122</td>
<td>39.61%</td>
</tr>
-
<tr>
<td>Asynchronous Operators</td>
<td>97</td>
<td>31.49%</td>
</tr>
-
<tr>
<td>Stateless web server</td>
<td>81</td>
<td>26.3%</td>
</tr>
-
<tr>
<td>Knative Executor</td>
<td>48</td>
<td>15.58%</td>
</tr>
-
<tr>
<td>I already have all I need</td>
<td>13</td>
@@ -1456,7 +1279,6 @@ Airflow core instead of adding more and more integrations.</p>
</tr>
</tbody>
</table>
-
<p>Production Docker image wins, and itโs not a surprise. We all know that deploying
Airflow is not a plug and play process, and thatโs why the official image is being
worked on by Jarek Potiuk. An unexpected result is that half of the users would like to
@@ -1464,19 +1286,15 @@ have a declarative way of creating DAGs. That seems to be something that is โa
as we always emphasize the possibility of writing workflows in pure python. Stories
about DAG generators are not new and confirm that thereโs a need for a way to
declare DAGs.</p>
-
<h2 id="data">Data</h2>
-
<p>If you think I missed something and you want to look for insights on your own the data is available
for you here:</p>
-
<ul>
-<li>Original data: <a href="https://storage.googleapis.com/airflow-survey/survey.csv" target="_blank">https://storage.googleapis.com/airflow-survey/survey.csv</a></li>
-<li>Processed: <a href="https://storage.googleapis.com/airflow-survey/airflow_survey_processed.csv" target="_blank">https://storage.googleapis.com/airflow-survey/airflow_survey_processed.csv</a></li>
+<li>Original data: <a href="https://storage.googleapis.com/airflow-survey/survey.csv">https://storage.googleapis.com/airflow-survey/survey.csv</a></li>
+<li>Processed: <a href="https://storage.googleapis.com/airflow-survey/airflow_survey_processed.csv">https://storage.googleapis.com/airflow-survey/airflow_survey_processed.csv</a></li>
</ul>
-
<p>The processed data includes multi-choice options one-hot encoded. If you find any interesting
-insight, please update the article (<a href="https://github.com/apache/airflow-site/blob/master/CONTRIBUTE.md" target="_blank">make PR</a>
+insight, please update the article (<a href="https://github.com/apache/airflow-site/blob/master/CONTRIBUTE.md">make PR</a>
to Airflow site).</p>
</div>
@@ -1821,7 +1639,7 @@ to Airflow site).</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow-two-point-oh-is-here/index.html b/blog/airflow-two-point-oh-is-here/index.html
index 6c5caee083..89c332b94a 100644
--- a/blog/airflow-two-point-oh-is-here/index.html
+++ b/blog/airflow-two-point-oh-is-here/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Apache Airflow 2.0 is here! | Apache Airflow</title><meta property="og:title" content="Apache Airflow 2.0 is here!" />
<meta property="og:description" content="We're proud to announce that Apache Airflow 2.0.0 has been released." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-two-point-oh-is-here/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow-two-point-oh-is-here/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2020-12-17T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Apache Airflow 2.0 is here!">
-<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.0.0 has been released.">
-
-
-<meta itemprop="datePublished" content="2020-12-17T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="1165">
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-
-<meta itemprop="keywords" content="Release," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Apache Airflow 2.0 is here!">
+<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.0.0 has been released."><meta itemprop="datePublished" content="2020-12-17T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="1165"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Release," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Apache Airflow 2.0 is here!"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -480,141 +472,87 @@
<p class="blogpost-content__metadata--description">We're proud to announce that Apache Airflow 2.0.0 has been released.</p>
</div>
<div class="markdown-content">
-
-
-<p>I am proud to announce that Apache Airflow 2.0.0 has been released.</p>
-
+ <p>I am proud to announce that Apache Airflow 2.0.0 has been released.</p>
<p>The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:</p>
-
<h2 id="a-new-way-of-writing-dags-the-taskflow-api-aip-31">A new way of writing dags: the TaskFlow API (AIP-31)</h2>
-
<p>(Known in 2.0.0alphas as Functional DAGs.)</p>
-
<p>DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use</p>
-
<p>Read more here:</p>
-
-<p><a href="http://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html" target="_blank">TaskFlow API Tutorial</a> <br />
-<a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#decorated-flows" target="_blank">TaskFlow API Documentation</a></p>
-
+<p><a href="http://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html">TaskFlow API Tutorial</a> <br>
+<a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#decorated-flows">TaskFlow API Documentation</a></p>
<p>A quick teaser of what DAGs can now look like:</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">airflow.decorators</span> <span class="kn">import</span> <span class="n">dag</span><span class="p">,</span> <span class="n">task</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.dates</span> <span class="kn">import</span> <span class="n">days_ago</span>
-<pre><code class="language-python">from airflow.decorators import dag, task
-from airflow.utils.dates import days_ago
-
-@dag(default_args={'owner': 'airflow'}, schedule_interval=None, start_date=days_ago(2))
-def tutorial_taskflow_api_etl():
- @task
- def extract():
- return {"1001": 301.27, "1002": 433.21, "1003": 502.22}
-
- @task
- def transform(order_data_dict: dict) -> dict:
- total_order_value = 0
-
- for value in order_data_dict.values():
- total_order_value += value
+<span class="nd">@dag</span><span class="p">(</span><span class="n">default_args</span><span class="o">=</span><span class="p">{</span><span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">},</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">days_ago</span><span class="p">(</sp [...]
+<span class="k">def</span> <span class="nf">tutorial_taskflow_api_etl</span><span class="p">():</span>
+ <span class="nd">@task</span>
+ <span class="k">def</span> <span class="nf">extract</span><span class="p">():</span>
+ <span class="k">return</span> <span class="p">{</span><span class="s2">"1001"</span><span class="p">:</span> <span class="mf">301.27</span><span class="p">,</span> <span class="s2">"1002"</span><span class="p">:</span> <span class="mf">433.21</span><span class="p">,</span> <span class="s2">"1003"</span><span class="p">:</span> <span class="mf">502.22</span><span class="p">}</span>
- return {"total_order_value": total_order_value}
+ <span class="nd">@task</span>
+ <span class="k">def</span> <span class="nf">transform</span><span class="p">(</span><span class="n">order_data_dict</span><span class="p">:</span> <span class="nb">dict</span><span class="p">)</span> <span class="o">-></span> <span class="nb">dict</span><span class="p">:</span>
+ <span class="n">total_order_value</span> <span class="o">=</span> <span class="mi">0</span>
- @task()
- def load(total_order_value: float):
+ <span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">order_data_dict</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
+ <span class="n">total_order_value</span> <span class="o">+=</span> <span class="n">value</span>
- print("Total order value is: %.2f" % total_order_value)
+ <span class="k">return</span> <span class="p">{</span><span class="s2">"total_order_value"</span><span class="p">:</span> <span class="n">total_order_value</span><span class="p">}</span>
- order_data = extract()
- order_summary = transform(order_data)
- load(order_summary["total_order_value"])
+ <span class="nd">@task</span><span class="p">()</span>
+ <span class="k">def</span> <span class="nf">load</span><span class="p">(</span><span class="n">total_order_value</span><span class="p">:</span> <span class="nb">float</span><span class="p">):</span>
-tutorial_etl_dag = tutorial_taskflow_api_etl()
-</code></pre>
+ <span class="nb">print</span><span class="p">(</span><span class="s2">"Total order value is: </span><span class="si">%.2f</span><span class="s2">"</span> <span class="o">%</span> <span class="n">total_order_value</span><span class="p">)</span>
-<h2 id="fully-specified-rest-api-aip-32">Fully specified REST API (AIP-32)</h2>
+ <span class="n">order_data</span> <span class="o">=</span> <span class="n">extract</span><span class="p">()</span>
+ <span class="n">order_summary</span> <span class="o">=</span> <span class="n">transform</span><span class="p">(</span><span class="n">order_data</span><span class="p">)</span>
+ <span class="n">load</span><span class="p">(</span><span class="n">order_summary</span><span class="p">[</span><span class="s2">"total_order_value"</span><span class="p">])</span>
+<span class="n">tutorial_etl_dag</span> <span class="o">=</span> <span class="n">tutorial_taskflow_api_etl</span><span class="p">()</span>
+</code></pre></div><h2 id="fully-specified-rest-api-aip-32">Fully specified REST API (AIP-32)</h2>
<p>We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification</p>
-
<p>Read more here:</p>
-
-<p><a href="http://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html" target="_blank">REST API Documentation</a>.</p>
-
+<p><a href="http://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html">REST API Documentation</a>.</p>
<h2 id="massive-scheduler-performance-improvements">Massive Scheduler performance improvements</h2>
-
<p>As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.</p>
-
-<p>Over at Astronomer.io we’ve <a href="https://www.astronomer.io/blog/airflow-2-scheduler" target="_blank">benchmarked the schedulerโit’s fast</a> (we had to triple check the numbers as we don’t quite believe them at first!)</p>
-
+<p>Over at Astronomer.io we’ve <a href="https://www.astronomer.io/blog/airflow-2-scheduler">benchmarked the schedulerโit’s fast</a> (we had to triple check the numbers as we don’t quite believe them at first!)</p>
<h2 id="scheduler-is-now-ha-compatible-aip-15">Scheduler is now HA compatible (AIP-15)</h2>
-
<p>It’s now possible and supported to run more than a single scheduler instance. This is super useful for both resiliency (in case a scheduler goes down) and scheduling performance.</p>
-
<p>To fully use this feature you need Postgres 9.6+ or MySQL 8+ (MySQL 5, and MariaDB won’t work with more than one scheduler I’m afraid).</p>
-
<p>There’s no config or other set up required to run more than one schedulerโjust start up a scheduler somewhere else (ensuring it has access to the DAG files) and it will cooperate with your existing schedulers through the database.</p>
-
-<p>For more information, read the <a href="http://airflow.apache.org/docs/apache-airflow/stable/scheduler.html#running-more-than-one-scheduler" target="_blank">Scheduler HA documentation</a>.</p>
-
+<p>For more information, read the <a href="http://airflow.apache.org/docs/apache-airflow/stable/scheduler.html#running-more-than-one-scheduler">Scheduler HA documentation</a>.</p>
<h2 id="task-groups-aip-34">Task Groups (AIP-34)</h2>
-
<p>SubDAGs were commonly used for grouping tasks in the UI, but they had many drawbacks in their execution behaviour (primarily that they only executed a single task in parallel!) To improve this experience, weโve introduced “Task Groups”: a method for organizing tasks which provides the same grouping behaviour as a subdag without any of the execution-time drawbacks.</p>
-
<p>SubDAGs will still work for now, but we think that any previous use of SubDAGs can now be replaced with task groups. If you find an example where this isn’t the case, please let us know by opening an issue on GitHub</p>
-
-<p>For more information, check out the <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskgroup" target="_blank">Task Group documentation</a>.</p>
-
+<p>For more information, check out the <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskgroup">Task Group documentation</a>.</p>
<h2 id="refreshed-ui">Refreshed UI</h2>
-
-<p>We’ve given the Airflow UI <a href="https://github.com/apache/airflow/pull/11195" target="_blank">a visual refresh</a> and updated some of the styling.</p>
-
-<p><img src="airflow-2.0-ui.gif" alt="Airflow 2.0's new UI" /></p>
-
+<p>We’ve given the Airflow UI <a href="https://github.com/apache/airflow/pull/11195">a visual refresh</a> and updated some of the styling.</p>
+<p><img src="airflow-2.0-ui.gif" alt="Airflow 2.0’s new UI"></p>
<p>We have also added an option to auto-refresh task states in Graph View so you no longer need to continuously press the refresh button :).</p>
-
-<p>Check out <a href="http://airflow.apache.org/docs/apache-airflow/stable/ui.html" target="_blank">the screenshots in the docs</a> for more.</p>
-
+<p>Check out <a href="http://airflow.apache.org/docs/apache-airflow/stable/ui.html">the screenshots in the docs</a> for more.</p>
<h2 id="smart-sensors-for-reduced-load-from-sensors-aip-17">Smart Sensors for reduced load from sensors (AIP-17)</h2>
-
<p>If you make heavy use of sensors in your Airflow cluster, you might find that sensor execution takes up a significant proportion of your cluster even with “reschedule” mode. To improve this, we’ve added a new mode called “Smart Sensors”.</p>
-
<p>This feature is in “early-access”: it’s been well-tested by Airbnb and is “stable”/usable, but we reserve the right to make backwards incompatible changes to it in a future release (if we have to. We’ll try very hard not to!)</p>
-
-<p>Read more about it in the <a href="https://airflow.apache.org/docs/apache-airflow/stable/smart-sensor.html" target="_blank">Smart Sensors documentation</a>.</p>
-
+<p>Read more about it in the <a href="https://airflow.apache.org/docs/apache-airflow/stable/smart-sensor.html">Smart Sensors documentation</a>.</p>
<h2 id="simplified-kubernetesexecutor">Simplified KubernetesExecutor</h2>
-
<p>For Airflow 2.0, we have re-architected the KubernetesExecutor in a fashion that is simultaneously faster, easier to understand, and more flexible for Airflow users. Users will now be able to access the full Kubernetes API to create a .yaml <code>pod_template_file</code> instead of specifying parameters in their airflow.cfg.</p>
-
<p>We have also replaced the <code>executor_config</code> dictionary with the <code>pod_override</code> parameter, which takes a Kubernetes V1Pod object for a1:1 setting override. These changes have removed over three thousand lines of code from the KubernetesExecutor, which makes it run faster and creates fewer potential errors.</p>
-
<p>Read more here:</p>
-
-<p><a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html?highlight=pod_override#pod-template-file" target="_blank">Docs on pod_template_file</a> <br />
-<a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html?highlight=pod_override#pod-override" target="_blank">Docs on pod_override</a></p>
-
+<p><a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html?highlight=pod_override#pod-template-file">Docs on pod_template_file</a> <br>
+<a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html?highlight=pod_override#pod-override">Docs on pod_override</a></p>
<h2 id="airflow-core-and-providers-splitting-airflow-into-60-packages">Airflow core and providers: Splitting Airflow into 60+ packages:</h2>
-
<p>Airflow 2.0 is not a monolithic “one to rule them all” package. Weโve split Airflow into core and 61 (for now) provider packages. Each provider package is for either a particular external service (Google, Amazon, Microsoft, Snowflake), a database (Postgres, MySQL), or a protocol (HTTP/FTP). Now you can create a custom Airflow installation from “building” blocks and choose only what you need, plus add whatever other requirements you might have. Some of the commo [...]
-
<p>The provider architecture should make it much easier to get a fully customized, yet consistent runtime with the right set of Python dependencies.</p>
-
<p>But thatโs not all: you can write your own custom providers and add things like custom connection types, customizations of the Connection Forms, and extra links to your operators in a manageable way. You can build your own provider and install it as a Python package and have your customizations visible right in the Airflow UI.</p>
-
-<p>Our very own Jarek Potiuk has written about <a href="https://higrys.medium.com/airflow-2-0-providers-1bd21ba3bd93" target="_blank">providers in much more detail</a> on Jarek’s blog.</p>
-
-<p>Docs on the <a href="http://airflow.apache.org/docs/apache-airflow-providers/" target="_blank">providers concept and writing custom providers</a> <br />
-Docs on <a href="http://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html" target="_blank">all providers packages available</a></p>
-
+<p>Our very own Jarek Potiuk has written about <a href="https://higrys.medium.com/airflow-2-0-providers-1bd21ba3bd93">providers in much more detail</a> on Jarek’s blog.</p>
+<p>Docs on the <a href="http://airflow.apache.org/docs/apache-airflow-providers/">providers concept and writing custom providers</a> <br>
+Docs on <a href="http://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html">all providers packages available</a></p>
<h2 id="security">Security</h2>
-
<p>As part of Airflow 2.0 effort, there has been a conscious focus on Security and reducing areas of exposure. This is represented across different functional areas in different forms. For example, in the new REST API, all operations now require authorization. Similarly, in the configuration settings, the Fernet key is now required to be specified.</p>
-
<h2 id="configuration">Configuration</h2>
-
<p>Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.</p>
-
<h2 id="thanks-to-all-of-you">Thanks to all of you</h2>
-
<p>We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the DAG. That said, please read through UPDATING.md to check what might affect you. For example: We have re-organized the layout of operators (they now all live under airflow.providers.*) but the old names should continue to work - you’ll just notice a lot of DeprecationWarnings that need to be fixed up.</p>
-
<p>Thank you so much to all the contributors who got us to this point, in no particular order: Kaxil Naik, Daniel Imberman, Jarek Potiuk, Tomek Urbaszek, Kamil Breguลa, Gerard Casas Saez, Xiaodong DENG, Kevin Yang, James Timmins, Yingbo Wang, Qian Yu, Ryan Hamilton and the 100s of others who keep making Airflow better for everyone.</p>
</div>
@@ -943,7 +881,7 @@ Docs on <a href="http://airflow.apache.org/docs/apache-airflow-providers/package
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow_summit_2021/index.html b/blog/airflow_summit_2021/index.html
index 2090d00317..60e5b3cc56 100644
--- a/blog/airflow_summit_2021/index.html
+++ b/blog/airflow_summit_2021/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Airflow Summit 2021 | Apache Airflow</title><meta property="og:title" content="Airflow Summit 2021" />
<meta property="og:description" content="We are thrilled about Airflow Summit 2021!" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow_summit_2021/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow_summit_2021/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2021-03-21T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Airflow Summit 2021">
-<meta itemprop="description" content="We are thrilled about Airflow Summit 2021!">
-
-
-<meta itemprop="datePublished" content="2021-03-21T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="77">
-
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="keywords" content="Community,Airflow Summit," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Airflow Summit 2021">
+<meta itemprop="description" content="We are thrilled about Airflow Summit 2021!"><meta itemprop="datePublished" content="2021-03-21T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="77"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Community,Airflow Summit," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Airflow Summit 2021"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -462,20 +454,15 @@
<p class="blogpost-content__metadata--description">We are thrilled about Airflow Summit 2021!</p>
</div>
<div class="markdown-content">
-
-
-<h2 id="airflow-summit-2021-is-here">Airflow Summit 2021 is here!</h2>
-
+ <h2 id="airflow-summit-2021-is-here">Airflow Summit 2021 is here!</h2>
<p>The summit will be held online, July 8-16, 2021. Join us from all over the world to find
out how Airflow is being used by leading companies, what is its roadmap and how you can
participate in its development.</p>
-
<h2 id="useful-information">Useful information:</h2>
-
<ul>
-<li>The official website: <a href="https://airflowsummit.org" target="_blank">https://airflowsummit.org</a></li>
-<li>Call for proposals is open until <strong>12 April 2021</strong>. To submit your talk go to <a href="https://sessionize.com/airflow-summit-2021/" target="_blank">https://sessionize.com/airflow-summit-2021/</a></li>
-<li>In case of any questions reach out to us via info@airflowsummit.org</li>
+<li>The official website: <a href="https://airflowsummit.org">https://airflowsummit.org</a></li>
+<li>Call for proposals is open until <strong>12 April 2021</strong>. To submit your talk go to <a href="https://sessionize.com/airflow-summit-2021/">https://sessionize.com/airflow-summit-2021/</a></li>
+<li>In case of any questions reach out to us via <a href="mailto:info@airflowsummit.org">info@airflowsummit.org</a></li>
</ul>
</div>
@@ -806,7 +793,7 @@ participate in its development.</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/airflow_summit_2022/index.html b/blog/airflow_summit_2022/index.html
index cbc42b4fe7..52dbe9cd4e 100644
--- a/blog/airflow_summit_2022/index.html
+++ b/blog/airflow_summit_2022/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Airflow Summit 2022 | Apache Airflow</title><meta property="og:title" content="Airflow Summit 2022" />
<meta property="og:description" content="Airflow Summit 2022 is here" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow_summit_2022/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/airflow_summit_2022/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2022-05-16T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Airflow Summit 2022">
-<meta itemprop="description" content="Airflow Summit 2022 is here">
-
-
-<meta itemprop="datePublished" content="2022-05-16T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="129">
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-
-<meta itemprop="keywords" content="Community,Airflow Summit," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Airflow Summit 2022">
+<meta itemprop="description" content="Airflow Summit 2022 is here"><meta itemprop="datePublished" content="2022-05-16T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="129"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Community,Airflow Summit," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Airflow Summit 2022"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -462,28 +454,19 @@
<p class="blogpost-content__metadata--description">Airflow Summit 2022 is here</p>
</div>
<div class="markdown-content">
-
-
-<p>The biggest Airflow Event of the Year returns May 23โ27! Airflow Summit 2022 will bring together the global
+ <p>The biggest Airflow Event of the Year returns May 23โ27! Airflow Summit 2022 will bring together the global
community of Apache Airflow practitioners and data leaders.</p>
-
-<h3 id="what-s-on-the-agenda">Whatโs on the Agenda</h3>
-
+<h3 id="whats-on-the-agenda">Whatโs on the Agenda</h3>
<p>During the free conference, you will hear about Apache Airflow best practices, trends in building data
pipelines, data governance, Airflow and machine learning, and the future of Airflow. There will also be
a series of presentations on non-code contributions driving the open-source project.</p>
-
<h3 id="how-to-attend">How to Attend</h3>
-
<p>This yearโs edition will include a variety of online sessions across different time zones.
Additionally, you can take part in local in-person events organized worldwide for data
communities to watch the event and network.</p>
-
<h3 id="interested">Interested?</h3>
-
-<p>๐ชถ <a href="https://www.crowdcast.io/e/airflowsummit2022/register?utm_campaign=Astronomer_marketing&utm_source=Astronomer%20website&utm_medium=website&utm_term=Airflow%20Summit" target="_blank">Register for Airflow Summit 2022</a> today</p>
-
-<p>๐ค <a href="https://airflowsummit.org/in-person-events/" target="_blank">Check out the in-person events</a> planned for Airflow Summit 2022.</p>
+<p>๐ชถ <a href="https://www.crowdcast.io/e/airflowsummit2022/register?utm_campaign=Astronomer_marketing&utm_source=Astronomer%20website&utm_medium=website&utm_term=Airflow%20Summit">Register for Airflow Summit 2022</a> today</p>
+<p>๐ค <a href="https://airflowsummit.org/in-person-events/">Check out the in-person events</a> planned for Airflow Summit 2022.</p>
</div>
@@ -813,7 +796,7 @@ communities to watch the event and network.</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/announcing-new-website/index.html b/blog/announcing-new-website/index.html
index a6e97ba41d..efdaa88460 100644
--- a/blog/announcing-new-website/index.html
+++ b/blog/announcing-new-website/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>New Airflow website | Apache Airflow</title><meta property="og:title" content="New Airflow website" />
<meta property="og:description" content="We are thrilled about our new website!" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/announcing-new-website/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/announcing-new-website/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="New Airflow website">
-<meta itemprop="description" content="We are thrilled about our new website!">
-
-
-<meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="282">
-
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="keywords" content="Community," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="New Airflow website">
+<meta itemprop="description" content="We are thrilled about our new website!"><meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="282"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Community," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="New Airflow website"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -460,30 +452,25 @@
<p class="blogpost-content__metadata--description">We are thrilled about our new website!</p>
</div>
<div class="markdown-content">
- <p>The brand <a href="https://airflow.apache.org/" target="_blank">new Airflow website</a> has arrived! Those who have been following the process know that the journey to update <a href="https://airflow.readthedocs.io/en/1.10.6/" target="_blank">the old Airflow website</a> started at the beginning of the year.
+ <p>The brand <a href="https://airflow.apache.org/">new Airflow website</a> has arrived! Those who have been following the process know that the journey to update <a href="https://airflow.readthedocs.io/en/1.10.6/">the old Airflow website</a> started at the beginning of the year.
Thanks to sponsorship from the Cloud Composer team at Google that allowed us to
-collaborate with <a href="https://www.polidea.com/" target="_blank">Polidea</a> and with their design studio <a href="https://utilodesign.com/" target="_blank">Utilo</a>, and deliver an awesome website.</p>
-
+collaborate with <a href="https://www.polidea.com/">Polidea</a> and with their design studio <a href="https://utilodesign.com/">Utilo</a>, and deliver an awesome website.</p>
<p>Documentation of open source projects is key to engaging new contributors in the maintenance,
development, and adoption of software. We want the Apache Airflow community to have
the best possible experience to contribute and use the project. We also took this opportunity to make the project
more accessible, and in doing so, increase its reach.</p>
-
<p>In the past three and a half months, we have updated everything: created a more efficient landing page,
enhanced information architecture, and improved UX & UI. Most importantly, the website now has capabilities
to be translated into many languages. This is our effort to foster a more inclusive community around
Apache Airflow, and we look forward to seeing contributions in Spanish, Chinese, Russian, and other languages as well!</p>
-
<p>We built our website on Docsy, a platform that is easy to use and contribute to. Follow
-<a href="https://github.com/apache/airflow-site/blob/master/README.md" target="_blank">these steps</a> to set up your environment and
+<a href="https://github.com/apache/airflow-site/blob/master/README.md">these steps</a> to set up your environment and
to create your first pull request. You may also use
the new website for your own open source project as a template.
-All of our <a href="https://github.com/apache/airflow-site/tree/master" target="_blank">code is open and hosted on GitHub</a>.</p>
-
+All of our <a href="https://github.com/apache/airflow-site/tree/master">code is open and hosted on GitHub</a>.</p>
<p>Share your questions, comments, and suggestions with us, to help us improve the website.
We hope that this new design makes finding documentation about Airflow easier,
and that its improved accessibility increases adoption and use of Apache Airflow around the world.</p>
-
<p>Happy browsing!</p>
</div>
@@ -816,7 +803,7 @@ and that its improved accessibility increases adoption and use of Apache Airflow
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/apache-airflow-for-newcomers/index.html b/blog/apache-airflow-for-newcomers/index.html
index d9e887374e..2b164678e8 100644
--- a/blog/apache-airflow-for-newcomers/index.html
+++ b/blog/apache-airflow-for-newcomers/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -33,24 +33,16 @@
<meta property="og:description" content="Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the process of planning, controlling, and optimizing when a particular task should be done.
Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows using python scripts." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/apache-airflow-for-newcomers/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/apache-airflow-for-newcomers/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2020-08-17T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
+
<meta itemprop="name" content="Apache Airflow For Newcomers">
<meta itemprop="description" content="Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the process of planning, controlling, and optimizing when a particular task should be done.
-Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows using python scripts.">
-
-
-<meta itemprop="datePublished" content="2020-08-17T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="1070">
-
-
-
-<meta itemprop="keywords" content="Community," />
-<meta name="twitter:card" content="summary_large_image"/>
+Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows using python scripts."><meta itemprop="datePublished" content="2020-08-17T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="1070"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Community," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Apache Airflow For Newcomers"/>
@@ -63,8 +55,8 @@ Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -77,8 +69,8 @@ Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -485,78 +477,55 @@ Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows
<p class="blogpost-content__metadata--description"></p>
</div>
<div class="markdown-content">
-
-
-<p>Apache Airflow is a platform to programmatically author, schedule, and monitor workflows.
+ <p>Apache Airflow is a platform to programmatically author, schedule, and monitor workflows.
A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the
path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the
process of planning, controlling, and optimizing when a particular task should be done.</p>
-
<h3 id="authoring-workflow-in-apache-airflow">Authoring Workflow in Apache Airflow.</h3>
-
-<p>Airflow makes it easy to author workflows using python scripts. A <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph" target="_blank">Directed Acyclic Graph</a>
+<p>Airflow makes it easy to author workflows using python scripts. A <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed Acyclic Graph</a>
(DAG) represents a workflow in Airflow. It is a collection of tasks in a way that shows each task’s
relationships and dependencies. You can have as many DAGs as you want, and Airflow will execute
them according to the task’s relationships and dependencies. If task B depends on the successful
execution of another task A, it means Airflow will run task A and only run task B after task A.
This dependency is very easy to express in Airflow. For example, the above scenario is expressed as</p>
-
-<pre><code class="language-python">task_A >> task_B
-</code></pre>
-
-<p>Also equivalent to</p>
-
-<pre><code class="language-python">task_A.set_downstream(task_B)
-</code></pre>
-
-<p><img src="Simple_dag.png" alt="Simple Dag" /></p>
-
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="n">task_A</span> <span class="o">>></span> <span class="n">task_B</span>
+</code></pre></div><p>Also equivalent to</p>
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="n">task_A</span><span class="o">.</span><span class="n">set_downstream</span><span class="p">(</span><span class="n">task_B</span><span class="p">)</span>
+</code></pre></div><p><img src="Simple_dag.png" alt="Simple Dag"></p>
<p>That helps Airflow to know that it needs to execute task A before task B. Tasks can have far more complex
relationships to each other than expressed above and Airflow figures out how and when to execute the tasks following
their relationships and dependencies.
-<img src="semicomplex.png" alt="Complex Dag" /></p>
-
+<img src="semicomplex.png" alt="Complex Dag"></p>
<p>Before we discuss the architecture of Airflow that makes scheduling, executing, and monitoring of
-workflow an easy thing, let us discuss the <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze environment</a>.</p>
-
+workflow an easy thing, let us discuss the <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst">Breeze environment</a>.</p>
<h3 id="breeze-environment">Breeze Environment</h3>
-
<p>The breeze environment is the development environment for Airflow where you can run tests, build images,
build documentations and so many other things. There are excellent
-<a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">documentation and video</a> on Breeze environment.
+<a href="https://github.com/apache/airflow/blob/master/BREEZE.rst">documentation and video</a> on Breeze environment.
Please check them out. You enter the Breeze environment by running the <code>./breeze</code> script. You can run all
the commands mentioned here in the Breeze environment.</p>
-
<h3 id="scheduler">Scheduler</h3>
-
<p>The scheduler is the component that monitors DAGs and triggers those tasks whose dependencies have
been met. It watches over the DAG folder, checking the tasks in each DAG and triggers them once they
are ready. It accomplishes this by spawning a process that runs periodically(every minute or so)
reading the metadata database to check the status of each task and decides what needs to be done.
The metadata database is where the status of all tasks are recorded. The status can be one of running,
- success, failed, etc.</p>
-
+success, failed, etc.</p>
<p>A task is said to be ready when its dependencies have been met. The dependencies include all the data
necessary for the task to be executed. It should be noted that the scheduler won’t trigger your tasks until
the period it covers has ended. If a task’s <code>schedule_interval</code> is <code>@daily</code>, the scheduler triggers the task
at the end of the day and not at the beginning. This is to ensure that the necessary data needed for the tasks
are ready. It is also possible to trigger tasks manually on the UI.</p>
-
-<p>In the <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze environment</a>, the scheduler is started by running the command <code>airflow scheduler</code>. It uses
+<p>In the <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst">Breeze environment</a>, the scheduler is started by running the command <code>airflow scheduler</code>. It uses
the configured production environment. The configuration can be specified in <code>airflow.cfg</code></p>
-
<h3 id="executor">Executor</h3>
-
<p>Executors are responsible for running tasks. They work with the scheduler to get information about
what resources are needed to run a task as the task is queued.</p>
-
-<p>By default, Airflow uses the <a href="https://airflow.apache.org/docs/stable/executor/sequential.html#sequential-executor" target="_blank">SequentialExecutor</a>.
- However, this executor is limited and it is the only executor that can be used with SQLite.</p>
-
-<p>There are many other <a href="https://airflow.apache.org/docs/stable/executor/index.html" target="_blank">executors</a>,
- the difference is on the resources they have and how they choose to use the resources. The available executors
- are:</p>
-
+<p>By default, Airflow uses the <a href="https://airflow.apache.org/docs/stable/executor/sequential.html#sequential-executor">SequentialExecutor</a>.
+However, this executor is limited and it is the only executor that can be used with SQLite.</p>
+<p>There are many other <a href="https://airflow.apache.org/docs/stable/executor/index.html">executors</a>,
+the difference is on the resources they have and how they choose to use the resources. The available executors
+are:</p>
<ul>
<li>Sequential Executor</li>
<li>Debug Executor</li>
@@ -566,71 +535,49 @@ what resources are needed to run a task as the task is queued.</p>
<li>Kubernetes Executor</li>
<li>Scaling Out with Mesos (community contributed)</li>
</ul>
-
<p>CeleryExecutor is a better executor compared to the SequentialExecutor. The CeleryExecutor uses several
workers to execute a job in a distributed way. If a worker node is ever down, the CeleryExecutor assign its
task to another worker node. This ensures high availability.</p>
-
<p>The CeleryExecutor works closely with the scheduler which adds a message to the queue and the Celery broker
which delivers the message to a Celery worker to execute.
You can find more information about the CeleryExecutor and how to configure it at the
-<a href="https://airflow.apache.org/docs/stable/executor/celery.html#celery-executor" target="_blank">documentation</a></p>
-
+<a href="https://airflow.apache.org/docs/stable/executor/celery.html#celery-executor">documentation</a></p>
<h3 id="webserver">Webserver</h3>
-
<p>The webserver is the web interface (UI) for Airflow. The UI is feature-rich. It makes it easy to
monitor and troubleshoot DAGs and Tasks.</p>
-
-<p><img src="airflow-ui.png" alt="airflow UI" /></p>
-
+<p><img src="airflow-ui.png" alt="airflow UI"></p>
<p>There are many actions you can perform on the UI. You can trigger a task, monitor the execution
including the duration of the task. The UI makes it possible to view the task’s dependencies in a
tree view and graph view. You can view task logs in the UI.</p>
-
<p>The web UI is started with the command <code>airflow webserver</code> in the breeze environment.</p>
-
<h3 id="backend">Backend</h3>
-
<p>By default, Airflow uses the SQLite backend for storing the configuration information, DAG states,
and much other useful information. This should not be used in production as SQLite can cause a data
loss.</p>
-
<p>You can use PostgreSQL or MySQL as a backend for airflow. It is easy to change to PostgreSQL or MySQL.</p>
-
<p>The command <code>./breeze --backend mysql</code> selects MySQL as the backend when starting the breeze environment.</p>
-
<h3 id="operators">Operators</h3>
-
<p>Operators determine what gets done by a task. Airflow has a lot of builtin Operators. Each operator
does a specific task. There’s a BashOperator that executes a bash command, the PythonOperator which
-calls a python function, AwsBatchOperator which executes a job on AWS Batch and <a href="https://airflow.apache.org/docs/stable/concepts.html#operators" target="_blank">many more</a>.</p>
-
+calls a python function, AwsBatchOperator which executes a job on AWS Batch and <a href="https://airflow.apache.org/docs/stable/concepts.html#operators">many more</a>.</p>
<h4 id="sensors">Sensors</h4>
-
<p>Sensors can be described as special operators that are used to monitor a long-running task.
Just like Operators, there are many predefined sensors in Airflow. These includes</p>
-
<ul>
<li>AthenaSensor: Asks for the state of the Query until it reaches a failure state or success state.</li>
<li>AzureCosmosDocumentSensor: Checks for the existence of a document which matches the given query in CosmosDB</li>
<li>GoogleCloudStorageObjectSensor: Checks for the existence of a file in Google Cloud Storage</li>
</ul>
-
-<p>A list of most of the available sensors can be found in this <a href="https://airflow.apache.org/docs/stable/_api/airflow/contrib/sensors/index.html?highlight=sensors#module-airflow.contrib.sensors" target="_blank">module</a></p>
-
+<p>A list of most of the available sensors can be found in this <a href="https://airflow.apache.org/docs/stable/_api/airflow/contrib/sensors/index.html?highlight=sensors#module-airflow.contrib.sensors">module</a></p>
<h3 id="contributing-to-airflow">Contributing to Airflow</h3>
-
<p>Airflow is an open source project, everyone is welcome to contribute. It is easy to get started thanks
-to the excellent <a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst" target="_blank">documentation on how to get started</a>.</p>
-
-<p>I joined the community about 12 weeks ago through the <a href="https://www.outreachy.org/" target="_blank">Outreachy Program</a> and have
-completed about <a href="https://github.com/apache/airflow/pulls/ephraimbuddy" target="_blank">40 PRs</a>.</p>
-
-<p>It has been an amazing experience! Thanks to my mentors <a href="https://github.com/potiuk" target="_blank">Jarek</a> and
-<a href="https://github.com/kaxil" target="_blank">Kaxil</a>, and the community members especially <a href="https://github.com/mik-laj" target="_blank">Kamil</a>
-and <a href="https://github.com/turbaszek" target="_blank">Tomek</a> for all their support. I’m grateful!</p>
-
-<p>Thank you so much, <a href="https://github.com/leahecole" target="_blank">Leah E. Cole</a>, for your wonderful reviews.</p>
+to the excellent <a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst">documentation on how to get started</a>.</p>
+<p>I joined the community about 12 weeks ago through the <a href="https://www.outreachy.org/">Outreachy Program</a> and have
+completed about <a href="https://github.com/apache/airflow/pulls/ephraimbuddy">40 PRs</a>.</p>
+<p>It has been an amazing experience! Thanks to my mentors <a href="https://github.com/potiuk">Jarek</a> and
+<a href="https://github.com/kaxil">Kaxil</a>, and the community members especially <a href="https://github.com/mik-laj">Kamil</a>
+and <a href="https://github.com/turbaszek">Tomek</a> for all their support. I’m grateful!</p>
+<p>Thank you so much, <a href="https://github.com/leahecole">Leah E. Cole</a>, for your wonderful reviews.</p>
</div>
@@ -962,7 +909,7 @@ and <a href="https://github.com/turbaszek" target="_blank">Tomek</a> for all the
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html b/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
index bf4233c586..5417ff5886 100644
--- a/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
+++ b/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>ApacheCon Europe 2019 โ Thoughts and Insights by Airflow Committers | Apache Airflow</title><meta property="og:title" content="ApacheCon Europe 2019 โ Thoughts and Insights by Airflow Committers" />
<meta property="og:description" content="Here come some thoughts by Airflow committers and contributors from the ApacheCon Europe 2019. Get to know the ASF community!" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="ApacheCon Europe 2019 โ Thoughts and Insights by Airflow Committers">
-<meta itemprop="description" content="Here come some thoughts by Airflow committers and contributors from the ApacheCon Europe 2019. Get to know the ASF community!">
-
-
-<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="138">
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-<meta itemprop="keywords" content="Community," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="ApacheCon Europe 2019 โ Thoughts and Insights by Airflow Committers">
+<meta itemprop="description" content="Here come some thoughts by Airflow committers and contributors from the ApacheCon Europe 2019. Get to know the ASF community!"><meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="138"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Community," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="ApacheCon Europe 2019 โ Thoughts and Insights by Airflow Committers"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -461,10 +453,8 @@
</div>
<div class="markdown-content">
<p>Is it possible to create an organization that delivers tens of projects used by millions, nearly no one is paid for doing their job, and still, it has been fruitfully carrying on for more than 20 years? Apache Software Foundation proves it is possible. For the last two decades, ASF has been crafting a model called the Apache Wayโa way of organizing and leading tech open source projects. Due to this approach, which is strongly based on the โcommunity over codeโ motto, we can enjoy [...]
-
<p>After this yearโs ApacheCon, Polideaโs engineers talked with Committers of Apache projects, such asโAizhamal Nurmamat kyzy, Felix Uellendall, and Fokko Driesprongโabout insights to what makes the ASF such an amazing organization.</p>
-
-<p>You can read the <a href="https://higrys.medium.com/apachecon-europe-2019-thoughts-and-insights-by-airflow-committers-9ff5f6938c99" target="_blank">insights after the ApacheCon 2019</a>.</p>
+<p>You can read the <a href="https://higrys.medium.com/apachecon-europe-2019-thoughts-and-insights-by-airflow-committers-9ff5f6938c99">insights after the ApacheCon 2019</a>.</p>
</div>
@@ -796,7 +786,7 @@
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/documenting-using-local-development-environments/index.html b/blog/documenting-using-local-development-environments/index.html
index a1a92ae4bb..e4f961e22d 100644
--- a/blog/documenting-using-local-development-environments/index.html
+++ b/blog/documenting-using-local-development-environments/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Documenting using local development environment | Apache Airflow</title><meta property="og:title" content="Documenting using local development environment" />
<meta property="og:description" content="The story behind documenting local development environment of Apache Airflow" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/documenting-using-local-development-environments/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/documenting-using-local-development-environments/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Documenting using local development environment">
-<meta itemprop="description" content="The story behind documenting local development environment of Apache Airflow">
-
-
-<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="256">
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-
-<meta itemprop="keywords" content="Development," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Documenting using local development environment">
+<meta itemprop="description" content="The story behind documenting local development environment of Apache Airflow"><meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="256"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Development," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Documenting using local development environment"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -480,18 +472,12 @@
<p class="blogpost-content__metadata--description">The story behind documenting local development environment of Apache Airflow</p>
</div>
<div class="markdown-content">
-
-
-<h2 id="documenting-local-development-environment-of-apache-airflow">Documenting local development environment of Apache Airflow</h2>
-
-<p>From Sept to November, 2019 I have been participating in a wonderful initiative, <a href="https://developers.google.com/season-of-docs" target="_blank">Google Season of Docs</a>.</p>
-
+ <h2 id="documenting-local-development-environment-of-apache-airflow">Documenting local development environment of Apache Airflow</h2>
+<p>From Sept to November, 2019 I have been participating in a wonderful initiative, <a href="https://developers.google.com/season-of-docs">Google Season of Docs</a>.</p>
<p>I had a pleasure to contribute to the Apache Airflow open source project as a technical writer.
My initial assignment was an extension to the GitHub-based Contribution guide.</p>
-
<p>From the very first days I have been pretty closely involved into inter-project communications
via emails/slack and had regular 1:1s with my mentor, Jarek Potiuk.</p>
-
<p>I got infected with Jarekโs enthusiasm to ease the on-boarding experience for
Airflow contributors. I do share this strategy and did my best to improve the structure,
language and DX. As a result, Jarek and I extended the current contributorโs docs and
@@ -500,21 +486,18 @@ infrastructure and providing a workflow example based on a real-life use case;
the Testing guide with an overview of a complex testing infrastructure for Apache Airflow;
and two guides dedicated to the Breeze dev environment and local virtual environment
(my initial assignment).</p>
-
<p>Iโm deeply grateful to my mentor and Airflow developers for their feedback,
patience and help while I was breaking through new challenges
(Iโve never worked on an open source project before),
and for their support of all my ideas! I think a key success factor for any contributor
is a responsive, supportive and motivated team, and I was lucky to join such
a team for 3 months.</p>
-
<p>Documents I worked on:</p>
-
<ul>
-<li><a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze development environment documentation</a></li>
-<li><a href="https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst" target="_blank">Local virtualenv environment documentation</a></li>
-<li><a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst" target="_blank">Contributing guide</a></li>
-<li><a href="https://github.com/apache/airflow/blob/master/TESTING.rst" target="_blank">Testing guide</a></li>
+<li><a href="https://github.com/apache/airflow/blob/master/BREEZE.rst">Breeze development environment documentation</a></li>
+<li><a href="https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst">Local virtualenv environment documentation</a></li>
+<li><a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst">Contributing guide</a></li>
+<li><a href="https://github.com/apache/airflow/blob/master/TESTING.rst">Testing guide</a></li>
</ul>
</div>
@@ -808,7 +791,7 @@ a team for 3 months.</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html b/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
index 2a3fd8c097..31719b1432 100644
--- a/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
+++ b/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -33,24 +33,16 @@
<meta property="og:description" content="I came across Google Season of Docs (GSoD) almost by accident, thanks to my extensive HackerNews and Twitter addiction. I was familiar with the Google Summer of Code but not with this program. It turns out it was the inaugural phase. I read the details, and the process felt a lot like GSoC except that this was about documentation.
About Me I have been writing tech articles on medium as well as my blog for the past 1." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2019-12-20T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
+
<meta itemprop="name" content="Experience in Google Season of Docs 2019 with Apache Airflow">
<meta itemprop="description" content="I came across Google Season of Docs (GSoD) almost by accident, thanks to my extensive HackerNews and Twitter addiction. I was familiar with the Google Summer of Code but not with this program. It turns out it was the inaugural phase. I read the details, and the process felt a lot like GSoC except that this was about documentation.
-About Me I have been writing tech articles on medium as well as my blog for the past 1.">
-
-
-<meta itemprop="datePublished" content="2019-12-20T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="1521">
-
-
-
-<meta itemprop="keywords" content="Documentation," />
-<meta name="twitter:card" content="summary_large_image"/>
+About Me I have been writing tech articles on medium as well as my blog for the past 1."><meta itemprop="datePublished" content="2019-12-20T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="1521"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Documentation," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Experience in Google Season of Docs 2019 with Apache Airflow"/>
@@ -63,8 +55,8 @@ About Me I have been writing tech articles on medium as well as my blog for the
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -77,8 +69,8 @@ About Me I have been writing tech articles on medium as well as my blog for the
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -498,124 +490,91 @@ About Me I have been writing tech articles on medium as well as my blog for the
<p class="blogpost-content__metadata--description"></p>
</div>
<div class="markdown-content">
-
-
-<p>I came across <a href="https://developers.google.com/season-of-docs" target="_blank">Google Season of Docs</a> (GSoD) almost by accident, thanks to my extensive HackerNews and Twitter addiction. I was familiar with the Google Summer of Code but not with this program.
+ <p>I came across <a href="https://developers.google.com/season-of-docs">Google Season of Docs</a> (GSoD) almost by accident, thanks to my extensive HackerNews and Twitter addiction. I was familiar with the Google Summer of Code but not with this program.
It turns out it was the inaugural phase. I read the details, and the process felt a lot like GSoC except that this was about documentation.</p>
-
<h2 id="about-me">About Me</h2>
-
<p>I have been writing tech articles on medium as well as my blog for the past 1.5 years. Blogging helps me test my understanding of the concepts as untangling the toughest of ideas in simple sentences requires a considerable time investment.</p>
-
<p>Also, I have been working as a Software Developer for the past three years, which involves writing documentation for my projects as well. I completed my B.Tech from IIT Roorkee. During my stay in college, I applied for GSoC once but didnโt make it through in the final list of selected candidates.</p>
-
<p>I saw GSoD as an excellent opportunity to improve my technical writing skills using feedback from the open-source community. I contributed some bug fixes and features to Apache Superset and Apache Druid, but this would be my first contribution as a technical writer.</p>
-
<h2 id="searching-for-the-organization">Searching for the organization</h2>
-
-<p>About 40+ organizations were participating in the GSoD. However, there were two which came as the right choice for me in the first instant. The first one was <a href="https://airflow.apache.org/" target="_blank">Apache Airflow</a> because I had already used Airflow extensively and also contributed some custom operators inside the forked version of my previous company.</p>
-
-<p>The second one was <a href="http://cassandra.apache.org/" target="_blank">Apache Cassandra</a>, on which I also had worked extensively but hadnโt done any code or doc changes.</p>
-
+<p>About 40+ organizations were participating in the GSoD. However, there were two which came as the right choice for me in the first instant. The first one was <a href="https://airflow.apache.org/">Apache Airflow</a> because I had already used Airflow extensively and also contributed some custom operators inside the forked version of my previous company.</p>
+<p>The second one was <a href="http://cassandra.apache.org/">Apache Cassandra</a>, on which I also had worked extensively but hadnโt done any code or doc changes.</p>
<p>Considering the total experience, I decided to go with the Airflow.</p>
-
<h2 id="project-selection">Project selection</h2>
-
-<p>After selecting the org, the next step was to choose the project. Again, my previous experience played a role here, and I ended up picking the <strong>How to create a workflow</strong> . The aim of the project was to write documentation which will help users in creating complex as well as custom DAGs.<br />
+<p>After selecting the org, the next step was to choose the project. Again, my previous experience played a role here, and I ended up picking the <strong>How to create a workflow</strong> . The aim of the project was to write documentation which will help users in creating complex as well as custom DAGs.<br>
The final deliverables were a bit different, though. More on that later.</p>
-
<p>After submitting my application, I got involved in my job until one day, I saw a mail from google confirming my selection as a Technical Writer for the project.</p>
-
<h2 id="community-bonding">Community Bonding</h2>
-
<p>Getting selected is just a beginning. I got the invite to the Airflow slack channel where most of the discussions happened.
-My mentor was <a href="https://github.com/ashb" target="_blank">Ash-Berlin Taylor</a> from Apache Airflow. I started talking to my mentor to get a general sense of what deliverables were expected. The deliverables were documented in <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Season+of+Docs+2019" target="_blank">confluence</a>.</p>
-
+My mentor was <a href="https://github.com/ashb">Ash-Berlin Taylor</a> from Apache Airflow. I started talking to my mentor to get a general sense of what deliverables were expected. The deliverables were documented in <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Season+of+Docs+2019">confluence</a>.</p>
<ul>
<li>A page for how to create a DAG that also includes:
-
<ul>
<li>Revamping the page related to scheduling a DAG</li>
<li>Adding tips for specific DAG conditions, such as rerunning a failed task</li>
-</ul></li>
+</ul>
+</li>
<li>A page for developing custom operators that includes:
-
<ul>
<li>Describing mechanisms that are important when creating an operator, such as template fields, UI color, hooks, connection, etc.</li>
<li>Describing the responsibility between the operator and the hook</li>
<li>Considerations for dealing with shared resources (such as connections and hooks)</li>
-</ul></li>
+</ul>
+</li>
<li>A page that describes how to define the relationships between tasks. The page should include information about:
-
<ul>
-<li><strong>ย >> <<ย </strong></li>
+<li>**ย >> <<ย **</li>
<li>set upstream / set downstream</li>
<li>helpers method ex. chain</li>
-</ul></li>
+</ul>
+</li>
<li>A page that describes the communication between tasks that also includes:
-
<ul>
<li>Revamping the page related to macros and XCOM</li>
-</ul></li>
</ul>
-
+</li>
+</ul>
<p>My mentor set the expectation early on that the deliverables were sort of like guidelines and not strict rules.
If I wanted to, I could choose to work on something else related to the project also, which was not under deliverables.
-After connecting with the mentor, I started engaging with the overall Airflow community. The people in the community were helpful, especially <a href="https://github.com/mik-laj" target="_blank">Kamil Bregula</a>. Kamil helped me in getting started with the guidelines to follow while writing the documentation for Airflow.</p>
-
+After connecting with the mentor, I started engaging with the overall Airflow community. The people in the community were helpful, especially <a href="https://github.com/mik-laj">Kamil Bregula</a>. Kamil helped me in getting started with the guidelines to follow while writing the documentation for Airflow.</p>
<h2 id="doc-development">Doc Development</h2>
-
<p>I picked DAG run as my first deliverable. I chose this topic as some parts of it were already documented but needed some additional text.
-I splitter the existing Scheduling & Triggers page into two new pages.
-1. Schedulers
-2. DAG Runs</p>
-
+I splitter the existing Scheduling & Triggers page into two new pages.</p>
+<ol>
+<li>Schedulers</li>
+<li>DAG Runs</li>
+</ol>
<p>Most of the details unrelated to schedulers were moved to DAG runs page, and then missing points such as how to re-run a task or DAG were added.
Once I was satisfied with my version, I asked my mentor and Kamil to review it. For the first version, I shared the text in the Google docs file in which the reviewers added comments.
However, the document started getting messy, and it became difficult to track the changes. The time had come now to raise a proper Pull Request.</p>
-
<p>This was the time when I faced my first challenge. The documentation of Apache Airflow is written using RST(reStructuredText) syntax, with which I was entirely unfamiliar. I had mostly worked in Markdown.
I spent the next couple of days understanding the syntax. Fortunately, it was quite easy to get acquainted.
-I raised the <a href="https://github.com/apache/airflow/pull/6295" target="_blank">Pull Request</a> and waited for the comments. Finally, after a few days when I saw the comments, they were mostly related to two things - grammar and formatting. There were also comments related to what I had missed or misinterpreted.</p>
-
+I raised the <a href="https://github.com/apache/airflow/pull/6295">Pull Request</a> and waited for the comments. Finally, after a few days when I saw the comments, they were mostly related to two things - grammar and formatting. There were also comments related to what I had missed or misinterpreted.</p>
<h3 id="using-correct-grammar">Using correct grammar</h3>
-
-<p>After discussing with Kamil, I decided to follow <a href="https://developers.google.com/style/" target="_blank">Googleโs Developer Documentation Guidelines</a>. These guidelines contain almost everything youโll need to consider while writing good documentation, such as always to use active voice.
+<p>After discussing with Kamil, I decided to follow <a href="https://developers.google.com/style/">Googleโs Developer Documentation Guidelines</a>. These guidelines contain almost everything youโll need to consider while writing good documentation, such as always to use active voice.
Secondly, I installed the Grammarly app. After writing a doc, I used to put it in Grammarly to check for errors. Then I corrected the errors, made some more changes, and then again pushed it to Grammarly. This was an iterative process until I arrived with a version of the doc, which was grammatically correct but not seemed to have been written by an AI.</p>
-
<h3 id="formatting">Formatting</h3>
-
<p>Formatting involves writing notes and tips, marking the airflow components correctly in the text, and making sure a user who is skimming through the docs doesnโt miss the critical text.
This required a bit of trial and error. I studied the current pattern in Airflow docs and made changes, pushed commits, incorporated new review comments, and then so on.</p>
-
<p>In the end, all the reviewers approved the PR, but it was not merged until two months later. This was because we doubted if some more pages, such as <strong>Concepts</strong>, should also be split up, resulting in a better-structured document. In the end, we decided to delay it until we discussed it with the broader community.</p>
-
-<p>My <a href="https://github.com/apache/airflow/pull/6348" target="_blank">second PR</a> was a completely new document. It was related to How to create your custom operator. For this, since now I was familiar with most of the syntax, I directly raised the PR without going via Google docs. I received a lot of comments again, but this time they were more related to what I had written rather than how I had written it.
+<p>My <a href="https://github.com/apache/airflow/pull/6348">second PR</a> was a completely new document. It was related to How to create your custom operator. For this, since now I was familiar with most of the syntax, I directly raised the PR without going via Google docs. I received a lot of comments again, but this time they were more related to what I had written rather than how I had written it.
e.g., Describing in detail how to use <strong>template fields</strong> and clean up my code examples. The fewer grammatical & formatting error comments showed I had made progress.
The PR was accepted within two weeks and gave me a huge confidence boost.</p>
-
<p>After my second PR, I was in a bit of a deadlock. My last remaining deliverable was related to <strong>Macros</strong>, but the scope wasnโt clear. I talked to my mentor, and he told me he didnโt mind if I can go off-track to work on something else while the community figured out what changes were needed.
-We discussed a lot of ideas. In the end, I decided to go with the Best Practices guide inspired by my mentorsโ <a href="https://drive.google.com/file/d/1E4zle8-fv5S1rrlcNUzjiEV19OMYvwoY/view?usp=sharing" target="_blank">talk on Apache Airflowย </a>in a meetup. Having faced challenges while running Airflow in production myself, I was highly motivated to write something like this so that other developers donโt suffer.
+We discussed a lot of ideas. In the end, I decided to go with the Best Practices guide inspired by my mentorsโ <a href="https://drive.google.com/file/d/1E4zle8-fv5S1rrlcNUzjiEV19OMYvwoY/view?usp=sharing">talk on Apache Airflowย </a>in a meetup. Having faced challenges while running Airflow in production myself, I was highly motivated to write something like this so that other developers donโt suffer.
The first draft was ready within two weeks. I called it <strong>Running Airflow in Production</strong>. However, after adding a few more pieces to the document, I realized it was better to call it <strong>Best Practices</strong> guide, which most of the open-source projects contained.</p>
-
-<p>People were enthusiastic about this <a href="https://github.com/apache/airflow/pull/6515" target="_blank">pull request</a> since a lot of them faced the challenges described in the doc. I had hit the nail on the head. After some deliberation over the next 1-2 weeks, my PR got accepted.</p>
-
+<p>People were enthusiastic about this <a href="https://github.com/apache/airflow/pull/6515">pull request</a> since a lot of them faced the challenges described in the doc. I had hit the nail on the head. After some deliberation over the next 1-2 weeks, my PR got accepted.</p>
<p>I then returned to my first PR and started making some changes related to the new review comments. After this, I discussed with my mentor about specific elements that were bugging him, such as getting people to understand how the schedule interval works in as few words as possible.
After a lot of trial and error, we arrived at a version with which both of us could make peace.</p>
-
<h2 id="final-evaluation">Final Evaluation</h2>
-
<p>On 12th September, I received mail from Google about the successful completion of the project. This meant my mentor liked my work. The Airflow community also appreciated the contributions.</p>
-
<p>My documents were finally published on Airflow website -</p>
-
<ul>
-<li><a href="https://airflow.readthedocs.io/en/latest/dag-run.html" target="_blank">DAG Runs</a></li>
-<li><a href="https://airflow.readthedocs.io/en/latest/scheduler.html" target="_blank">Scheduler</a></li>
-<li><a href="https://airflow.readthedocs.io/en/latest/howto/custom-operator.html" target="_blank">Creating a custom operator</a></li>
-<li><a href="https://airflow.readthedocs.io/en/latest/best-practices.html" target="_blank">Best Practices</a></li>
+<li><a href="https://airflow.readthedocs.io/en/latest/dag-run.html">DAG Runs</a></li>
+<li><a href="https://airflow.readthedocs.io/en/latest/scheduler.html">Scheduler</a></li>
+<li><a href="https://airflow.readthedocs.io/en/latest/howto/custom-operator.html">Creating a custom operator</a></li>
+<li><a href="https://airflow.readthedocs.io/en/latest/best-practices.html">Best Practices</a></li>
</ul>
-
<p>I also started getting invited in the PR reviews of other developers. I am looking forward to more contributions to the project in the coming year.</p>
</div>
@@ -865,7 +824,7 @@ After a lot of trial and error, we arrived at a version with which both of us co
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/experience-with-airflow-as-an-outreachy-intern/index.html b/blog/experience-with-airflow-as-an-outreachy-intern/index.html
index ec99dd1dc2..cda940484f 100644
--- a/blog/experience-with-airflow-as-an-outreachy-intern/index.html
+++ b/blog/experience-with-airflow-as-an-outreachy-intern/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -33,24 +33,16 @@
<meta property="og:description" content="Outreachy is a program which organises three months paid internships with FOSS projects for people who are typically underrepresented in those projects.
Contribution Period The first thing I had to do was choose a project under an organisation. After going through all the projects I chose โExtending the REST API of Apache Airflowโ, because I had a good idea of what REST API(s) are, so I thought it would be easier to get started with the contributions." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/experience-with-airflow-as-an-outreachy-intern/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/experience-with-airflow-as-an-outreachy-intern/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2020-08-30T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
+
<meta itemprop="name" content="Journey with Airflow as an Outreachy Intern">
<meta itemprop="description" content="Outreachy is a program which organises three months paid internships with FOSS projects for people who are typically underrepresented in those projects.
-Contribution Period The first thing I had to do was choose a project under an organisation. After going through all the projects I chose โExtending the REST API of Apache Airflowโ, because I had a good idea of what REST API(s) are, so I thought it would be easier to get started with the contributions.">
-
-
-<meta itemprop="datePublished" content="2020-08-30T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="532">
-
-
-
-<meta itemprop="keywords" content="Community," />
-<meta name="twitter:card" content="summary_large_image"/>
+Contribution Period The first thing I had to do was choose a project under an organisation. After going through all the projects I chose โExtending the REST API of Apache Airflowโ, because I had a good idea of what REST API(s) are, so I thought it would be easier to get started with the contributions."><meta itemprop="datePublished" content="2020-08-30T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="528"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="Community," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Journey with Airflow as an Outreachy Intern"/>
@@ -63,8 +55,8 @@ Contribution Period The first thing I had to do was choose a project under an or
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -77,8 +69,8 @@ Contribution Period The first thing I had to do was choose a project under an or
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -483,57 +475,53 @@ Contribution Period The first thing I had to do was choose a project under an or
<p class="blogpost-content__metadata--description"></p>
</div>
<div class="markdown-content">
-
-
-<p><a href="https://www.outreachy.org/" target="_blank">Outreachy</a> is a program which organises three months paid internships with FOSS
+ <p><a href="https://www.outreachy.org/">Outreachy</a> is a program which organises three months paid internships with FOSS
projects for people who are typically underrepresented in those projects.</p>
-
<h3 id="contribution-period">Contribution Period</h3>
-
<p>The first thing I had to do was choose a project under an organisation. After going through all the projects
I chose โExtending the REST API of Apache Airflowโ, because I had a good idea of what REST API(s) are, so I
thought it would be easier to get started with the contributions. The next step was to set up Airflowโs dev
-environment which thanks to <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst" target="_blank">Breeze</a>, was a breeze.
+environment which thanks to <a href="https://github.com/apache/airflow/blob/master/BREEZE.rst">Breeze</a>, was a breeze.
Since I had never contributed to FOSS before so this part was overwhelming but there were plenty of issues
labelled โgood first issuesโ with detailed descriptions and some even had code snippets so luckily that nudged
me in the right direction. These things about Airflow and the positive vibes from the community were the reasons
why I chose to stick with Airflow as my Outreachy project.</p>
-
<h3 id="internship-period">Internship Period</h3>
-
<p>My first PR was followed by many new experiences one of them being that I introduced a
-<a href="https://github.com/apache/airflow/pull/7680#issuecomment-619763051" target="_blank">bug</a> in it;).
+<a href="https://github.com/apache/airflow/pull/7680#issuecomment-619763051">bug</a> in it;).
But nonetheless it made me familiar with the feedback loop and the feedback on my subsequent
-<a href="https://github.com/apache/airflow/pulls?q=is%3Apr+author%3AOmairK+" target="_blank">PRs</a> was the focal point of the overall
+<a href="https://github.com/apache/airflow/pulls?q=is%3Apr+author%3AOmairK+">PRs</a> was the focal point of the overall
learning experience I went through, which boosted my confidence to contribute more and move out of my comfort zone.
I wanted to learn more about the things that happen under the Airflowโs hood so I started filtering out recent PRs
dealing with different components and I would go through the code changes along with discussion that would help me
-get a better understanding of the whole workflow. <a href="https://lists.apache.org/list.html?dev@airflow.apache.org" target="_blank">Airflowโs mailing list</a>
+get a better understanding of the whole workflow. <a href="https://lists.apache.org/list.html?dev@airflow.apache.org">Airflowโs mailing list</a>
was also a great source of knowledge.</p>
-
<p>The API related PRs that I worked on helped me with some of the important concepts like:</p>
-
-<p>1) <a href="https://github.com/apache/airflow/pull/9329" target="_blank">Pool CRUD endpoints</a> where pools limit the execution parallelism.</p>
-
-<p>2) <a href="https://github.com/apache/airflow/pull/9597" target="_blank">Tasks</a> determine the actual work that has to be carried out.</p>
-
-<p>3) <a href="https://github.com/apache/airflow/pull/9473" target="_blank">DAG</a> which represents the structure for a collection
- of tasks. It keeps track of tasks, their dependencies and the sequence in which they have to run.</p>
-
-<p>4) <a href="https://github.com/apache/airflow/pull/9473" target="_blank">Dag Runs</a> that are the instantiation of DAG(s) in time.</p>
-
+<ol>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9329">Pool CRUD endpoints</a> where pools limit the execution parallelism.</p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9597">Tasks</a> determine the actual work that has to be carried out.</p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9473">DAG</a> which represents the structure for a collection
+of tasks. It keeps track of tasks, their dependencies and the sequence in which they have to run.</p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9473">Dag Runs</a> that are the instantiation of DAG(s) in time.</p>
+</li>
+</ol>
<p>Through actively and passively participating in discussions I learnt that even if there is a difference of opinion
-one could always learn from the different approaches, and <a href="https://github.com/apache/airflow/pull/8721" target="_blank">this PR</a> with
+one could always learn from the different approaches, and <a href="https://github.com/apache/airflow/pull/8721">this PR</a> with
more than 300+ comments is the proof of it. I also started reviewing small PRs which gave me the amazing opportunity
to interact with new people. Throughout my internship I learnt a lot about different frameworks and technologies
but the biggest takeaway for me was that a code is read more often than it’s written, and I started writing code with
that in mind.</p>
-
<h3 id="wrapping-up">Wrapping Up</h3>
-
<p>So with my project of extending Airflowโs REST API as well as the Outreachy internship coming to an end I would like
-to thank my mentors <a href="https://github.com/potiuk" target="_blank">Jarek Potiuk</a>, <a href="https://github.com/kaxil" target="_blank">Kaxil Naik</a> and
-<a href="https://github.com/mik-laj" target="_blank">Kamil Breguลa</a> for the patience and the time they invested in mentoring me and
+to thank my mentors <a href="https://github.com/potiuk">Jarek Potiuk</a>, <a href="https://github.com/kaxil">Kaxil Naik</a> and
+<a href="https://github.com/mik-laj">Kamil Breguลa</a> for the patience and the time they invested in mentoring me and
the Airflow community for making me feel so welcomed. I plan to stick around and contribute to give back to the
community that has been made my summer, one to remember.</p>
@@ -867,7 +855,7 @@ community that has been made my summer, one to remember.</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/implementing-stable-api-for-apache-airflow/index.html b/blog/implementing-stable-api-for-apache-airflow/index.html
index 093f23b3ec..20290cc7fe 100644
--- a/blog/implementing-stable-api-for-apache-airflow/index.html
+++ b/blog/implementing-stable-api-for-apache-airflow/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -32,23 +32,15 @@
<title>Implementing Stable API for Apache Airflow | Apache Airflow</title><meta property="og:title" content="Implementing Stable API for Apache Airflow" />
<meta property="og:description" content="An Outreachy intern's progress report on contributing to Apache Airflow REST API." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/implementing-stable-api-for-apache-airflow/" />
-
-<meta property="og:image" content="/images/feature-image.png" />
+<meta property="og:url" content="/blog/implementing-stable-api-for-apache-airflow/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="article:section" content="blog" />
<meta property="article:published_time" content="2020-07-19T00:00:00+00:00" />
-<meta property="article:modified_time" content="2022-07-12T07:00:28-06:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Implementing Stable API for Apache Airflow">
-<meta itemprop="description" content="An Outreachy intern's progress report on contributing to Apache Airflow REST API.">
-
-
-<meta itemprop="datePublished" content="2020-07-19T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2022-07-12T07:00:28-06:00" />
-<meta itemprop="wordCount" content="703">
-
+<meta property="article:modified_time" content="2022-07-12T12:22:50-04:00" /><meta property="og:site_name" content="Apache Airflow" />
-
-<meta itemprop="keywords" content="REST API," />
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="name" content="Implementing Stable API for Apache Airflow">
+<meta itemprop="description" content="An Outreachy intern's progress report on contributing to Apache Airflow REST API."><meta itemprop="datePublished" content="2020-07-19T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2022-07-12T12:22:50-04:00" />
+<meta itemprop="wordCount" content="703"><meta itemprop="image" content="/images/feature-image.png"/>
+<meta itemprop="keywords" content="REST API," /><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Implementing Stable API for Apache Airflow"/>
@@ -60,8 +52,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -74,8 +66,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -480,93 +472,79 @@
<p class="blogpost-content__metadata--description">An Outreachy intern's progress report on contributing to Apache Airflow REST API.</p>
</div>
<div class="markdown-content">
-
-
-<p>My <a href="https://outreachy.org" target="_blank">Outreachy internship</a> is coming to its ends which is also the best time to look back and
+ <p>My <a href="https://outreachy.org">Outreachy internship</a> is coming to its ends which is also the best time to look back and
reflect on the progress so far.</p>
-
<p>The goal of my project is to Extend and Improve the Apache Airflow REST API. In this post,
I will be sharing my progress so far.</p>
-
<p>We started a bit late implementing the REST API because it took time for the OpenAPI 3.0
-specification we were to use for the project to be merged. Thanks to <a href="https://github.com/mik-laj" target="_blank">Kamil</a>,
+specification we were to use for the project to be merged. Thanks to <a href="https://github.com/mik-laj">Kamil</a>,
who paved the way for us to start implementing the REST API endpoints. Below are the endpoints I
implemented and the challenges I encountered, including how I overcame them.</p>
-
<h3 id="implementing-the-read-only-connection-endpoints">Implementing The Read-Only Connection Endpoints</h3>
-
-<p>The <a href="https://github.com/apache/airflow/pull/9095" target="_blank">read-only connection endpoints</a> were the first endpoint I implemented. Looking back,
+<p>The <a href="https://github.com/apache/airflow/pull/9095">read-only connection endpoints</a> were the first endpoint I implemented. Looking back,
I can see how much I have improved.</p>
-
-<p>I started by implementing the database schema for the Connection table using <a href="https://marshmallow.readthedocs.io/en/2.x-line/" target="_blank">Marshmallow 2</a>.
+<p>I started by implementing the database schema for the Connection table using <a href="https://marshmallow.readthedocs.io/en/2.x-line/">Marshmallow 2</a>.
We had to use Marshmallow 2 because Flask-AppBuilder was still using it and Flask-AppBuilder
is deeply integrated to Apache Airflow. This meant I had to unlearn Marshmallow 3 that I had
- been studying before this realization, but thankfully, <a href="https://marshmallow.readthedocs.io/en/stable/index.html" target="_blank">Marshmallow 3</a> isn’t too
- different, so I was able to start using Marshmallow 2 in no time.</p>
-
+been studying before this realization, but thankfully, <a href="https://marshmallow.readthedocs.io/en/stable/index.html">Marshmallow 3</a> isn’t too
+different, so I was able to start using Marshmallow 2 in no time.</p>
<p>This first PR would have been more difficult than it was unless there had been any reference
-endpoint to look at. <a href="https://github.com/mik-laj" target="_blank">Kamil</a> implemented a <a href="https://github.com/apache/airflow/pull/9045" target="_blank">draft PR</a> in which I took inspiration from.
+endpoint to look at. <a href="https://github.com/mik-laj">Kamil</a> implemented a <a href="https://github.com/apache/airflow/pull/9045">draft PR</a> in which I took inspiration from.
Thanks to this, It was easy for me to write the unit tests. It was also in this endpoint that
- I learned using <a href="https://github.com/wolever/parameterized" target="_blank">parameterized</a> in unit tests :D.</p>
-
+I learned using <a href="https://github.com/wolever/parameterized">parameterized</a> in unit tests :D.</p>
<h3 id="implementing-the-read-only-dagruns-endpoints">Implementing The Read-Only DagRuns Endpoints</h3>
-
-<p>This <a href="https://github.com/apache/airflow/pull/9153" target="_blank">endpoint</a> came with its many challenges, especially on filtering with <code>datetimes</code>.
+<p>This <a href="https://github.com/apache/airflow/pull/9153">endpoint</a> came with its many challenges, especially on filtering with <code>datetimes</code>.
This was because the <code>connexion</code> library we were using to build the REST API was not validating
date-time format in OpenAPI 3.0 specification, what I eventually found out, was intentional.
Connexion dropped <code>strict-rfc3339</code> because of the later license which is not compatible with
Apache 2.0 license.</p>
-
<p>I implemented a workaround on this, by defining a function called <code>conn_parse_datetime</code> in the
-API utils module. This was later refactored and thankfully, <a href="https://github.com/mik-laj" target="_blank">Kamil</a>
- implemented a decorator that allowed us to have cleaner code on the views while using this function.</p>
-
+API utils module. This was later refactored and thankfully, <a href="https://github.com/mik-laj">Kamil</a>
+implemented a decorator that allowed us to have cleaner code on the views while using this function.</p>
<p>Then we tried using <code>rfc3339-validator</code> whose license is compatible with Apache 2.0 licence but
- later discarded this because with our custom date parser we were able to use duration and
- not just date times.</p>
-
+later discarded this because with our custom date parser we were able to use duration and
+not just date times.</p>
<h3 id="other-endpoints">Other Endpoints</h3>
-
<p>I implemented some different other endpoints. One peculiar issue I faced was because of Marshmallow 2
not giving error when extra fields are in the request body. I implemented a <code>validate_unknown</code>
method on the schema to handle this. Thankfully, Flask-AppBuilder updated to using Marshmallow 3,
we quickly updated Flask-AppBuilder in Apache Airflow and started using Marshmallow 3 too.</p>
-
<p>Here are some PRs I contributed that are related to the REST API:</p>
-
<ol>
-<li><p><a href="https://github.com/apache/airflow/pull/9227" target="_blank">Add event log endpoints</a>
-The event log would help users get information on operations performed at the UI</p></li>
-
-<li><p><a href="https://github.com/apache/airflow/pull/9266" target="_blank">Add CRUD endpoints for connection</a>
-This PR performs DELETE, PATCH and POST operations on <code>Connection</code></p></li>
-
-<li><p><a href="https://github.com/apache/airflow/pull/9331" target="_blank">Add log endpoint</a>
-This PR enables users to get Task Instances log entries</p></li>
-
-<li><p><a href="https://github.com/apache/airflow/pull/9431" target="_blank">Move limit & offset to kwargs in views plus work on a configurable maximum limit</a>
-This helped us in having a neat code on the views and added configurable maximum limit on query results.</p></li>
-
-<li><p><a href="https://github.com/apache/airflow/pull/9648" target="_blank">Update FlaskAppBuilder to v3</a>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9227">Add event log endpoints</a>
+The event log would help users get information on operations performed at the UI</p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9266">Add CRUD endpoints for connection</a>
+This PR performs DELETE, PATCH and POST operations on <code>Connection</code></p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9331">Add log endpoint</a>
+This PR enables users to get Task Instances log entries</p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9431">Move limit & offset to kwargs in views plus work on a configurable maximum limit</a>
+This helped us in having a neat code on the views and added configurable maximum limit on query results.</p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9648">Update FlaskAppBuilder to v3</a>
This enabled Airflow to start using v3 of Flask App Builder and also made it possible for the API to use
- a modern database serializer/deserializer</p></li>
-
-<li><p><a href="https://github.com/apache/airflow/pull/9771" target="_blank">Add migration guide from the experimental REST API to the stable REST API</a>
-This would enable users to start using the stable REST API in less time.</p></li>
+a modern database serializer/deserializer</p>
+</li>
+<li>
+<p><a href="https://github.com/apache/airflow/pull/9771">Add migration guide from the experimental REST API to the stable REST API</a>
+This would enable users to start using the stable REST API in less time.</p>
+</li>
</ol>
-
<h3 id="follow-ups">Follow-Ups</h3>
-
<p>There is still lots of works to be done on the REST API including writing helpful documentation.
I still follow up on these and hopefully, we will complete the REST API before the internship ends.</p>
-
-<p>I am very grateful to my mentors, <a href="https://github.com/potiuk" target="_blank">Jarek</a> and <a href="https://github.com/kaxil" target="_blank">Kaxil</a> for their
-patience with me and for surviving my never-ending questions. <a href="https://github.com/mik-laj" target="_blank">Kamil</a> and <a href="https://github.com/turbaszek" target="_blank">Tomek</a>
+<p>I am very grateful to my mentors, <a href="https://github.com/potiuk">Jarek</a> and <a href="https://github.com/kaxil">Kaxil</a> for their
+patience with me and for surviving my never-ending questions. <a href="https://github.com/mik-laj">Kamil</a> and <a href="https://github.com/turbaszek">Tomek</a>
have been very supportive and I appreciate them for their support and amazing code reviews.</p>
-
-<p>Thanks to <a href="https://github.com/leahecole" target="_blank">Leah E. Cole</a> and <a href="https://github.com/mschickensoup" target="_blank">Karolina Rosรณล</a>, for their
+<p>Thanks to <a href="https://github.com/leahecole">Leah E. Cole</a> and <a href="https://github.com/mschickensoup">Karolina Rosรณล</a>, for their
wonderful reviews. I’m grateful.</p>
-
<p>Thanks for reading!</p>
</div>
@@ -895,7 +873,7 @@ wonderful reviews. I’m grateful.</p>
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/index.html b/blog/index.html
index 93e138043a..0a5bc5c5b7 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
-<meta name="generator" content="Hugo 0.58.3" />
+<meta name="generator" content="Hugo 0.91.2" />
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
@@ -33,14 +33,10 @@
<title>Blog | Apache Airflow</title><meta property="og:title" content="Blog" />
<meta property="og:description" content="Platform created by the community to programmatically author, schedule and monitor workflows." />
<meta property="og:type" content="website" />
-<meta property="og:url" content="/blog/" />
+<meta property="og:url" content="/blog/" /><meta property="og:image" content="/images/feature-image.png"/><meta property="og:site_name" content="Apache Airflow" />
-<meta property="og:image" content="/images/feature-image.png" />
-<meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Blog">
-<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
-
-<meta name="twitter:card" content="summary_large_image"/>
+<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows."><meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
<meta name="twitter:title" content="Blog"/>
@@ -52,8 +48,8 @@
-<link rel="preload" href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" as="style">
-<link href="/scss/main.min.0e7882f371bce728f76bf8b7ddd1a316055cb9c6fc7ff482c3679b51f0369984.css" rel="stylesheet" integrity="">
+<link rel="preload" href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" as="style">
+<link href="/scss/main.min.6668c09ec5024c20cbf3e5c6d1f29463e1b7a1797526dc9cade6bda004c6da9d.css" rel="stylesheet" integrity="">
<script
src="https://code.jquery.com/jquery-3.3.1.min.js"
@@ -66,8 +62,8 @@
- <link rel="preload" href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" as="style">
- <link href="/scss/main-custom.min.c1bafd243a3b8eac3d0e0518111642bc80af69f05c204c47682d42ef29747534.css" rel="stylesheet" integrity="">
+ <link rel="preload" href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" as="style">
+ <link href="/scss/main-custom.min.8fb7313b3fad4e78fe5ead2437604a16375e2ddfe624d37e328483cfa86ad1e1.css" rel="stylesheet" integrity="">
@@ -467,7 +463,7 @@
- <a class="tag" href="/blog/tags/rest-api/">Rest api</a>
+ <a class="tag" href="/blog/tags/rest-api/">Rest API</a>
@@ -1347,7 +1343,7 @@
-<script src="/js/main.min.29b0315468c00226fa6f4556a9cebc0ac4fe1ce1457a01b22c0a06b329877383.js" integrity="sha256-KbAxVGjAAib6b0VWqc68CsT+HOFFegGyLAoGsymHc4M=" crossorigin="anonymous"></script>
+<script src="/js/main.min.5c74b870c6953931a705f390a49c7e4c0a842ec5c83b24354758dd674343ed0d.js" integrity="sha256-XHS4cMaVOTGnBfOQpJx+TAqELsXIOyQ1R1jdZ0ND7Q0=" crossorigin="anonymous"></script>
diff --git a/blog/index.xml b/blog/index.xml
index f73efd4ee4..dca9deaf63 100644
--- a/blog/index.xml
+++ b/blog/index.xml
@@ -22,63 +22,44 @@
<description>
-
-
-<h1 id="airflow-user-survey-2022">Airflow User Survey 2022</h1>
-
+ <h1 id="airflow-user-survey-2022">Airflow User Survey 2022</h1>
<p>This yearโs survey has come and gone, and with it weโve got a new batch of data for everyone! We collected 210 responses over two weeks. We continue to see growth in both contributions and downloads over the last two years, and expect that trend will continue through 2022.</p>
-
-<p>The raw response data will be made available here soon, in the meantime, feel free to email john.thomas@astronomer.io for a copy.</p>
-
-<h2 id="tl-dr">TL;DR</h2>
-
+<p>The raw response data will be made available here soon, in the meantime, feel free to email <a href="mailto:john.thomas@astronomer.io">john.thomas@astronomer.io</a> for a copy.</p>
+<h2 id="tldr">TL;DR</h2>
<h3 id="overview-of-the-user">Overview of the user</h3>
-
<ul>
-<li>Like previous years, more than half of the Airflow users are Data Engineers (54%). Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%) are also active Airflow users! There was a slight increase in the representation of Solutions Architect roles compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.o [...]
-<li>Airflow is used and popular in bigger companies, 64% of Airflow users work for companies with 200+ employees which is an 11 percent increase compared to <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>.</li>
+<li>Like previous years, more than half of the Airflow users are Data Engineers (54%). Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%) are also active Airflow users! There was a slight increase in the representation of Solutions Architect roles compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/& [...]
+<li>Airflow is used and popular in bigger companies, 64% of Airflow users work for companies with 200+ employees which is an 11 percent increase compared to <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>.</li>
<li>62% of the survey participants have more than 6 Airflow users in their company.</li>
-<li>More Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend [...]
-<li>Airflow documentation is a critical source of information, with more than 90% (15% increase compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>) of survey participants using the documentation. Airflow documentation is also one of the top areas to improve! Whatโs interesting, also Stack Overflow usage is critical, with about 60% users declaring to use it as a source of i [...]
+<li>More Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow ( 85.7% in <a href="https://airfl [...]
+<li>Airflow documentation is a critical source of information, with more than 90% (15% increase compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>) of survey participants using the documentation. Airflow documentation is also one of the top areas to improve! Whatโs interesting, also Stack Overflow usage is critical, with about 60% users declaring to use it as a source of information (24% increase [...]
</ul>
-
<h3 id="deployments">Deployments</h3>
-
<ul>
<li>85% of the Airflow users have between 1 to 7 active Airflow instances. 62.5% of the Airflow users have between 11 to 250 DAGs in their largest Airflow instance. 75% of the surveyed Airflow users have between 1 to 100 tasks per DAG.</li>
<li>Close to 85% of users use one of the Airflow 2 versions, 9.2% users still use 1.10.15, while the remaining 6.3% are still using olderAirflow 1 versions. The good news is that the majority of users on Airflow 1 are planning migration to Airflow 2 quite soon, with resources and capacity being the main blockers.</li>
-<li>In comparison to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>, more users were interested in monitoring in general and specifically in using tools such as external monitoring services (40.7%, up from 29.6%) and information from metabase (35.7%, up from 25.1%).</li>
+<li>In comparison to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>, more users were interested in monitoring in general and specifically in using tools such as external monitoring services (40.7%, up from 29.6%) and information from metabase (35.7%, up from 25.1%).</li>
<li>Celery (52.7%) and Kubernetes (39.4%) are the most common executors used.</li>
</ul>
-
<h3 id="usage">Usage</h3>
-
<ul>
<li>81.3% of Airflow users who responded to the survey donโt have any customisation of Airflow.</li>
<li>Xcom (69.8%) is the most popular method to pass inputs and outputs between tasks, however Saving and Retrieving Inputs and Outputs from Storage still plays an important role (49%).</li>
<li>Lineage itself is a quite new topic for Airflow users, most of them donโt use lineage solutions but might be interested if supported by Airflow (47.5%), are not familiar with data lineage (29%) or that data lineage is not their concern (13%).</li>
-<li>The Airflow web UI is used heavily for Monitoring Runs (95.9%), Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%). The top 3 views used are: List of DAGs, Task Logs and DAG Runs, which is very similar to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/ [...]
+<li>The Airflow web UI is used heavily for Monitoring Runs (95.9%), Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%). The top 3 views used are: List of DAGs, Task Logs and DAG Runs, which is very similar to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/"> [...]
</ul>
-
<h3 id="community-and-contribution">Community and contribution</h3>
-
<ul>
<li>Most Airflow users (57.1%) are aware they could contribute but do not, and an additional 21.7% contribute very rarely. 14.8% of users were not aware they could contribute. There is much more to be done to engage our community to be more active contributors and raise the current 6.4% of users who actively contribute, especially considering that one important blocker for contribution is lack of knowledge on how to start (37.7%).</li>
</ul>
-
<h3 id="the-future-of-airflow">The future of Airflow</h3>
-
<ul>
<li>The top area for improvement is still the Airflow web UI (49.5%), closely followed by more telemetry for logging, monitoring and alerting purposes (48%). However all those efforts should go in line with improved documentation (36.6.%) and resources about using the Airflow, especially when we take into account the need of onboarding new users (36.6%).</li>
<li>DAG Versioning(66.2%) is a winner for new features in Airflow, and itโs not a surprise as this feature may positively impact daily work of Airflow users. It is followed by three other ideas: Dependency management and Data-driven scheduling (42.6%), More dynamic task structure (42.1%) and Multi-Tenancy (37.9%).</li>
</ul>
-
<h2 id="overview-of-the-user-1">Overview of the user</h2>
-
<h3 id="what-best-describes-your-current-occupation-single-choice">What best describes your current occupation? (single choice)</h3>
-
-<p><img src="images/image1.png" alt="alt_text" title="user_occupations" /></p>
-
+<p><img src="images/image1.png" alt="alt_text" title="user_occupations"></p>
<table>
<thead>
<tr>
@@ -87,62 +68,52 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Data Engineer</td>
<td>114</td>
<td>54%</td>
</tr>
-
<tr>
<td>Solutions Architect</td>
<td>27</td>
<td>13%</td>
</tr>
-
<tr>
<td>Developer</td>
<td>25</td>
<td>12%</td>
</tr>
-
<tr>
<td>DevOps</td>
<td>12</td>
<td>6%</td>
</tr>
-
<tr>
<td>Data Scientist</td>
<td>8</td>
<td>4%</td>
</tr>
-
<tr>
<td>Support Engineer</td>
<td>5</td>
<td>2%</td>
</tr>
-
<tr>
<td>Data Analyst</td>
<td>3</td>
<td>1%</td>
</tr>
-
<tr>
<td>Business Analyst</td>
<td>2</td>
<td>1%</td>
</tr>
-
<tr>
<td>Other</td>
<td>14</td>
@@ -150,13 +121,9 @@
</tr>
</tbody>
</table>
-
-<p>According to the survey, more than half of Airflow users are Data Engineers (54%). Roles of the remaining Airflow users might be broken down into Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%). The 2022 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> and <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" [...]
-
+<p>According to the survey, more than half of Airflow users are Data Engineers (54%). Roles of the remaining Airflow users might be broken down into Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%). The 2022 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/">those from 2019</a> and <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> with a [...]
<h3 id="how-often-do-you-interact-with-airflow-single-choice">How often do you interact with Airflow? (single choice)</h3>
-
-<p><img src="images/image2.png" alt="alt_text" title="interaction_frequency" /></p>
-
+<p><img src="images/image2.png" alt="alt_text" title="interaction_frequency"></p>
<table>
<thead>
<tr>
@@ -165,32 +132,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Every day</td>
<td>154</td>
<td>73%</td>
</tr>
-
<tr>
<td>At least once per week</td>
<td>36</td>
<td>17%</td>
</tr>
-
<tr>
<td>At least once per month</td>
<td>11</td>
<td>5%</td>
</tr>
-
<tr>
<td>Less than once per month</td>
<td>9</td>
@@ -198,13 +160,9 @@
</tr>
</tbody>
</table>
-
<p>Users who took the survey are actively using Airflow as part of their current role. 73% of Airflow users who responded use it on a daily basis, 17% weekly.</p>
-
<h3 id="how-many-people-work-at-your-company-single-choice">How many people work at your company? (single choice)</h3>
-
-<p><img src="images/image3.png" alt="alt_text" title="company_size" /></p>
-
+<p><img src="images/image3.png" alt="alt_text" title="company_size"></p>
<table>
<thead>
<tr>
@@ -213,38 +171,32 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>201-5000</td>
<td>85</td>
<td>41%</td>
</tr>
-
<tr>
<td>5000+</td>
<td>49</td>
<td>23%</td>
</tr>
-
<tr>
<td>51-200</td>
<td>46</td>
<td>22%</td>
</tr>
-
<tr>
<td>11-50</td>
<td>20</td>
<td>10%</td>
</tr>
-
<tr>
<td>1-10</td>
<td>9</td>
@@ -252,13 +204,9 @@
</tr>
</tbody>
</table>
-
-<p>Airflow is a framework that is used and popular in bigger companies, 64% of Airflow users who responded (compared to 52.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a>) work for companies bigger than 200 employees (41% in companies size 201-5000 and 23% in companies size 5000+).</p>
-
+<p>Airflow is a framework that is used and popular in bigger companies, 64% of Airflow users who responded (compared to 52.7% in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a>) work for companies bigger than 200 employees (41% in companies size 201-5000 and 23% in companies size 5000+).</p>
<h3 id="how-many-people-at-your-company-use-airflow-single-choice">How many people at your company use Airflow? (single choice)</h3>
-
-<p><img src="images/image4.png" alt="alt_text" title="airflow_usage" /></p>
-
+<p><img src="images/image4.png" alt="alt_text" title="airflow_usage"></p>
<table>
<thead>
<tr>
@@ -267,32 +215,27 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>6-20</td>
<td>80</td>
<td>38%</td>
</tr>
-
<tr>
<td>1-5</td>
<td>61</td>
<td>29%</td>
</tr>
-
<tr>
<td>51-200</td>
<td>49</td>
<td>24%</td>
</tr>
-
<tr>
<td>200+</td>
<td>18</td>
@@ -300,11 +243,8 @@
</tr>
</tbody>
</table>
-
<p>Airflow is generally used by small to medium-sized teams. 62% of the survey participants have more than 6 Airflow users in their company (38% have between 6 and 200 users, 24% between 51-200 users).</p>
-
<h3 id="how-likely-are-you-to-recommend-apache-airflow-single-choice">How likely are you to recommend Apache Airflow? (single choice)</h3>
-
<table>
<thead>
<tr>
@@ -314,7 +254,6 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
@@ -322,35 +261,30 @@
<td>% 2020</td>
<td>% 2022</td>
</tr>
-
<tr>
<td>Very Likely</td>
<td>45.4%</td>
<td>61.6%</td>
<td>65.9%</td>
</tr>
-
<tr>
<td>Likely</td>
<td>40.3%</td>
<td>30.4%</td>
<td>26.9%</td>
</tr>
-
<tr>
<td>Neutral</td>
<td>10.7%</td>
<td>5.4%</td>
<td>6.3%</td>
</tr>
-
<tr>
<td>Unlikely</td>
<td>2.6%</td>
<td>1.5%</td>
<td>0.5%</td>
</tr>
-
<tr>
<td>Very Unlikely</td>
<td>1%</td>
@@ -359,11 +293,8 @@
</tr>
</tbody>
</table>
-
-<p>According to the survey, more Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users a [...]
-
+<p>According to the survey, more Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> and <a href="https://airflow.apache.org/blog/airflow-survey/">2019</a>. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow (92% in <a hr [...]
<h3 id="what-is-your-source-of-information-about-airflow-multiple-choice">What is your source of information about Airflow? (multiple choice)</h3>
-
<table>
<thead>
<tr>
@@ -372,68 +303,57 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>Documentation</td>
<td>189</td>
<td>90.4%</td>
</tr>
-
<tr>
<td>Airflow website (Blog, etc.)</td>
<td>142</td>
<td>67.9%</td>
</tr>
-
<tr>
<td>Stack Overflow</td>
<td>126</td>
<td>60.3%</td>
</tr>
-
<tr>
<td>Github Issues</td>
<td>104</td>
<td>49.8%</td>
</tr>
-
<tr>
<td>Slack</td>
<td>96</td>
<td>45.9%</td>
</tr>
-
<tr>
<td>Airflow Summit Videos</td>
<td>88</td>
<td>42.1%</td>
</tr>
-
<tr>
<td>GitHub Discussions</td>
<td>76</td>
<td>36.4%</td>
</tr>
-
<tr>
<td>Airflow Community Webinars</td>
<td>41</td>
<td>19.6%</td>
</tr>
-
<tr>
<td>Astronomer Registry</td>
<td>51</td>
<td>24.4%</td>
</tr>
-
<tr>
<td>Airflow Mailing List</td>
<td>34</td>
@@ -441,15 +361,10 @@
</tr>
</tbody>
</table>
-
-<p>Airflow documentation is a critical source of information, with more than 90% of survey participants using the documentation. It is of increasing importance compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user" target="_blank">2020</a> where documentation was at about 75% level. Moreover, more than 60% of users are getting information from the Airflow website (67.9% ) and Stack Overflow (60.3%) wh [...]
-
+<p>Airflow documentation is a critical source of information, with more than 90% of survey participants using the documentation. It is of increasing importance compared to results from <a href="https://airflow.apache.org/blog/airflow-survey-2020/#overview-of-the-user">2020</a> where documentation was at about 75% level. Moreover, more than 60% of users are getting information from the Airflow website (67.9% ) and Stack Overflow (60.3%) which is also a big increa [...]
<h2 id="deployments-1">Deployments</h2>
-
<h3 id="how-many-active-dags-do-you-have-in-your-largest-airflow-instance-single-choice">How many active DAGs do you have in your largest Airflow instance? (single choice)</h3>
-
-<p><img src="images/image5.png" alt="alt_text" title="active_dags" /></p>
-
+<p><img src="images/image5.png" alt="alt_text" title="active_dags"></p>
<table>
<thead>
<tr>
@@ -458,50 +373,42 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>51-250</td>
<td>66</td>
<td>31.7%</td>
</tr>
-
<tr>
<td>11-50</td>
<td>64</td>
<td>30.8%</td>
</tr>
-
<tr>
<td>5-10</td>
<td>25</td>
<td>12.0%</td>
</tr>
-
<tr>
<td>251-500</td>
<td>20</td>
<td>9.6%</td>
</tr>
-
<tr>
<td>&lt;5</td>
<td>14</td>
<td>6.7%</td>
</tr>
-
<tr>
<td>1000+</td>
<td>10</td>
<td>4.8%</td>
</tr>
-
<tr>
<td>501-1000</td>
<td>9</td>
@@ -509,13 +416,9 @@
</tr>
</tbody>
</table>
-
<p>62.5% of the Airflow users surveyed have between 11 to 250 DAGs in their largest Airflow instance.</p>
-
<h3 id="how-many-active-airflow-instances-do-you-have-single-choice">How many active Airflow instances do you have? (single choice)</h3>
-
-<p><img src="images/image6.png" alt="alt_text" title="image_tooltip" /></p>
-
+<p><img src="images/image6.png" alt="alt_text" title="image_tooltip"></p>
<table>
<thead>
<tr>
@@ -524,50 +427,42 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>1</td>
<td>52</td>
<td>25.2%</td>
</tr>
-
<tr>
<td>2</td>
<td>46</td>
<td>22.3%</td>
</tr>
-
<tr>
<td>4-7</td>
<td>40</td>
<td>19.4%</td>
</tr>
-
<tr>
<td>3</td>
<td>37</td>
<td>18.0%</td>
</tr>
-
<tr>
<td>20+</td>
<td>19</td>
<td>9.2%</td>
</tr>
-
<tr>
<td>8-10</td>
<td>7</td>
<td>3.4%</td>
</tr>
-
<tr>
<td>11-20</td>
<td>5</td>
@@ -575,13 +470,9 @@
</tr>
</tbody>
</table>
-
<p>85% of the Airflow users surveyed have between 1 and 7 active Airflow instances, and nearly 50% have only 1 or 2.</p>
-
-<h3 id="what-is-the-maximum-number-of-tasks-that-you-have-used-in-a-single-dag-single-choice">What is the maximum number of tasks that you have used in a single DAG?(single choice)</h3>
-
-<p><img src="images/image7.png" alt="alt_text" title="maximum tasks" /></p>
-
+<h3 id="what-is-the-maximum-number-of-tasks-that-you-have-used-in-a-single-dagsingle-choice">What is the maximum number of tasks that you have used in a single DAG?(single choice)</h3>
+<p><img src="images/image7.png" alt="alt_text" title="maximum tasks"></p>
<table>
<thead>
<tr>
@@ -590,62 +481,52 @@
<th></th>
</tr>
</thead>
-
<tbody>
<tr>
<td></td>
<td>No.</td>
<td>%</td>
</tr>
-
<tr>
<td>11-25</td>
<td>51</td>
<td>24.5%</td>
</tr>
-
<tr>
<td>26-50</td>
<td>41</td>
<td>19.7%</td>
</tr>
-
<tr>
<td>51-100</td>
<td>35</td>
<td>16.8%</td>
</tr>
-
<tr>
<td>&lt;10</td>
<td>29</td>
<td>13.9%</td>
</tr>
-
<tr>
<td>101-250</td>
<td>23</td>
<td>11.1%</td>
</tr>
-
<tr>
<td>501-1000</td>
<td>9</td>
<td>4.3%</td>
</tr>
-
<tr>
<td>1000-2500</td>
<td>8</td>
<td>3.8%</td>
</tr>
-
... 31538 lines suppressed ...