You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by gi...@apache.org on 2021/03/09 22:02:01 UTC
[airflow-site] branch asf-site updated: Deploying to asf-site from @ 409ceb40633f127ed0f4fb508bfdae2e7d77b319 🚀
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 437d74b Deploying to asf-site from @ 409ceb40633f127ed0f4fb508bfdae2e7d77b319 🚀
437d74b is described below
commit 437d74b265dd519983ff212629cbfc221900339f
Author: turbaszek <tu...@gmail.com>
AuthorDate: Tue Mar 9 22:01:22 2021 +0000
Deploying to asf-site from @ 409ceb40633f127ed0f4fb508bfdae2e7d77b319 🚀
---
_gen/indexes/en/blog-index.json | 2 +-
_gen/indexes/en/blog-posts.json | 2 +-
blog/airflow-1.10.10/index.html | 4 +-
blog/airflow-1.10.12/index.html | 4 +-
blog/airflow-1.10.8-1.10.9/index.html | 4 +-
...What_best_describes_your_current_occupation.png | Bin 0 -> 91268 bytes
.../What_executor_type_do_you_use.png | Bin 0 -> 59914 bytes
...al_services_do_you_use_in_your_Airflow_DAGs.png | Bin 0 -> 104634 bytes
blog/airflow-survey-2020/Where_are_you_based.png | Bin 0 -> 95584 bytes
.../index.html | 1601 ++++++++++++++------
blog/airflow-survey/index.html | 52 +-
blog/airflow-two-point-oh-is-here/index.html | 8 +-
blog/announcing-new-website/index.html | 4 +-
blog/apache-airflow-for-newcomers/index.html | 4 +-
.../index.html | 4 +-
.../index.html | 4 +-
.../index.html | 4 +-
.../index.html | 4 +-
.../index.html | 4 +-
blog/index.html | 39 +
blog/index.xml | 1599 +++++++++++++++++++
.../index.html | 4 +-
blog/tags/community/index.html | 41 +-
blog/tags/community/index.xml | 1601 +++++++++++++++++++-
blog/tags/survey/index.html | 41 +-
blog/tags/survey/index.xml | 1601 +++++++++++++++++++-
blog/tags/users/index.html | 41 +-
blog/tags/users/index.xml | 1601 +++++++++++++++++++-
index.html | 32 +-
index.xml | 1599 +++++++++++++++++++
search/index.html | 4 +-
sitemap.xml | 107 +-
tags/index.html | 2 +-
tags/index.xml | 2 +-
use-cases/adobe/index.html | 4 +-
use-cases/big-fish-games/index.html | 4 +-
use-cases/dish/index.html | 4 +-
use-cases/experity/index.html | 4 +-
use-cases/onefootball/index.html | 4 +-
use-cases/plarium-krasnodar/index.html | 4 +-
use-cases/sift/index.html | 4 +-
41 files changed, 9455 insertions(+), 592 deletions(-)
diff --git a/_gen/indexes/en/blog-index.json b/_gen/indexes/en/blog-index.json
index 22238a3..cc28fd7 100644
--- a/_gen/indexes/en/blog-index.json
+++ b/_gen/indexes/en/blog-index.json
@@ -1 +1 @@
-{"version":"2.3.8","fields":["title","description","author","content","tags","url"],"fieldVectors":[["title/It's a \"Breeze\" to develop Apache Airflow",[0,1.665,1,0.777,2,0.031,3,0.382,4,0.477]],["description/It's a \"Breeze\" to develop Apache Airflow",[2,0.032,4,0.313,5,1.405,6,0.735,7,1.405,8,0.735,9,0.735,10,0.612,11,0.735,12,0.612,13,1.405,14,0.423,15,1.405,16,0.111]],["author/It's a \"Breeze\" to develop Apache Airflow",[11,1.186,17,1.434]],["content/It's a \"Breeze\" to develop A [...]
\ No newline at end of file
+{"version":"2.3.8","fields":["title","description","author","content","tags","url"],"fieldVectors":[["title/It's a \"Breeze\" to develop Apache Airflow",[0,1.4,1,0.83,2,0.029,3,0.369,4,0.472]],["description/It's a \"Breeze\" to develop Apache Airflow",[2,0.032,4,0.331,5,1.527,6,0.689,7,1.527,8,0.819,9,0.819,10,0.581,11,0.819,12,0.581,13,1.527,14,0.489,15,1.527,16,0.02]],["author/It's a \"Breeze\" to develop Apache Airflow",[11,1.251,17,1.499]],["content/It's a \"Breeze\" to develop Apach [...]
\ No newline at end of file
diff --git a/_gen/indexes/en/blog-posts.json b/_gen/indexes/en/blog-posts.json
index 06bb888..165a10a 100644
--- a/_gen/indexes/en/blog-posts.json
+++ b/_gen/indexes/en/blog-posts.json
@@ -1 +1 @@
-[{"content":"## The story behind the Airflow Breeze tool\nInitially, we started contributing to this fantastic open-source project [Apache Airflow] with a team of three which then grew to five. When we kicked it off a year ago, I realized pretty soon where the biggest bottlenecks and areas for improvement in terms of productivity were. Even with the help of our client, who provided us with a “homegrown” development environment it took us literally days to set it up and learn some basics. [...]
\ No newline at end of file
+[{"content":"## The story behind the Airflow Breeze tool\nInitially, we started contributing to this fantastic open-source project [Apache Airflow] with a team of three which then grew to five. When we kicked it off a year ago, I realized pretty soon where the biggest bottlenecks and areas for improvement in terms of productivity were. Even with the help of our client, who provided us with a “homegrown” development environment it took us literally days to set it up and learn some basics. [...]
\ No newline at end of file
diff --git a/blog/airflow-1.10.10/index.html b/blog/airflow-1.10.10/index.html
index 64c6392..cc95299 100644
--- a/blog/airflow-1.10.10/index.html
+++ b/blog/airflow-1.10.10/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-04-09T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow 1.10.10">
<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.10">
<meta itemprop="datePublished" content="2020-04-09T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="1143">
diff --git a/blog/airflow-1.10.12/index.html b/blog/airflow-1.10.12/index.html
index 30541a6..f776d0c 100644
--- a/blog/airflow-1.10.12/index.html
+++ b/blog/airflow-1.10.12/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-08-25T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow 1.10.12">
<meta itemprop="description" content="We are happy to present Apache Airflow 1.10.12">
<meta itemprop="datePublished" content="2020-08-25T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="911">
diff --git a/blog/airflow-1.10.8-1.10.9/index.html b/blog/airflow-1.10.8-1.10.9/index.html
index c49738a..7abe65b 100644
--- a/blog/airflow-1.10.8-1.10.9/index.html
+++ b/blog/airflow-1.10.8-1.10.9/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-02-23T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow 1.10.8 & 1.10.9">
<meta itemprop="description" content="We are happy to present the new 1.10.8 and 1.10.9 releases of Apache Airflow.">
<meta itemprop="datePublished" content="2020-02-23T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="437">
diff --git a/blog/airflow-survey-2020/What_best_describes_your_current_occupation.png b/blog/airflow-survey-2020/What_best_describes_your_current_occupation.png
new file mode 100644
index 0000000..63ca2fb
Binary files /dev/null and b/blog/airflow-survey-2020/What_best_describes_your_current_occupation.png differ
diff --git a/blog/airflow-survey-2020/What_executor_type_do_you_use.png b/blog/airflow-survey-2020/What_executor_type_do_you_use.png
new file mode 100644
index 0000000..9c0b3af
Binary files /dev/null and b/blog/airflow-survey-2020/What_executor_type_do_you_use.png differ
diff --git a/blog/airflow-survey-2020/What_external_services_do_you_use_in_your_Airflow_DAGs.png b/blog/airflow-survey-2020/What_external_services_do_you_use_in_your_Airflow_DAGs.png
new file mode 100644
index 0000000..fa15a74
Binary files /dev/null and b/blog/airflow-survey-2020/What_external_services_do_you_use_in_your_Airflow_DAGs.png differ
diff --git a/blog/airflow-survey-2020/Where_are_you_based.png b/blog/airflow-survey-2020/Where_are_you_based.png
new file mode 100644
index 0000000..130ed36
Binary files /dev/null and b/blog/airflow-survey-2020/Where_are_you_based.png differ
diff --git a/blog/airflow-survey/index.html b/blog/airflow-survey-2020/index.html
similarity index 74%
copy from blog/airflow-survey/index.html
copy to blog/airflow-survey-2020/index.html
index cf62056..b203b7b 100644
--- a/blog/airflow-survey/index.html
+++ b/blog/airflow-survey-2020/index.html
@@ -29,21 +29,21 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Airflow Survey 2019 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2019" />
-<meta property="og:description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss." />
+<title>Airflow Survey 2020 | Apache Airflow</title><meta property="og:title" content="Airflow Survey 2020" />
+<meta property="og:description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance." />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-survey/" />
+<meta property="og:url" content="/blog/airflow-survey-2020/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Airflow Survey 2019">
-<meta itemprop="description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss.">
+<meta property="article:published_time" content="2021-03-09T00:00:00+00:00" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta itemprop="name" content="Airflow Survey 2020">
+<meta itemprop="description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.">
-<meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
-<meta itemprop="wordCount" content="1775">
+<meta itemprop="datePublished" content="2021-03-09T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
+<meta itemprop="wordCount" content="2106">
@@ -51,8 +51,8 @@
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
-<meta name="twitter:title" content="Airflow Survey 2019"/>
-<meta name="twitter:description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss."/>
+<meta name="twitter:title" content="Airflow Survey 2020"/>
+<meta name="twitter:description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance."/>
<script type="application/javascript">
@@ -80,7 +80,7 @@ if (!doNotTrack) {
crossorigin="anonymous"></script>
-<meta name="description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss." />
+<meta name="description" content="We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance." />
@@ -447,15 +447,15 @@ if (!doNotTrack) {
</div>
- <span class="bodytext__medium--brownish-grey">Wed, Dec 11, 2019</span>
+ <span class="bodytext__medium--brownish-grey">Tue, Mar 9, 2021</span>
</div>
- <p class="blogpost-content__metadata--title">Airflow Survey 2019</p>
+ <p class="blogpost-content__metadata--title">Airflow Survey 2020</p>
<div class="blogpost-content__metadata--author">
<span class="blogpost-content__metadata--author">
Tomek Urbaszek
</span>
- <a href="https://twitter.com/Nuclearriot/" class="blogpost-content__metadata--social-media-icon">
+ <a href="https://twitter.com/turbaszek/" class="blogpost-content__metadata--social-media-icon">
<svg xmlns="http://www.w3.org/2000/svg" width="22" height="21" viewBox="0 0 22 21">
<g id="Group_1746" data-name="Group 1746" transform="translate(.076 -.055)">
<ellipse id="Ellipse_19" cx="11" cy="10.5" fill="#51504f" data-name="Ellipse 19" rx="11" ry="10.5"
@@ -471,7 +471,7 @@ if (!doNotTrack) {
</a>
- <a href="https://github.com/nuclearpinguin/" class="blogpost-content__metadata--social-media-icon">
+ <a href="https://github.com/turbaszek/" class="blogpost-content__metadata--social-media-icon">
<svg xmlns="http://www.w3.org/2000/svg" width="21.737" height="21.2" viewBox="0 0 21.737 21.2">
<path id="Path_1378" d="M33.971 1181.31a10.87 10.87 0 0 0-3.435 21.182c.543.1.742-.236.742-.524 0-.258-.009-.941-.015-1.848-3.023.657-3.661-1.457-3.661-1.457a2.876 2.876 0 0 0-1.207-1.59c-.987-.674.075-.661.075-.661a2.283 2.283 0 0 1 1.665 1.12 2.314 2.314 0 0 0 3.163.9 2.322 2.322 0 0 1 .69-1.453c-2.413-.274-4.951-1.207-4.951-5.371a4.2 4.2 0 0 1 1.119-2.917 3.908 3.908 0 0 1 .107-2.876s.913-.292 2.989 1.114a10.3 10.3 0 0 1 5.442 0c2.075-1.406 2.986-1.114 2.986-1.114a3.9 3.9 0 0 1 .1 [...]
</svg>
@@ -493,21 +493,462 @@ if (!doNotTrack) {
</a>
</div>
- <p class="blogpost-content__metadata--description">Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss.</p>
+ <p class="blogpost-content__metadata--description">We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.</p>
</div>
<div class="markdown-content">
-<h1 id="apache-airflow-survey-2019">Apache Airflow Survey 2019</h1>
+<h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
-<p>Apache Airflow is <a href="https://www.astronomer.io/blog/why-airflow/" target="_blank">growing faster than ever</a>.
-Thus, receiving and adjusting to our users’ feedback is a must. We created
-<a href="https://forms.gle/XAzR1pQBZiftvPQM7" target="_blank">survey</a> and we got <strong>308</strong> responses.
-Let’s see who Airflow users are, how they play with it, and what they miss.</p>
+<p>World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
+crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
+an amount of active contributors. So listening and understanding our community is of high importance.</p>
-<h1 id="overview-of-the-user">Overview of the user</h1>
+<p>It’s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!</p>
-<p><strong>What best describes your current occupation?</strong></p>
+<h2 id="overview-of-the-user">Overview of the user</h2>
+
+<p><img src="What_best_describes_your_current_occupation.png" alt="" /></p>
+
+<p><strong>What best describes your current occupation? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Data Engineer</td>
+<td>115</td>
+<td>56.65</td>
+</tr>
+
+<tr>
+<td>Developer</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>DevOps</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Solutions Architect</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>Data Scientist</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>Data Analyst</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>Support Engineer</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> with
+visible slight increase in ML use cases.</p>
+
+<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
+
+<p><strong>How many people work in your company? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>200+</td>
+<td>107</td>
+<td>52.71</td>
+</tr>
+
+<tr>
+<td>51-200</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>11-50</td>
+<td>37</td>
+<td>18.23</td>
+</tr>
+
+<tr>
+<td>1-10</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1-5</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>6-20</td>
+<td>75</td>
+<td>36.95</td>
+</tr>
+
+<tr>
+<td>21-50</td>
+<td>23</td>
+<td>11.33</td>
+</tr>
+
+<tr>
+<td>50+</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+</tbody>
+</table>
+
+<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
+
+<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we are happy with Airflow</td>
+<td>174</td>
+<td>85.71</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
+(8.37% both).</p>
+
+<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api" target="_blank">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.</p>
+
+<p>The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.</p>
+
+<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I wish we could</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>No</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+</tbody>
+</table>
+
+<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there’s something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.</p>
+
+<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020 %</th>
+<th>2019 %</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Very Likely</td>
+<td>125</td>
+<td>61.58</td>
+<td>45.45%</td>
+</tr>
+
+<tr>
+<td>Likely</td>
+<td>62</td>
+<td>30.54</td>
+<td>40.26%</td>
+</tr>
+
+<tr>
+<td>Neutral</td>
+<td>11</td>
+<td>5.42</td>
+<td>10.71%</td>
+</tr>
+
+<tr>
+<td>Unlikely</td>
+<td>3</td>
+<td>1.48</td>
+<td>2.60%</td>
+</tr>
+
+<tr>
+<td>Very unlikely</td>
+<td>2</td>
+<td>0.99</td>
+<td>0.97%</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
+
+<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Documentation</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Airflow website</td>
+<td>139</td>
+<td>68.47</td>
+</tr>
+
+<tr>
+<td>Slack</td>
+<td>128</td>
+<td>63.05</td>
+</tr>
+
+<tr>
+<td>Github</td>
+<td>127</td>
+<td>62.56</td>
+</tr>
+
+<tr>
+<td>Stack Overflow</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Airflow Summit Videos</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>The dev mailing list</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Awesome Apache Airflow repository</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here we see that Airflow documentation is the crucial source of information. What’s interesting is that more
+than 60% of users are getting information from Github and Slack channels.</p>
+
+<p><img src="Where_are_you_based.png" alt="" /></p>
+
+<h2 id="airflow-uses-cases">Airflow uses cases</h2>
+
+<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we use vanilla Airflow</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Yes, we have small patches (no fork)</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Yes, we have separate fork</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No guide on best practises on developing DAGs</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>There is no easy option to launch Airflow</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Small number of tutorials on different aspects of using Airflow</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Documentation is not clear enough</td>
+<td>53</td>
+<td>26.11</td>
+</tr>
+
+<tr>
+<td>There is no easy option to deploy DAGs to an Airflow instance</td>
+<td>52</td>
+<td>25.62</td>
+</tr>
+
+<tr>
+<td>No problems</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Small number of blogs regarding Airflow</td>
+<td>30</td>
+<td>14.78</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
<table>
<thead>
@@ -520,98 +961,114 @@ Let’s see who Airflow users are, how they play with it, and what they miss.</p
<tbody>
<tr>
-<td>Data Engineer</td>
-<td>194</td>
-<td>62.99%</td>
+<td>Original Airflow Graphical User Interface</td>
+<td>199</td>
+<td>98.03</td>
</tr>
<tr>
-<td>Developer</td>
-<td>34</td>
-<td>11.04%</td>
+<td>CLI</td>
+<td>88</td>
+<td>43.35</td>
</tr>
<tr>
-<td>Architect</td>
-<td>23</td>
-<td>7.47%</td>
+<td>API</td>
+<td>48</td>
+<td>23.65</td>
</tr>
<tr>
-<td>Data Scientist</td>
-<td>19</td>
-<td>6.17%</td>
+<td>Custom (own created) Airflow Graphical User Interface</td>
+<td>12</td>
+<td>5.91</td>
</tr>
<tr>
-<td>Data Analyst</td>
-<td>13</td>
-<td>4.22%</td>
+<td>Other</td>
+<td>3</td>
+<td>1.48</td>
</tr>
+</tbody>
+</table>
+<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
<tr>
-<td>DevOps</td>
-<td>13</td>
-<td>4.22%</td>
+<th></th>
+<th>No.</th>
+<th>%</th>
</tr>
+</thead>
+<tbody>
<tr>
-<td>IT Administrator</td>
-<td>2</td>
-<td>0.65%</td>
+<td>Yes, by triggering another DAG</td>
+<td>87</td>
+<td>42.86</td>
</tr>
<tr>
-<td>Machine Learning Engineer</td>
-<td>2</td>
-<td>0.65%</td>
+<td>No, I don’t combine multiple DAGs</td>
+<td>79</td>
+<td>38.92</td>
</tr>
<tr>
-<td>Manager</td>
-<td>2</td>
-<td>0.65%</td>
+<td>Yes, through SubDAG</td>
+<td>40</td>
+<td>19.7</td>
</tr>
<tr>
-<td>Operations</td>
-<td>2</td>
-<td>0.65%</td>
+<td>Other</td>
+<td>18</td>
+<td>8.87</td>
</tr>
+</tbody>
+</table>
+
+<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
+<table>
+<thead>
<tr>
-<td>Chief Data Officer</td>
-<td>1</td>
-<td>0.32%</td>
+<th></th>
+<th>No.</th>
+<th>%</th>
</tr>
+</thead>
+<tbody>
<tr>
-<td>Engineering Manager</td>
-<td>1</td>
-<td>0.32%</td>
+<td>Using existing dedicated operators / hooks</td>
+<td>147</td>
+<td>72.41</td>
</tr>
<tr>
-<td>Intern</td>
-<td>1</td>
-<td>0.32%</td>
+<td>Using Bash / Python operator</td>
+<td>140</td>
+<td>68.97</td>
</tr>
<tr>
-<td>Product owner</td>
-<td>1</td>
-<td>0.32%</td>
+<td>Using own custom operators / hooks</td>
+<td>138</td>
+<td>67.98</td>
</tr>
<tr>
-<td>Quant</td>
-<td>1</td>
-<td>0.32%</td>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
</tr>
</tbody>
</table>
-<p><strong>In your day to day job, what do you use Airflow for?</strong></p>
+<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
<table>
<thead>
@@ -624,32 +1081,52 @@ Let’s see who Airflow users are, how they play with it, and what they miss.</p
<tbody>
<tr>
-<td>Data processing (ETL)</td>
-<td>298</td>
-<td>96.75%</td>
+<td>Amazon Web Services</td>
+<td>121</td>
+<td>59.61</td>
</tr>
<tr>
-<td>Artificial Intelligence and Machine Learning Pipelines</td>
-<td>90</td>
-<td>29.22%</td>
+<td>Internal company systems</td>
+<td>113</td>
+<td>55.67</td>
</tr>
<tr>
-<td>Automating DevOps operations</td>
-<td>64</td>
-<td>20.78%</td>
+<td>Google Cloud Platform / Google APIs</td>
+<td>97</td>
+<td>47.78</td>
+</tr>
+
+<tr>
+<td>Hadoop / Spark / Flink / Other Apache software</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Microsoft Azure</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>I do not use external services in my Airflow DAGs</td>
+<td>5</td>
+<td>2.46</td>
</tr>
</tbody>
</table>
-<p>According to the survey, most of the Airflow users are the “data” people. Moreover,
-28.57% uses Airflow to both ETL and ML pipelines meaning that those two fields
-are somehow connected. Only five respondents use Airflow for DevOps operations only,
-That means that other 59 people who use Airflow for DevOps stuff use it also for
-ETL / ML purposes.</p>
+<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt="" /></p>
-<p><strong>How many active DAGs do you have in your largest Airflow instance?</strong></p>
+<p><strong>Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)</strong></p>
<table>
<thead>
@@ -662,64 +1139,160 @@ ETL / ML purposes.</p>
<tbody>
<tr>
-<td>0-20</td>
-<td>115</td>
-<td>37.34%</td>
+<td>Adding new operators/sensors and hooks</td>
+<td>119</td>
+<td>58.62</td>
+</tr>
+
+<tr>
+<td>I don’t use Airflow plugins</td>
+<td>69</td>
+<td>33.99</td>
+</tr>
+
+<tr>
+<td>Adding AppBuilder views & menu items</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>Adding new executors</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Adding OperatorExtraLinks</td>
+<td>13</td>
+<td>6.4</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td></td>
+<td></td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you use Airflow’s data lineage feature? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, I will use such feature if fully supported in Airflow</td>
+<td>105</td>
+<td>51.72</td>
+</tr>
+
+<tr>
+<td>No, data lineage isn’t a concern for my usage.</td>
+<td>68</td>
+<td>33.5</td>
+</tr>
+
+<tr>
+<td>Yes, I use another data lineage product</td>
+<td>24</td>
+<td>11.82</td>
+</tr>
+
+<tr>
+<td>Yes, I use custom implementation</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+
+<tr>
+<td>Yes, I use Airflow’s experimental data lineage feature</td>
+<td>1</td>
+<td>0.49</td>
+</tr>
+</tbody>
+</table>
+
+<p>When asked what lineage product users use, the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.</p>
+
+<h2 id="deployment">Deployment</h2>
+
+<p><strong>How many active DAGs do you have in your largest Airflow instance? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>< 20</td>
+<td>64</td>
+<td>32</td>
</tr>
<tr>
<td>21-40</td>
-<td>65</td>
-<td>21.10%</td>
+<td>33</td>
+<td>16</td>
</tr>
<tr>
<td>41-60</td>
-<td>44</td>
-<td>14.29%</td>
+<td>13</td>
+<td>6</td>
</tr>
<tr>
<td>61-100</td>
-<td>28</td>
-<td>9.09%</td>
+<td>32</td>
+<td>16</td>
</tr>
<tr>
<td>101-200</td>
-<td>28</td>
-<td>9.09%</td>
+<td>31</td>
+<td>15</td>
</tr>
<tr>
<td>201-300</td>
-<td>7</td>
-<td>2.27%</td>
+<td>8</td>
+<td>4</td>
</tr>
<tr>
<td>301-999</td>
-<td>8</td>
-<td>2.60%</td>
+<td>12</td>
+<td>6</td>
</tr>
<tr>
<td>1000+</td>
-<td>13</td>
-<td>4.22%</td>
+<td>10</td>
+<td>5</td>
</tr>
</tbody>
</table>
-<p>The majority of users do not exceed 100 active DAGs per Airflow instance. However,
-as we can see there are users who exceed thousands of DAGs with a maximum number 5000.</p>
-
-<p><strong>What is the maximum number of tasks that you have used in one DAG?</strong></p>
+<p><strong>What is the maximum number of tasks that you have used in one DAG? (open question)</strong></p>
<table>
<thead>
<tr>
-<th></th>
+<th>Number of DAGs</th>
<th>No.</th>
<th>%</th>
</tr>
@@ -727,66 +1300,62 @@ as we can see there are users who exceed thousands of DAGs with a maximum number
<tbody>
<tr>
-<td>0-10</td>
-<td>61</td>
-<td>19.81%</td>
+<td>< 10</td>
+<td>42</td>
+<td>21</td>
</tr>
<tr>
<td>11-20</td>
-<td>60</td>
-<td>19.48%</td>
+<td>31</td>
+<td>15</td>
</tr>
<tr>
<td>21-30</td>
-<td>31</td>
-<td>10.06%</td>
+<td>15</td>
+<td>7</td>
</tr>
<tr>
<td>31-40</td>
-<td>21</td>
-<td>6.82%</td>
+<td>11</td>
+<td>5</td>
</tr>
<tr>
<td>41-50</td>
-<td>26</td>
-<td>8.44%</td>
+<td>22</td>
+<td>11</td>
</tr>
<tr>
<td>51-100</td>
-<td>36</td>
-<td>11.69%</td>
+<td>39</td>
+<td>19</td>
</tr>
<tr>
<td>101-200</td>
-<td>28</td>
-<td>9.09%</td>
+<td>16</td>
+<td>8</td>
</tr>
<tr>
<td>201-500</td>
-<td>21</td>
-<td>6.82%</td>
+<td>16</td>
+<td>8</td>
</tr>
<tr>
<td>501+</td>
-<td>24</td>
-<td>11.54%</td>
+<td>11</td>
+<td>5</td>
</tr>
</tbody>
</table>
-<p>The given maximum number of tasks in a single DAG was 10 000 (!). The number of tasks
-depends on the purposes of a DAG, so it’s rather hard to say if users have “simple”
-or “complicated” workflows.</p>
-
-<p><strong>When onboarding new members to Airflow, what is the biggest problem?</strong></p>
+<p><strong>Which version of Airflow do you use currently? (single choice)</strong></p>
<table>
<thead>
@@ -799,48 +1368,80 @@ or “complicated” workflows.</p>
<tbody>
<tr>
-<td>No guide on best practises on developing DAGs</td>
-<td>160</td>
-<td>51.95%</td>
+<td>1.10.14</td>
+<td>55</td>
+<td>27.09</td>
</tr>
<tr>
-<td>Small number of tutorials on different aspects of using Airflow</td>
-<td>57</td>
-<td>18.51%</td>
+<td>2.0.0+</td>
+<td>45</td>
+<td>22.17</td>
</tr>
<tr>
-<td>Documentation is not clear enough</td>
-<td>42</td>
-<td>13.64%</td>
+<td>1.10.12</td>
+<td>27</td>
+<td>13.3</td>
</tr>
<tr>
-<td>Small number of blogs regarding Airflow</td>
-<td>6</td>
-<td>1.95%</td>
+<td>1.10.10</td>
+<td>26</td>
+<td>12.81</td>
</tr>
<tr>
-<td>Other</td>
-<td>43</td>
-<td>13.96%</td>
+<td>1.10.11</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>1.10.5 or older</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>1.10.9</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>1.10.13</td>
+<td>7</td>
+<td>3.45</td>
+</tr>
+
+<tr>
+<td>1.10.6</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.7</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.8</td>
+<td>3</td>
+<td>1.48</td>
</tr>
</tbody>
</table>
-<p>This is an important result. Using Airflow is all about writing and scheduling DAGs.
-No guide or any other complete resource on best practices for developing Dags is a big
-problem. Diving deep in the “other” answers, we can find that:</p>
+<p>This was probably one of the most important questions in the survey. While it’s good to see
+that more than 60% of users use one of three latest Airflow versions, it’s worrying that the rest
+are using versions that are old or have known security vulnerabilities.</p>
-<ul>
-<li>Airflow’s “magic” (scheduler, executors, schedule times) is hard to understand</li>
-<li>DAG testing is not easy to do and to explain</li>
-<li>Airflow UI needs some love.</li>
-</ul>
+<p>Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.</p>
-<p><strong>How likely are you to recommend Apache Airflow?</strong></p>
+<p><strong>What meta-database do you use? (single choice)</strong></p>
<table>
<thead>
@@ -849,125 +1450,125 @@ problem. Diving deep in the “other” answers, we can find that:</p>
<th>No.</th>
<th>%</th>
</tr>
-</thead>
+</thead>
+
+<tbody>
+<tr>
+<td>Postgres 12</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Postgres 9.6</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Postgres 11</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
-<tbody>
<tr>
-<td>Very Likely</td>
-<td>140</td>
-<td>45.45%</td>
+<td>MySQL 5.7</td>
+<td>27</td>
+<td>13.3</td>
</tr>
<tr>
-<td>Likely</td>
-<td>124</td>
-<td>40.26%</td>
+<td>MySQL 8.0</td>
+<td>20</td>
+<td>9.85</td>
</tr>
<tr>
-<td>Neutral</td>
-<td>33</td>
-<td>10.71%</td>
+<td>Postgres 10</td>
+<td>20</td>
+<td>9.85</td>
</tr>
<tr>
-<td>Unlikely</td>
-<td>8</td>
-<td>2.60%</td>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
</tr>
<tr>
-<td>Very unlikely</td>
-<td>3</td>
-<td>0.97%</td>
+<td>Postgres 13</td>
+<td>18</td>
+<td>8.87</td>
</tr>
</tbody>
</table>
-<p>This means that more than 85% of people who use Airflow like it. It seems Airflow does
-its job nicely. However, we have to remember that this survey is likely biased - it’s
-more likely that you respond to the survey if you like the tool you use. Should we
-focus then on those 11 people who did not like Airflow? It’s a good question.</p>
+<p>This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.</p>
-<h2 id="airflow-usage">Airflow usage</h2>
+<p>It’s good to know that users rather avoid using SQLite in production deployments!</p>
-<p><strong>Which interface(s) of Airflow do you use as part of your current role?</strong></p>
+<p><strong>What executor type do you use? (single choice)</strong></p>
+
+<p><img src="What_executor_type_do_you_use.png" alt="" /></p>
<table>
<thead>
<tr>
<th></th>
<th>No.</th>
-<th>%</th>
+<th>2020</th>
+<th>2019</th>
</tr>
</thead>
<tbody>
<tr>
-<td>Original Airflow Graphical User Interface</td>
-<td>297</td>
-<td>96.43%</td>
-</tr>
-
-<tr>
-<td>CLI</td>
-<td>126</td>
-<td>40.91%</td>
+<td>Celery</td>
+<td>100</td>
+<td>49.26%</td>
+<td>44.81%</td>
</tr>
<tr>
-<td>Original Airflow Graphical User Interface, CLI</td>
-<td>117</td>
-<td>37.99%</td>
+<td>Kubernetes</td>
+<td>48</td>
+<td>23.65%</td>
+<td>16.88%</td>
</tr>
<tr>
-<td>API</td>
-<td>60</td>
-<td>19.48%</td>
+<td>Local</td>
+<td>40</td>
+<td>19.7%</td>
+<td>27.60%</td>
</tr>
<tr>
-<td>Original Airflow Graphical User Interface, CLI, API</td>
-<td>32</td>
-<td>10.39%</td>
+<td>Sequential</td>
+<td>10</td>
+<td>4.93%</td>
+<td>7.14%</td>
</tr>
<tr>
-<td>Custom (own created) Airflow Graphical User Interface</td>
-<td>25</td>
-<td>8.12%</td>
+<td>Other</td>
+<td>5</td>
+<td>2.46%</td>
+<td>3.57</td>
</tr>
</tbody>
</table>
-<p>It’s visible that usage of CLI goes in pair with using Airflow web UI. Our
-survey included some UX related questions to allow us to understand how users
-use Airflow webserver.</p>
-
-<p><strong>What do you use the Graphical User Interface for?</strong></p>
-
-<p><img src="plot1.png" alt="" /></p>
-
-<p><strong>What do you use CLI for?</strong></p>
-
-<p><img src="plot2.png" alt="" /></p>
-
-<p><strong>In Airflow, which UI view(s) are important for you?</strong></p>
+<p>In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users’ deployments are growing, and they need more scalable solutions.</p>
-<p><img src="plot3.png" alt="" /></p>
+<p>Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.</p>
-<p>Here we see that the majority uses Web UI mostly for monitoring purposes:</p>
-
-<ul>
-<li>Monitoring DAGs</li>
-<li>Accessing logs</li>
-</ul>
-
-<p>An interesting result is that many people seem not to use backfilling as
-there’s no other way than to do it by CLI.</p>
-
-<p><strong>What executor type do you use?</strong></p>
+<p><strong>What metrics do you use to monitor Airflow? (multiple choice)</strong></p>
<table>
<thead>
@@ -980,43 +1581,41 @@ there’s no other way than to do it by CLI.</p>
<tbody>
<tr>
-<td>Celery</td>
-<td>138</td>
-<td>44.81%</td>
+<td>I do not use monitoring</td>
+<td>65</td>
+<td>32.02</td>
</tr>
<tr>
-<td>Local</td>
-<td>85</td>
-<td>27.60%</td>
+<td>External monitoring service</td>
+<td>60</td>
+<td>29.56</td>
</tr>
<tr>
-<td>Kubernetes</td>
-<td>52</td>
-<td>16.88%</td>
+<td>Information from metadatabase</td>
+<td>51</td>
+<td>25.12</td>
</tr>
<tr>
-<td>Sequential</td>
-<td>22</td>
-<td>7.14%</td>
+<td>Statsd</td>
+<td>49</td>
+<td>24.14</td>
</tr>
<tr>
<td>Other</td>
-<td>11</td>
-<td>3.57</td>
+<td>31</td>
+<td>15.27</td>
</tr>
</tbody>
</table>
-<p>The other option mostly consisted of information that someone uses a few types or is
-migrating from one executor to another. What can be observed is an increase in usage
-of Local and Kubernetes executors when compared to results from an earlier <a href="https://ash.berlintaylor.com/writings/2019/02/airflow-user-survey-2019/" target="_blank">survey done
-by Ash</a>.</p>
+<p>The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.</p>
-<p><strong>Do you use Kubernetes-based deployments for Airflow?</strong></p>
+<p><strong>How do you deploy Airflow? (single choice)</strong></p>
<table>
<thead>
@@ -1029,49 +1628,54 @@ by Ash</a>.</p>
<tbody>
<tr>
-<td>No - we do not plan to use Kubernetes near term</td>
-<td>88</td>
-<td>28.57%</td>
+<td>On virtual machines (for example using AWS EC2)</td>
+<td>64</td>
+<td>31.53</td>
</tr>
<tr>
-<td>Yes - setup on our own via Helm Chart or similar</td>
-<td>65</td>
-<td>21.10%</td>
+<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
+<td>35</td>
+<td>17.24</td>
</tr>
<tr>
-<td>Not yet - but we use Kubernetes in our organization and we could move</td>
-<td>61</td>
-<td>19.81%</td>
+<td>On Kubernetes (using custom deployments)</td>
+<td>29</td>
+<td>14.29</td>
</tr>
<tr>
-<td>Yes - via managed service in the cloud (Composer / Astronomer etc.)</td>
-<td>45</td>
-<td>14.61%</td>
+<td>On premises</td>
+<td>28</td>
+<td>13.79</td>
</tr>
<tr>
-<td>Not yet - but we plan to deploy Kubernetes in our organization soon</td>
-<td>42</td>
-<td>13.64%</td>
+<td>On Kubernetes (using another helm chart)</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using Apache Airflow’s helm chart)</td>
+<td>17</td>
+<td>8.37</td>
</tr>
<tr>
<td>Other</td>
-<td>7</td>
-<td>2.27%</td>
+<td>12</td>
+<td>5.91</td>
</tr>
</tbody>
</table>
-<p>The most interesting thing is that there’s nearly 30% of users who do not use Kubernetes,
-and they are not going to move. This means we should keep other deployment options in
-mind when working on Airflow 2.0. On the other hand, almost 70% of the users already
-use Kubernetes, or it’s a viable option for them.</p>
+<p>Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
+10 percent more than in 2019. There’s slightly increase in usage of Airflow via
+managed services (14.61% in 2019).</p>
-<p><strong>Do you combine multiple DAGs?</strong></p>
+<p><strong>Do you use containerisation for deployment? (single choice)</strong></p>
<table>
<thead>
@@ -1084,35 +1688,35 @@ use Kubernetes, or it’s a viable option for them.</p>
<tbody>
<tr>
-<td>No, I don’t combine multiple DAGs</td>
-<td>127</td>
-<td>41.23%</td>
+<td>Yes, using helm chart / kubernetes</td>
+<td>58</td>
+<td>28.57</td>
</tr>
<tr>
-<td>Yes, through SubDAG</td>
-<td>73</td>
-<td>23.70%</td>
+<td>No, I don’t use containerisation</td>
+<td>57</td>
+<td>28.08</td>
</tr>
<tr>
-<td>Yes, by triggering another DAG</td>
-<td>72</td>
-<td>23.38%</td>
+<td>Yes, single docker image</td>
+<td>49</td>
+<td>24.14</td>
</tr>
<tr>
-<td>Other</td>
-<td>36</td>
-<td>11.69%</td>
+<td>Yes, using docker compose</td>
+<td>39</td>
+<td>19.21</td>
</tr>
</tbody>
</table>
-<p>In the other category, 9 people explicitly mentioned using <code>ExternalTaskSensor</code>,
-and I think it could be treated as running subDAGs by triggering other DAGs.</p>
+<p>Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
+42% of those users use docker-compose for deployments.</p>
-<p><strong>Do you use Airflow Plugins? If yes, what do you use it for?</strong></p>
+<p><strong>How do you distribute your DAGs? (single choice)</strong></p>
<table>
<thead>
@@ -1125,60 +1729,43 @@ and I think it could be treated as running subDAGs by triggering other DAGs.</p>
<tbody>
<tr>
-<td>Adding new operators/sensors and hooks</td>
-<td>187</td>
-<td>60.71%</td>
+<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
+<td>79</td>
+<td>38.92</td>
</tr>
<tr>
-<td>I don’t use Airflow plugins</td>
-<td>109</td>
-<td>35.39%</td>
+<td>Bake them into the docker image</td>
+<td>56</td>
+<td>27.59</td>
</tr>
<tr>
-<td>Adding AppBuilder views & menu items</td>
-<td>31</td>
-<td>10.06%</td>
+<td>Shared files system</td>
+<td>34</td>
+<td>16.75</td>
</tr>
<tr>
-<td>Adding new executor</td>
-<td>18</td>
-<td>5.84%</td>
+<td>Other</td>
+<td>20</td>
+<td>9.85</td>
</tr>
<tr>
-<td>Adding OperatorExtraLinks</td>
-<td>7</td>
-<td>2.27%</td>
+<td>I don’t know</td>
+<td>14</td>
+<td>6.9</td>
</tr>
</tbody>
</table>
-<p>The high percentage - 60% for “Adding new operators/sensors and hooks” is quite a
-surprising result for some of us - especially that you do not actually need to use the
-plugin mechanism to add any of those. Those are standard python objects, and you can
-simply drop your hooks/operators/sensors code to <code>PYTHONPATH</code> environment variable and
-they will work. It seems that this may be a result of a lack of best practices guide.</p>
-
-<p>Plugins are more useful for adding views and menu items - yet only 10%.
-OperatorExtraLinks are even more useful (though relatively new) feature, so it’s not
-entirely surprising they are hardly used.</p>
-
-<p>It was also kind of surprising that someone at all uses plugins to use their own
-executors. We considered removing that option recently - but now we have to rethink
-our approach.</p>
+<p>The most popular way of distributing DAGs seems to be using a synchronizing process. About
+40% of users use this process together with Kubernetes deployments.</p>
-<p><strong>What metrics do you use to monitor Airflow?</strong></p>
+<h2 id="future-of-airflow">Future of Airflow</h2>
-<p>There were a lot of different responses. Some use Prometheus and other services,
-others do not use any monitoring. One of the interesting responses linked to this
-solution for <a href="https://github.com/mastak/airflow_operators_metrics" target="_blank">airflow_operators_metrics</a>.</p>
-
-<h2 id="external-services">External services</h2>
-
-<p><strong>What external services do you use in your Airflow DAGs?</strong></p>
+<p><strong>In your opinion, what could be improved in Airflow? (multiple choice)</strong></p>
<table>
<thead>
@@ -1191,83 +1778,98 @@ solution for <a href="https://github.com/mastak/airflow_operators_metrics" targe
<tbody>
<tr>
-<td>Amazon Web Services</td>
-<td>160</td>
-<td>51.95%</td>
+<td>Web UI</td>
+<td>100</td>
+<td>49.26</td>
</tr>
<tr>
-<td>Internal company systems</td>
-<td>150</td>
-<td>48.7%</td>
+<td>Examples, how-to, onboarding documentation</td>
+<td>90</td>
+<td>44.33</td>
</tr>
<tr>
-<td>Hadoop / Spark / Flink / Other Apache software</td>
-<td>119</td>
-<td>38.64%</td>
+<td>Logging, monitoring and alerting</td>
+<td>90</td>
+<td>44.33</td>
</tr>
<tr>
-<td>Google Cloud Platform / Google APIs</td>
-<td>112</td>
-<td>36.36%</td>
+<td>Technical documentation</td>
+<td>90</td>
+<td>44.33</td>
</tr>
<tr>
-<td>Microsoft Azure</td>
-<td>28</td>
-<td>9.09%</td>
+<td>Scheduler performance</td>
+<td>83</td>
+<td>40.89</td>
</tr>
<tr>
-<td>I do not use external services in my Airflow DAGs</td>
-<td>18</td>
-<td>5.84%</td>
+<td>DAG authoring</td>
+<td>64</td>
+<td>31.53</td>
</tr>
-</tbody>
-</table>
-<p>It’s not surprising that Amazon Web Services is leading the way as they are considered the most mature
-cloud provider. Internal system and other Apache products on the next two positions are
-quite understandable if we take into account that the majority uses Airflow for ETL processes.</p>
-
-<p><strong>What external services do you use in your Airflow DAGs? (Mixed providers)</strong></p>
+<tr>
+<td>Authentication and authorization</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
-<table>
-<thead>
<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
+<td>REST API</td>
+<td>51</td>
+<td>25.12</td>
</tr>
-</thead>
-<tbody>
<tr>
-<td>Google Cloud Platform / Google APIs, Amazon Web Services</td>
+<td>Other</td>
<td>44</td>
-<td>14.29%</td>
+<td>21.67</td>
</tr>
<tr>
-<td>Amazon Web Services, Microsoft Azure</td>
-<td>5</td>
-<td>1.62%</td>
+<td>Reliability</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>External integration e.g. AWS, GCP, Apache products</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Security</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Everything work fine for me</td>
+<td>14</td>
+<td>6.9</td>
</tr>
<tr>
-<td>Google Cloud Platform / Google APIs, Microsoft Azure</td>
+<td>I don’t know</td>
<td>4</td>
-<td>1.3%</td>
+<td>1.97</td>
</tr>
</tbody>
</table>
-<p>This result is not surprising because companies usually prefer to stick with one cloud
-provider.</p>
-
-<p><strong>How do you integrate with external services?</strong></p>
+<p><strong>Which features would most interest you? (multiple choice)</strong></p>
<table>
<thead>
@@ -1280,121 +1882,124 @@ provider.</p>
<tbody>
<tr>
-<td>Using Bash / Python operator</td>
-<td>220</td>
-<td>71.43%</td>
+<td>DAG versioning</td>
+<td>109</td>
+<td>53.69</td>
</tr>
<tr>
-<td>Using existing, dedicated operators / hooks</td>
-<td>217</td>
-<td>70.45%</td>
+<td>Builtin statistics</td>
+<td>71</td>
+<td>34.98</td>
</tr>
<tr>
-<td>Using own, custom operators / hooks</td>
-<td>216</td>
-<td>70.13%</td>
+<td>Improved data lineage</td>
+<td>65</td>
+<td>32.02</td>
</tr>
-</tbody>
-</table>
-<p>We had some anecdotal evidence that people use more Python/Bash operators than the
-dedicated ones - but it looks like all ways of using Airflow to connect to external
-services are equally popular.</p>
+<tr>
+<td>Scheduling at the start of the interval</td>
+<td>63</td>
+<td>31.03</td>
+</tr>
-<h2 id="what-can-be-improved">What can be improved</h2>
+<tr>
+<td>Stateless workers</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
-<p><strong>In your opinion, what could be improved in Airflow?</strong></p>
+<tr>
+<td>More option to configure schedules (time units, increments)</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
-<table>
-<thead>
<tr>
-<th></th>
-<th>No.</th>
-<th>%</th>
+<td>Multi-tenant deployment</td>
+<td>49</td>
+<td>24.14</td>
</tr>
-</thead>
-<tbody>
<tr>
-<td>Scheduler performance</td>
-<td>189</td>
-<td>61.36%</td>
+<td>DAG fetcher (AIP-5)</td>
+<td>39</td>
+<td>19.21</td>
</tr>
<tr>
-<td>Web UI</td>
-<td>180</td>
-<td>58.44%</td>
+<td>Generic transfer operator</td>
+<td>34</td>
+<td>16.75</td>
</tr>
<tr>
-<td>Logging, monitoring and alerting</td>
-<td>145</td>
-<td>47.08%</td>
+<td>Other</td>
+<td>33</td>
+<td>16.26</td>
</tr>
<tr>
-<td>Examples, how-to, onboarding documentation</td>
-<td>143</td>
-<td>46.43%</td>
+<td>I have everything I need</td>
+<td>11</td>
+<td>5.42</td>
</tr>
<tr>
-<td>Technical documentation</td>
-<td>137</td>
-<td>44.48%</td>
+<td>Nothing</td>
+<td>11</td>
+<td>5.42</td>
</tr>
+</tbody>
+</table>
+
+<p><strong>Will you consider migrating to Airflow 2.0? (single choice)</strong></p>
+<table>
+<thead>
<tr>
-<td>Reliability</td>
-<td>112</td>
-<td>36.36%</td>
+<th></th>
+<th>No.</th>
+<th>%</th>
</tr>
+</thead>
+<tbody>
<tr>
-<td>REST API</td>
-<td>96</td>
-<td>31.17%</td>
+<td>Yes, as soon as possible</td>
+<td>81</td>
+<td>39.9</td>
</tr>
<tr>
-<td>Authentication and authorization</td>
-<td>89</td>
-<td>28.9%</td>
+<td>Yes, once it’s mature (for example after 2.1)</td>
+<td>72</td>
+<td>35.47</td>
</tr>
<tr>
-<td>External integration e.g. AWS, GCP, Apache product</td>
-<td>49</td>
-<td>15.91%</td>
+<td>I am already using Airflow 2.0+</td>
+<td>39</td>
+<td>19.21</td>
</tr>
<tr>
-<td>CLI</td>
-<td>41</td>
-<td>13.31%</td>
+<td>I don’t know yet</td>
+<td>8</td>
+<td>3.94</td>
</tr>
<tr>
-<td>I don’t know</td>
-<td>5</td>
-<td>1.62%</td>
+<td>No, I do not plan to migrate</td>
+<td>3</td>
+<td>1.48</td>
</tr>
</tbody>
</table>
-<p>The results are rather quite self-explaining. Improved performance of Airflow, better
-UI, and more telemetry are desirable. But this should go in pair with improved
-documentation and resources about using the Airflow, especially when we
-take into account the problem of onboarding new users.</p>
-
-<p>Another interesting point from that question is that only 16% think that operators
-should be extended and improved. This suggests that we should focus on improving
-Airflow core instead of adding more and more integrations.</p>
-
-<p><strong>What would be the most interesting feature for you?</strong></p>
+<p><strong>What are the features of Airflow 2.0 you are most excited about? (multiple choice)</strong></p>
<table>
<thead>
@@ -1407,70 +2012,76 @@ Airflow core instead of adding more and more integrations.</p>
<tbody>
<tr>
-<td>Production-ready Airflow docker image</td>
-<td>175</td>
-<td>56.82%</td>
+<td>General performance improvements</td>
+<td>133</td>
+<td>65.52</td>
</tr>
<tr>
-<td>Declarative way of writing DAGs / automated DAGs generation</td>
-<td>155</td>
-<td>50.32%</td>
+<td>Refreshed WebUI</td>
+<td>102</td>
+<td>50.25</td>
</tr>
<tr>
-<td>Horizontal Autoscaling</td>
-<td>122</td>
-<td>39.61%</td>
+<td>Scheduler HA</td>
+<td>99</td>
+<td>48.77</td>
</tr>
<tr>
-<td>Asynchronous Operators</td>
-<td>97</td>
-<td>31.49%</td>
+<td>Official docker image</td>
+<td>84</td>
+<td>41.38</td>
</tr>
<tr>
-<td>Stateless web server</td>
-<td>81</td>
-<td>26.3%</td>
+<td>@task decorator</td>
+<td>56</td>
+<td>27.59</td>
</tr>
<tr>
-<td>Knative Executor</td>
-<td>48</td>
-<td>15.58%</td>
+<td>Official helm chart</td>
+<td>51</td>
+<td>25.12</td>
</tr>
<tr>
-<td>I already have all I need</td>
-<td>13</td>
-<td>4.22%</td>
+<td>Providers packages</td>
+<td>41</td>
+<td>20.2</td>
</tr>
-</tbody>
-</table>
-<p>Production Docker image wins, and it’s not a surprise. We all know that deploying
-Airflow is not a plug and play process, and that’s why the official image is being
-worked on by Jarek Potiuk. An unexpected result is that half of the users would like to
-have a declarative way of creating DAGs. That seems to be something that is “against Airflow”
-as we always emphasize the possibility of writing workflows in pure python. Stories
-about DAG generators are not new and confirm that there’s a need for a way to
-declare DAGs.</p>
+<tr>
+<td>Configurable XCom backends</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>CeleryKubernetesExecutor</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
-<h2 id="data">Data</h2>
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
-<p>If you think I missed something and you want to look for insights on your own the data is available
-for you here:</p>
+<h2 id="summary">Summary</h2>
-<ul>
-<li>Original data: <a href="https://storage.googleapis.com/airflow-survey/survey.csv" target="_blank">https://storage.googleapis.com/airflow-survey/survey.csv</a></li>
-<li>Processed: <a href="https://storage.googleapis.com/airflow-survey/airflow_survey_processed.csv" target="_blank">https://storage.googleapis.com/airflow-survey/airflow_survey_processed.csv</a></li>
-</ul>
+<p>From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
+This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and
+are willing to upgrade to new ones.</p>
-<p>The processed data includes multi-choice options one-hot encoded. If you find any interesting
-insight, please update the article (<a href="https://github.com/apache/airflow-site/blob/master/CONTRIBUTE.md" target="_blank">make PR</a>
-to Airflow site).</p>
+<p>Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow
+deployments. However, we hope that with the increase of adoption there will be an increase in people willing
+to share their experience and tools.</p>
</div>
@@ -1528,16 +2139,16 @@ to Airflow site).</p>
<div class="pager">
- <a href="/blog/announcing-new-website/">
+ <a href="/blog/airflow-two-point-oh-is-here/">
<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Previous</button>
</a>
- <a href="/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/">
+ <a >
-<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Next</button>
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" disabled>Next</button>
</a>
</div>
@@ -1550,6 +2161,54 @@ to Airflow site).</p>
+ <div class="blog-pager">
+ <h5 class="header__xsmall--greyish-brown">Read also</h5>
+ <div class="pager">
+ <div class="list-items">
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/community/">Community</a>
+
+
+
+ <a class="tag" href="/blog/tags/survey/">Survey</a>
+
+
+
+ <a class="tag" href="/blog/tags/users/">Users</a>
+
+
+ </div>
+ <span class="bodytext__medium--brownish-grey box-event__blogpost--date">Wed, Dec 11, 2019</span>
+ </div>
+ <p class="box-event__blogpost--header">Airflow Survey 2019</p>
+ <p class="box-event__blogpost--author">Tomek Urbaszek</p>
+ <p class="box-event__blogpost--description">Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss.</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-survey/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
</div>
@@ -1558,7 +2217,7 @@ to Airflow site).</p>
<div class="base-layout--button">
- <a href=https://github.com/apache/airflow-site/edit/master/landing-pages/site/content/en/blog/airflow-survey/index.md>
+ <a href=https://github.com/apache/airflow-site/edit/master/landing-pages/site/content/en/blog/airflow-survey-2020/index.md>
diff --git a/blog/airflow-survey/index.html b/blog/airflow-survey/index.html
index cf62056..fdc86cf 100644
--- a/blog/airflow-survey/index.html
+++ b/blog/airflow-survey/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Airflow Survey 2019">
<meta itemprop="description" content="Receiving and adjusting to our users’ feedback is a must. Let’s see who Airflow users are, how they play with it, and what they miss.">
<meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="1775">
@@ -1550,6 +1550,54 @@ to Airflow site).</p>
+ <div class="blog-pager">
+ <h5 class="header__xsmall--greyish-brown">Read also</h5>
+ <div class="pager">
+ <div class="list-items">
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/community/">Community</a>
+
+
+
+ <a class="tag" href="/blog/tags/survey/">Survey</a>
+
+
+
+ <a class="tag" href="/blog/tags/users/">Users</a>
+
+
+ </div>
+ <span class="bodytext__medium--brownish-grey box-event__blogpost--date">Tue, Mar 9, 2021</span>
+ </div>
+ <p class="box-event__blogpost--header">Airflow Survey 2020</p>
+ <p class="box-event__blogpost--author">Tomek Urbaszek</p>
+ <p class="box-event__blogpost--description">We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-survey-2020/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
</div>
diff --git a/blog/airflow-two-point-oh-is-here/index.html b/blog/airflow-two-point-oh-is-here/index.html
index 0fb173c..97ef57e 100644
--- a/blog/airflow-two-point-oh-is-here/index.html
+++ b/blog/airflow-two-point-oh-is-here/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-12-17T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow 2.0 is here!">
<meta itemprop="description" content="We're proud to announce that Apache Airflow 2.0.0 has been released.">
<meta itemprop="datePublished" content="2020-12-17T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="1166">
@@ -672,10 +672,10 @@ Docs on the <a href="http://airflow.apache.org/docs/apache-airflow-providers/pac
<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Previous</button>
</a>
- <a >
+ <a href="/blog/airflow-survey-2020/">
-<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" disabled>Next</button>
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Next</button>
</a>
</div>
diff --git a/blog/announcing-new-website/index.html b/blog/announcing-new-website/index.html
index 5c7dbf6..f547905 100644
--- a/blog/announcing-new-website/index.html
+++ b/blog/announcing-new-website/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="New Airflow website">
<meta itemprop="description" content="We are thrilled about our new website!">
<meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="282">
diff --git a/blog/apache-airflow-for-newcomers/index.html b/blog/apache-airflow-for-newcomers/index.html
index ca57551..be680e3 100644
--- a/blog/apache-airflow-for-newcomers/index.html
+++ b/blog/apache-airflow-for-newcomers/index.html
@@ -37,14 +37,14 @@ Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-08-17T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow For Newcomers">
<meta itemprop="description" content="Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the process of planning, controlling, and optimizing when a particular task should be done.
Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows using python scripts.">
<meta itemprop="datePublished" content="2020-08-17T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="1070">
diff --git a/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html b/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
index 8095fb0..0a530de 100644
--- a/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
+++ b/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="ApacheCon Europe 2019 — Thoughts and Insights by Airflow Committers">
<meta itemprop="description" content="Here come some thoughts by Airflow committers and contributors from the ApacheCon Europe 2019. Get to know the ASF community!">
<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="138">
diff --git a/blog/documenting-using-local-development-environments/index.html b/blog/documenting-using-local-development-environments/index.html
index 9df8bcd..d28df7e 100644
--- a/blog/documenting-using-local-development-environments/index.html
+++ b/blog/documenting-using-local-development-environments/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Documenting using local development environment">
<meta itemprop="description" content="The story behind documenting local development environment of Apache Airflow">
<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="256">
diff --git a/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html b/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
index a276938..f7607a4 100644
--- a/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
+++ b/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
@@ -37,14 +37,14 @@ About Me I have been writing tech articles on medium as well as my blog for the
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-12-20T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Experience in Google Season of Docs 2019 with Apache Airflow">
<meta itemprop="description" content="I came across Google Season of Docs (GSoD) almost by accident, thanks to my extensive HackerNews and Twitter addiction. I was familiar with the Google Summer of Code but not with this program. It turns out it was the inaugural phase. I read the details, and the process felt a lot like GSoC except that this was about documentation.
About Me I have been writing tech articles on medium as well as my blog for the past 1.">
<meta itemprop="datePublished" content="2019-12-20T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="1521">
diff --git a/blog/experience-with-airflow-as-an-outreachy-intern/index.html b/blog/experience-with-airflow-as-an-outreachy-intern/index.html
index 71f9134..5d0cd5b 100644
--- a/blog/experience-with-airflow-as-an-outreachy-intern/index.html
+++ b/blog/experience-with-airflow-as-an-outreachy-intern/index.html
@@ -37,14 +37,14 @@ Contribution Period The first thing I had to do was choose a project under an or
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-08-30T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Journey with Airflow as an Outreachy Intern">
<meta itemprop="description" content="Outreachy is a program which organises three months paid internships with FOSS projects for people who are typically underrepresented in those projects.
Contribution Period The first thing I had to do was choose a project under an organisation. After going through all the projects I chose “Extending the REST API of Apache Airflow”, because I had a good idea of what REST API(s) are, so I thought it would be easier to get started with the contributions.">
<meta itemprop="datePublished" content="2020-08-30T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="532">
diff --git a/blog/implementing-stable-api-for-apache-airflow/index.html b/blog/implementing-stable-api-for-apache-airflow/index.html
index 40c9441..b2652c4 100644
--- a/blog/implementing-stable-api-for-apache-airflow/index.html
+++ b/blog/implementing-stable-api-for-apache-airflow/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-07-19T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Implementing Stable API for Apache Airflow">
<meta itemprop="description" content="An Outreachy intern's progress report on contributing to Apache Airflow REST API.">
<meta itemprop="datePublished" content="2020-07-19T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="703">
diff --git a/blog/index.html b/blog/index.html
index 4fa31d7..b13d6c0 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -484,6 +484,45 @@ if (!doNotTrack) {
<div class="tags-container">
+ <a class="tag" href="/blog/tags/community/">Community</a>
+
+
+
+ <a class="tag" href="/blog/tags/survey/">Survey</a>
+
+
+
+ <a class="tag" href="/blog/tags/users/">Users</a>
+
+
+ </div>
+ <span class="bodytext__medium--brownish-grey box-event__blogpost--date">Tue, Mar 9, 2021</span>
+ </div>
+ <p class="box-event__blogpost--header">Airflow Survey 2020</p>
+ <p class="box-event__blogpost--author">Tomek Urbaszek</p>
+ <p class="box-event__blogpost--description">We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-survey-2020/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
<a class="tag" href="/blog/tags/release/">Release</a>
diff --git a/blog/index.xml b/blog/index.xml
index e5cba91..b45ddfa 100644
--- a/blog/index.xml
+++ b/blog/index.xml
@@ -14,6 +14,1605 @@
<item>
+ <title>Blog: Airflow Survey 2020</title>
+ <link>/blog/airflow-survey-2020/</link>
+ <pubDate>Tue, 09 Mar 2021 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-survey-2020/</guid>
+ <description>
+
+
+
+
+<h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
+
+<p>World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
+crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
+an amount of active contributors. So listening and understanding our community is of high importance.</p>
+
+<p>It&rsquo;s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!</p>
+
+<h2 id="overview-of-the-user">Overview of the user</h2>
+
+<p><img src="What_best_describes_your_current_occupation.png" alt="" /></p>
+
+<p><strong>What best describes your current occupation? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Data Engineer</td>
+<td>115</td>
+<td>56.65</td>
+</tr>
+
+<tr>
+<td>Developer</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>DevOps</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Solutions Architect</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>Data Scientist</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>Data Analyst</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>Support Engineer</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> with
+visible slight increase in ML use cases.</p>
+
+<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
+
+<p><strong>How many people work in your company? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>200+</td>
+<td>107</td>
+<td>52.71</td>
+</tr>
+
+<tr>
+<td>51-200</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>11-50</td>
+<td>37</td>
+<td>18.23</td>
+</tr>
+
+<tr>
+<td>1-10</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1-5</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>6-20</td>
+<td>75</td>
+<td>36.95</td>
+</tr>
+
+<tr>
+<td>21-50</td>
+<td>23</td>
+<td>11.33</td>
+</tr>
+
+<tr>
+<td>50+</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+</tbody>
+</table>
+
+<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
+
+<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we are happy with Airflow</td>
+<td>174</td>
+<td>85.71</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
+(8.37% both).</p>
+
+<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api" target="_blank">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.</p>
+
+<p>The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.</p>
+
+<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I wish we could</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>No</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+</tbody>
+</table>
+
+<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there&rsquo;s something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.</p>
+
+<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020 %</th>
+<th>2019 %</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Very Likely</td>
+<td>125</td>
+<td>61.58</td>
+<td>45.45%</td>
+</tr>
+
+<tr>
+<td>Likely</td>
+<td>62</td>
+<td>30.54</td>
+<td>40.26%</td>
+</tr>
+
+<tr>
+<td>Neutral</td>
+<td>11</td>
+<td>5.42</td>
+<td>10.71%</td>
+</tr>
+
+<tr>
+<td>Unlikely</td>
+<td>3</td>
+<td>1.48</td>
+<td>2.60%</td>
+</tr>
+
+<tr>
+<td>Very unlikely</td>
+<td>2</td>
+<td>0.99</td>
+<td>0.97%</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
+
+<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Documentation</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Airflow website</td>
+<td>139</td>
+<td>68.47</td>
+</tr>
+
+<tr>
+<td>Slack</td>
+<td>128</td>
+<td>63.05</td>
+</tr>
+
+<tr>
+<td>Github</td>
+<td>127</td>
+<td>62.56</td>
+</tr>
+
+<tr>
+<td>Stack Overflow</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Airflow Summit Videos</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>The dev mailing list</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Awesome Apache Airflow repository</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here we see that Airflow documentation is the crucial source of information. What&rsquo;s interesting is that more
+than 60% of users are getting information from Github and Slack channels.</p>
+
+<p><img src="Where_are_you_based.png" alt="" /></p>
+
+<h2 id="airflow-uses-cases">Airflow uses cases</h2>
+
+<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we use vanilla Airflow</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Yes, we have small patches (no fork)</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Yes, we have separate fork</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No guide on best practises on developing DAGs</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>There is no easy option to launch Airflow</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Small number of tutorials on different aspects of using Airflow</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Documentation is not clear enough</td>
+<td>53</td>
+<td>26.11</td>
+</tr>
+
+<tr>
+<td>There is no easy option to deploy DAGs to an Airflow instance</td>
+<td>52</td>
+<td>25.62</td>
+</tr>
+
+<tr>
+<td>No problems</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Small number of blogs regarding Airflow</td>
+<td>30</td>
+<td>14.78</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Original Airflow Graphical User Interface</td>
+<td>199</td>
+<td>98.03</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>88</td>
+<td>43.35</td>
+</tr>
+
+<tr>
+<td>API</td>
+<td>48</td>
+<td>23.65</td>
+</tr>
+
+<tr>
+<td>Custom (own created) Airflow Graphical User Interface</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, by triggering another DAG</td>
+<td>87</td>
+<td>42.86</td>
+</tr>
+
+<tr>
+<td>No, I don&rsquo;t combine multiple DAGs</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Yes, through SubDAG</td>
+<td>40</td>
+<td>19.7</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using existing dedicated operators / hooks</td>
+<td>147</td>
+<td>72.41</td>
+</tr>
+
+<tr>
+<td>Using Bash / Python operator</td>
+<td>140</td>
+<td>68.97</td>
+</tr>
+
+<tr>
+<td>Using own custom operators / hooks</td>
+<td>138</td>
+<td>67.98</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Amazon Web Services</td>
+<td>121</td>
+<td>59.61</td>
+</tr>
+
+<tr>
+<td>Internal company systems</td>
+<td>113</td>
+<td>55.67</td>
+</tr>
+
+<tr>
+<td>Google Cloud Platform / Google APIs</td>
+<td>97</td>
+<td>47.78</td>
+</tr>
+
+<tr>
+<td>Hadoop / Spark / Flink / Other Apache software</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Microsoft Azure</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>I do not use external services in my Airflow DAGs</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+</tbody>
+</table>
+
+<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt="" /></p>
+
+<p><strong>Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Adding new operators/sensors and hooks</td>
+<td>119</td>
+<td>58.62</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t use Airflow plugins</td>
+<td>69</td>
+<td>33.99</td>
+</tr>
+
+<tr>
+<td>Adding AppBuilder views &amp; menu items</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>Adding new executors</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Adding OperatorExtraLinks</td>
+<td>13</td>
+<td>6.4</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td></td>
+<td></td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you use Airflow&rsquo;s data lineage feature? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, I will use such feature if fully supported in Airflow</td>
+<td>105</td>
+<td>51.72</td>
+</tr>
+
+<tr>
+<td>No, data lineage isn’t a concern for my usage.</td>
+<td>68</td>
+<td>33.5</td>
+</tr>
+
+<tr>
+<td>Yes, I use another data lineage product</td>
+<td>24</td>
+<td>11.82</td>
+</tr>
+
+<tr>
+<td>Yes, I use custom implementation</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+
+<tr>
+<td>Yes, I use Airflow&rsquo;s experimental data lineage feature</td>
+<td>1</td>
+<td>0.49</td>
+</tr>
+</tbody>
+</table>
+
+<p>When asked what lineage product users use, the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.</p>
+
+<h2 id="deployment">Deployment</h2>
+
+<p><strong>How many active DAGs do you have in your largest Airflow instance? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 20</td>
+<td>64</td>
+<td>32</td>
+</tr>
+
+<tr>
+<td>21-40</td>
+<td>33</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>41-60</td>
+<td>13</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>61-100</td>
+<td>32</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>201-300</td>
+<td>8</td>
+<td>4</td>
+</tr>
+
+<tr>
+<td>301-999</td>
+<td>12</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>1000+</td>
+<td>10</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What is the maximum number of tasks that you have used in one DAG? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 10</td>
+<td>42</td>
+<td>21</td>
+</tr>
+
+<tr>
+<td>11-20</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>21-30</td>
+<td>15</td>
+<td>7</td>
+</tr>
+
+<tr>
+<td>31-40</td>
+<td>11</td>
+<td>5</td>
+</tr>
+
+<tr>
+<td>41-50</td>
+<td>22</td>
+<td>11</td>
+</tr>
+
+<tr>
+<td>51-100</td>
+<td>39</td>
+<td>19</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>201-500</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>501+</td>
+<td>11</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which version of Airflow do you use currently? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1.10.14</td>
+<td>55</td>
+<td>27.09</td>
+</tr>
+
+<tr>
+<td>2.0.0+</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+
+<tr>
+<td>1.10.12</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>1.10.10</td>
+<td>26</td>
+<td>12.81</td>
+</tr>
+
+<tr>
+<td>1.10.11</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>1.10.5 or older</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>1.10.9</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>1.10.13</td>
+<td>7</td>
+<td>3.45</td>
+</tr>
+
+<tr>
+<td>1.10.6</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.7</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.8</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>This was probably one of the most important questions in the survey. While it&rsquo;s good to see
+that more than 60% of users use one of three latest Airflow versions, it&rsquo;s worrying that the rest
+are using versions that are old or have known security vulnerabilities.</p>
+
+<p>Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.</p>
+
+<p><strong>What meta-database do you use? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Postgres 12</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Postgres 9.6</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Postgres 11</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>MySQL 5.7</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>MySQL 8.0</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Postgres 10</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>Postgres 13</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p>This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.</p>
+
+<p>It&rsquo;s good to know that users rather avoid using SQLite in production deployments!</p>
+
+<p><strong>What executor type do you use? (single choice)</strong></p>
+
+<p><img src="What_executor_type_do_you_use.png" alt="" /></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020</th>
+<th>2019</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Celery</td>
+<td>100</td>
+<td>49.26%</td>
+<td>44.81%</td>
+</tr>
+
+<tr>
+<td>Kubernetes</td>
+<td>48</td>
+<td>23.65%</td>
+<td>16.88%</td>
+</tr>
+
+<tr>
+<td>Local</td>
+<td>40</td>
+<td>19.7%</td>
+<td>27.60%</td>
+</tr>
+
+<tr>
+<td>Sequential</td>
+<td>10</td>
+<td>4.93%</td>
+<td>7.14%</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>5</td>
+<td>2.46%</td>
+<td>3.57</td>
+</tr>
+</tbody>
+</table>
+
+<p>In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users&rsquo; deployments are growing, and they need more scalable solutions.</p>
+
+<p>Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.</p>
+
+<p><strong>What metrics do you use to monitor Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I do not use monitoring</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>External monitoring service</td>
+<td>60</td>
+<td>29.56</td>
+</tr>
+
+<tr>
+<td>Information from metadatabase</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Statsd</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+</tbody>
+</table>
+
+<p>The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.</p>
+
+<p><strong>How do you deploy Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>On virtual machines (for example using AWS EC2)</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
+<td>35</td>
+<td>17.24</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using custom deployments)</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+
+<tr>
+<td>On premises</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using another helm chart)</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using Apache Airflow&rsquo;s helm chart)</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
+10 percent more than in 2019. There&rsquo;s slightly increase in usage of Airflow via
+managed services (14.61% in 2019).</p>
+
+<p><strong>Do you use containerisation for deployment? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, using helm chart / kubernetes</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>No, I don’t use containerisation</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Yes, single docker image</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Yes, using docker compose</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+</tbody>
+</table>
+
+<p>Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
+42% of those users use docker-compose for deployments.</p>
+
+<p><strong>How do you distribute your DAGs? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Bake them into the docker image</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Shared files system</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+</tbody>
+</table>
+
+<p>The most popular way of distributing DAGs seems to be using a synchronizing process. About
+40% of users use this process together with Kubernetes deployments.</p>
+
+<h2 id="future-of-airflow">Future of Airflow</h2>
+
+<p><strong>In your opinion, what could be improved in Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Web UI</td>
+<td>100</td>
+<td>49.26</td>
+</tr>
+
+<tr>
+<td>Examples, how-to, onboarding documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Logging, monitoring and alerting</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Technical documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Scheduler performance</td>
+<td>83</td>
+<td>40.89</td>
+</tr>
+
+<tr>
+<td>DAG authoring</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Authentication and authorization</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>REST API</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>Reliability</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>External integration e.g. AWS, GCP, Apache products</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Security</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Everything work fine for me</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which features would most interest you? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>DAG versioning</td>
+<td>109</td>
+<td>53.69</td>
+</tr>
+
+<tr>
+<td>Builtin statistics</td>
+<td>71</td>
+<td>34.98</td>
+</tr>
+
+<tr>
+<td>Improved data lineage</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>Scheduling at the start of the interval</td>
+<td>63</td>
+<td>31.03</td>
+</tr>
+
+<tr>
+<td>Stateless workers</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>More option to configure schedules (time units, increments)</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Multi-tenant deployment</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>DAG fetcher (AIP-5)</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>Generic transfer operator</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>I have everything I need</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+
+<tr>
+<td>Nothing</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Will you consider migrating to Airflow 2.0? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, as soon as possible</td>
+<td>81</td>
+<td>39.9</td>
+</tr>
+
+<tr>
+<td>Yes, once it’s mature (for example after 2.1)</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>I am already using Airflow 2.0+</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t know yet</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>No, I do not plan to migrate</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What are the features of Airflow 2.0 you are most excited about? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>General performance improvements</td>
+<td>133</td>
+<td>65.52</td>
+</tr>
+
+<tr>
+<td>Refreshed WebUI</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>Scheduler HA</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>Official docker image</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>@task decorator</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Official helm chart</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Providers packages</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>Configurable XCom backends</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>CeleryKubernetesExecutor</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<h2 id="summary">Summary</h2>
+
+<p>From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
+This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and
+are willing to upgrade to new ones.</p>
+
+<p>Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow
+deployments. However, we hope that with the increase of adoption there will be an increase in people willing
+to share their experience and tools.</p>
+
+ </description>
+ </item>
+
+ <item>
<title>Blog: Apache Airflow 2.0 is here!</title>
<link>/blog/airflow-two-point-oh-is-here/</link>
<pubDate>Thu, 17 Dec 2020 00:00:00 +0000</pubDate>
diff --git a/blog/its-a-breeze-to-develop-apache-airflow/index.html b/blog/its-a-breeze-to-develop-apache-airflow/index.html
index 7cd55b6..89f7d1c 100644
--- a/blog/its-a-breeze-to-develop-apache-airflow/index.html
+++ b/blog/its-a-breeze-to-develop-apache-airflow/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2021-03-08T00:32:31+01:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2021-03-09T22:55:49+01:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="It's a "Breeze" to develop Apache Airflow">
<meta itemprop="description" content="A Principal Software Engineer's journey to developer productivity. Learn how Jarek and his team speeded up and simplified Airflow development for the community.">
<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2021-03-08T00:32:31+01:00" />
+<meta itemprop="dateModified" content="2021-03-09T22:55:49+01:00" />
<meta itemprop="wordCount" content="124">
diff --git a/blog/tags/community/index.html b/blog/tags/community/index.html
index db250ad..bede26f 100644
--- a/blog/tags/community/index.html
+++ b/blog/tags/community/index.html
@@ -37,7 +37,7 @@
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="og:updated_time" content="2020-08-30T00:00:00+00:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="og:updated_time" content="2021-03-09T00:00:00+00:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Community">
<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
@@ -496,6 +496,45 @@ if (!doNotTrack) {
<a class="tag" href="/blog/tags/community/">Community</a>
+
+ <a class="tag" href="/blog/tags/survey/">Survey</a>
+
+
+
+ <a class="tag" href="/blog/tags/users/">Users</a>
+
+
+ </div>
+ <span class="bodytext__medium--brownish-grey box-event__blogpost--date">Tue, Mar 9, 2021</span>
+ </div>
+ <p class="box-event__blogpost--header">Airflow Survey 2020</p>
+ <p class="box-event__blogpost--author">Tomek Urbaszek</p>
+ <p class="box-event__blogpost--description">We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-survey-2020/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/community/">Community</a>
+
+
</div>
<span class="bodytext__medium--brownish-grey box-event__blogpost--date">Sun, Aug 30, 2020</span>
</div>
diff --git a/blog/tags/community/index.xml b/blog/tags/community/index.xml
index ada5143..a5e4ef1 100644
--- a/blog/tags/community/index.xml
+++ b/blog/tags/community/index.xml
@@ -4,7 +4,7 @@
<link>/blog/tags/community/</link>
<description>Recent content in Community on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
- <lastBuildDate>Sun, 30 Aug 2020 00:00:00 +0000</lastBuildDate>
+ <lastBuildDate>Tue, 09 Mar 2021 00:00:00 +0000</lastBuildDate>
<atom:link href="/blog/tags/community/index.xml" rel="self" type="application/rss+xml" />
@@ -15,6 +15,1605 @@
<item>
+ <title>Blog: Airflow Survey 2020</title>
+ <link>/blog/airflow-survey-2020/</link>
+ <pubDate>Tue, 09 Mar 2021 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-survey-2020/</guid>
+ <description>
+
+
+
+
+<h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
+
+<p>World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
+crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
+an amount of active contributors. So listening and understanding our community is of high importance.</p>
+
+<p>It&rsquo;s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!</p>
+
+<h2 id="overview-of-the-user">Overview of the user</h2>
+
+<p><img src="What_best_describes_your_current_occupation.png" alt="" /></p>
+
+<p><strong>What best describes your current occupation? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Data Engineer</td>
+<td>115</td>
+<td>56.65</td>
+</tr>
+
+<tr>
+<td>Developer</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>DevOps</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Solutions Architect</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>Data Scientist</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>Data Analyst</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>Support Engineer</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> with
+visible slight increase in ML use cases.</p>
+
+<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
+
+<p><strong>How many people work in your company? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>200+</td>
+<td>107</td>
+<td>52.71</td>
+</tr>
+
+<tr>
+<td>51-200</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>11-50</td>
+<td>37</td>
+<td>18.23</td>
+</tr>
+
+<tr>
+<td>1-10</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1-5</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>6-20</td>
+<td>75</td>
+<td>36.95</td>
+</tr>
+
+<tr>
+<td>21-50</td>
+<td>23</td>
+<td>11.33</td>
+</tr>
+
+<tr>
+<td>50+</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+</tbody>
+</table>
+
+<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
+
+<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we are happy with Airflow</td>
+<td>174</td>
+<td>85.71</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
+(8.37% both).</p>
+
+<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api" target="_blank">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.</p>
+
+<p>The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.</p>
+
+<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I wish we could</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>No</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+</tbody>
+</table>
+
+<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there&rsquo;s something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.</p>
+
+<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020 %</th>
+<th>2019 %</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Very Likely</td>
+<td>125</td>
+<td>61.58</td>
+<td>45.45%</td>
+</tr>
+
+<tr>
+<td>Likely</td>
+<td>62</td>
+<td>30.54</td>
+<td>40.26%</td>
+</tr>
+
+<tr>
+<td>Neutral</td>
+<td>11</td>
+<td>5.42</td>
+<td>10.71%</td>
+</tr>
+
+<tr>
+<td>Unlikely</td>
+<td>3</td>
+<td>1.48</td>
+<td>2.60%</td>
+</tr>
+
+<tr>
+<td>Very unlikely</td>
+<td>2</td>
+<td>0.99</td>
+<td>0.97%</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
+
+<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Documentation</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Airflow website</td>
+<td>139</td>
+<td>68.47</td>
+</tr>
+
+<tr>
+<td>Slack</td>
+<td>128</td>
+<td>63.05</td>
+</tr>
+
+<tr>
+<td>Github</td>
+<td>127</td>
+<td>62.56</td>
+</tr>
+
+<tr>
+<td>Stack Overflow</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Airflow Summit Videos</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>The dev mailing list</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Awesome Apache Airflow repository</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here we see that Airflow documentation is the crucial source of information. What&rsquo;s interesting is that more
+than 60% of users are getting information from Github and Slack channels.</p>
+
+<p><img src="Where_are_you_based.png" alt="" /></p>
+
+<h2 id="airflow-uses-cases">Airflow uses cases</h2>
+
+<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we use vanilla Airflow</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Yes, we have small patches (no fork)</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Yes, we have separate fork</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No guide on best practises on developing DAGs</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>There is no easy option to launch Airflow</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Small number of tutorials on different aspects of using Airflow</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Documentation is not clear enough</td>
+<td>53</td>
+<td>26.11</td>
+</tr>
+
+<tr>
+<td>There is no easy option to deploy DAGs to an Airflow instance</td>
+<td>52</td>
+<td>25.62</td>
+</tr>
+
+<tr>
+<td>No problems</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Small number of blogs regarding Airflow</td>
+<td>30</td>
+<td>14.78</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Original Airflow Graphical User Interface</td>
+<td>199</td>
+<td>98.03</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>88</td>
+<td>43.35</td>
+</tr>
+
+<tr>
+<td>API</td>
+<td>48</td>
+<td>23.65</td>
+</tr>
+
+<tr>
+<td>Custom (own created) Airflow Graphical User Interface</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, by triggering another DAG</td>
+<td>87</td>
+<td>42.86</td>
+</tr>
+
+<tr>
+<td>No, I don&rsquo;t combine multiple DAGs</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Yes, through SubDAG</td>
+<td>40</td>
+<td>19.7</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using existing dedicated operators / hooks</td>
+<td>147</td>
+<td>72.41</td>
+</tr>
+
+<tr>
+<td>Using Bash / Python operator</td>
+<td>140</td>
+<td>68.97</td>
+</tr>
+
+<tr>
+<td>Using own custom operators / hooks</td>
+<td>138</td>
+<td>67.98</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Amazon Web Services</td>
+<td>121</td>
+<td>59.61</td>
+</tr>
+
+<tr>
+<td>Internal company systems</td>
+<td>113</td>
+<td>55.67</td>
+</tr>
+
+<tr>
+<td>Google Cloud Platform / Google APIs</td>
+<td>97</td>
+<td>47.78</td>
+</tr>
+
+<tr>
+<td>Hadoop / Spark / Flink / Other Apache software</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Microsoft Azure</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>I do not use external services in my Airflow DAGs</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+</tbody>
+</table>
+
+<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt="" /></p>
+
+<p><strong>Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Adding new operators/sensors and hooks</td>
+<td>119</td>
+<td>58.62</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t use Airflow plugins</td>
+<td>69</td>
+<td>33.99</td>
+</tr>
+
+<tr>
+<td>Adding AppBuilder views &amp; menu items</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>Adding new executors</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Adding OperatorExtraLinks</td>
+<td>13</td>
+<td>6.4</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td></td>
+<td></td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you use Airflow&rsquo;s data lineage feature? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, I will use such feature if fully supported in Airflow</td>
+<td>105</td>
+<td>51.72</td>
+</tr>
+
+<tr>
+<td>No, data lineage isn’t a concern for my usage.</td>
+<td>68</td>
+<td>33.5</td>
+</tr>
+
+<tr>
+<td>Yes, I use another data lineage product</td>
+<td>24</td>
+<td>11.82</td>
+</tr>
+
+<tr>
+<td>Yes, I use custom implementation</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+
+<tr>
+<td>Yes, I use Airflow&rsquo;s experimental data lineage feature</td>
+<td>1</td>
+<td>0.49</td>
+</tr>
+</tbody>
+</table>
+
+<p>When asked what lineage product users use, the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.</p>
+
+<h2 id="deployment">Deployment</h2>
+
+<p><strong>How many active DAGs do you have in your largest Airflow instance? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 20</td>
+<td>64</td>
+<td>32</td>
+</tr>
+
+<tr>
+<td>21-40</td>
+<td>33</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>41-60</td>
+<td>13</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>61-100</td>
+<td>32</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>201-300</td>
+<td>8</td>
+<td>4</td>
+</tr>
+
+<tr>
+<td>301-999</td>
+<td>12</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>1000+</td>
+<td>10</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What is the maximum number of tasks that you have used in one DAG? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 10</td>
+<td>42</td>
+<td>21</td>
+</tr>
+
+<tr>
+<td>11-20</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>21-30</td>
+<td>15</td>
+<td>7</td>
+</tr>
+
+<tr>
+<td>31-40</td>
+<td>11</td>
+<td>5</td>
+</tr>
+
+<tr>
+<td>41-50</td>
+<td>22</td>
+<td>11</td>
+</tr>
+
+<tr>
+<td>51-100</td>
+<td>39</td>
+<td>19</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>201-500</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>501+</td>
+<td>11</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which version of Airflow do you use currently? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1.10.14</td>
+<td>55</td>
+<td>27.09</td>
+</tr>
+
+<tr>
+<td>2.0.0+</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+
+<tr>
+<td>1.10.12</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>1.10.10</td>
+<td>26</td>
+<td>12.81</td>
+</tr>
+
+<tr>
+<td>1.10.11</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>1.10.5 or older</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>1.10.9</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>1.10.13</td>
+<td>7</td>
+<td>3.45</td>
+</tr>
+
+<tr>
+<td>1.10.6</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.7</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.8</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>This was probably one of the most important questions in the survey. While it&rsquo;s good to see
+that more than 60% of users use one of three latest Airflow versions, it&rsquo;s worrying that the rest
+are using versions that are old or have known security vulnerabilities.</p>
+
+<p>Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.</p>
+
+<p><strong>What meta-database do you use? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Postgres 12</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Postgres 9.6</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Postgres 11</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>MySQL 5.7</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>MySQL 8.0</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Postgres 10</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>Postgres 13</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p>This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.</p>
+
+<p>It&rsquo;s good to know that users rather avoid using SQLite in production deployments!</p>
+
+<p><strong>What executor type do you use? (single choice)</strong></p>
+
+<p><img src="What_executor_type_do_you_use.png" alt="" /></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020</th>
+<th>2019</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Celery</td>
+<td>100</td>
+<td>49.26%</td>
+<td>44.81%</td>
+</tr>
+
+<tr>
+<td>Kubernetes</td>
+<td>48</td>
+<td>23.65%</td>
+<td>16.88%</td>
+</tr>
+
+<tr>
+<td>Local</td>
+<td>40</td>
+<td>19.7%</td>
+<td>27.60%</td>
+</tr>
+
+<tr>
+<td>Sequential</td>
+<td>10</td>
+<td>4.93%</td>
+<td>7.14%</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>5</td>
+<td>2.46%</td>
+<td>3.57</td>
+</tr>
+</tbody>
+</table>
+
+<p>In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users&rsquo; deployments are growing, and they need more scalable solutions.</p>
+
+<p>Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.</p>
+
+<p><strong>What metrics do you use to monitor Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I do not use monitoring</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>External monitoring service</td>
+<td>60</td>
+<td>29.56</td>
+</tr>
+
+<tr>
+<td>Information from metadatabase</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Statsd</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+</tbody>
+</table>
+
+<p>The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.</p>
+
+<p><strong>How do you deploy Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>On virtual machines (for example using AWS EC2)</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
+<td>35</td>
+<td>17.24</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using custom deployments)</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+
+<tr>
+<td>On premises</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using another helm chart)</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using Apache Airflow&rsquo;s helm chart)</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
+10 percent more than in 2019. There&rsquo;s slightly increase in usage of Airflow via
+managed services (14.61% in 2019).</p>
+
+<p><strong>Do you use containerisation for deployment? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, using helm chart / kubernetes</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>No, I don’t use containerisation</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Yes, single docker image</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Yes, using docker compose</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+</tbody>
+</table>
+
+<p>Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
+42% of those users use docker-compose for deployments.</p>
+
+<p><strong>How do you distribute your DAGs? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Bake them into the docker image</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Shared files system</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+</tbody>
+</table>
+
+<p>The most popular way of distributing DAGs seems to be using a synchronizing process. About
+40% of users use this process together with Kubernetes deployments.</p>
+
+<h2 id="future-of-airflow">Future of Airflow</h2>
+
+<p><strong>In your opinion, what could be improved in Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Web UI</td>
+<td>100</td>
+<td>49.26</td>
+</tr>
+
+<tr>
+<td>Examples, how-to, onboarding documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Logging, monitoring and alerting</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Technical documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Scheduler performance</td>
+<td>83</td>
+<td>40.89</td>
+</tr>
+
+<tr>
+<td>DAG authoring</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Authentication and authorization</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>REST API</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>Reliability</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>External integration e.g. AWS, GCP, Apache products</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Security</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Everything work fine for me</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which features would most interest you? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>DAG versioning</td>
+<td>109</td>
+<td>53.69</td>
+</tr>
+
+<tr>
+<td>Builtin statistics</td>
+<td>71</td>
+<td>34.98</td>
+</tr>
+
+<tr>
+<td>Improved data lineage</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>Scheduling at the start of the interval</td>
+<td>63</td>
+<td>31.03</td>
+</tr>
+
+<tr>
+<td>Stateless workers</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>More option to configure schedules (time units, increments)</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Multi-tenant deployment</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>DAG fetcher (AIP-5)</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>Generic transfer operator</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>I have everything I need</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+
+<tr>
+<td>Nothing</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Will you consider migrating to Airflow 2.0? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, as soon as possible</td>
+<td>81</td>
+<td>39.9</td>
+</tr>
+
+<tr>
+<td>Yes, once it’s mature (for example after 2.1)</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>I am already using Airflow 2.0+</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t know yet</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>No, I do not plan to migrate</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What are the features of Airflow 2.0 you are most excited about? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>General performance improvements</td>
+<td>133</td>
+<td>65.52</td>
+</tr>
+
+<tr>
+<td>Refreshed WebUI</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>Scheduler HA</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>Official docker image</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>@task decorator</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Official helm chart</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Providers packages</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>Configurable XCom backends</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>CeleryKubernetesExecutor</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<h2 id="summary">Summary</h2>
+
+<p>From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
+This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and
+are willing to upgrade to new ones.</p>
+
+<p>Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow
+deployments. However, we hope that with the increase of adoption there will be an increase in people willing
+to share their experience and tools.</p>
+
+ </description>
+ </item>
+
+ <item>
<title>Blog: Journey with Airflow as an Outreachy Intern</title>
<link>/blog/experience-with-airflow-as-an-outreachy-intern/</link>
<pubDate>Sun, 30 Aug 2020 00:00:00 +0000</pubDate>
diff --git a/blog/tags/survey/index.html b/blog/tags/survey/index.html
index a52f407..0a97094 100644
--- a/blog/tags/survey/index.html
+++ b/blog/tags/survey/index.html
@@ -37,7 +37,7 @@
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="og:updated_time" content="2019-12-11T00:00:00+00:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="og:updated_time" content="2021-03-09T00:00:00+00:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="survey">
<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
@@ -505,6 +505,45 @@ if (!doNotTrack) {
</div>
+ <span class="bodytext__medium--brownish-grey box-event__blogpost--date">Tue, Mar 9, 2021</span>
+ </div>
+ <p class="box-event__blogpost--header">Airflow Survey 2020</p>
+ <p class="box-event__blogpost--author">Tomek Urbaszek</p>
+ <p class="box-event__blogpost--description">We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-survey-2020/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/community/">Community</a>
+
+
+
+ <a class="tag" href="/blog/tags/survey/">Survey</a>
+
+
+
+ <a class="tag" href="/blog/tags/users/">Users</a>
+
+
+ </div>
<span class="bodytext__medium--brownish-grey box-event__blogpost--date">Wed, Dec 11, 2019</span>
</div>
<p class="box-event__blogpost--header">Airflow Survey 2019</p>
diff --git a/blog/tags/survey/index.xml b/blog/tags/survey/index.xml
index a5b9fe9..de9c398 100644
--- a/blog/tags/survey/index.xml
+++ b/blog/tags/survey/index.xml
@@ -4,7 +4,7 @@
<link>/blog/tags/survey/</link>
<description>Recent content in survey on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
- <lastBuildDate>Wed, 11 Dec 2019 00:00:00 +0000</lastBuildDate>
+ <lastBuildDate>Tue, 09 Mar 2021 00:00:00 +0000</lastBuildDate>
<atom:link href="/blog/tags/survey/index.xml" rel="self" type="application/rss+xml" />
@@ -15,6 +15,1605 @@
<item>
+ <title>Blog: Airflow Survey 2020</title>
+ <link>/blog/airflow-survey-2020/</link>
+ <pubDate>Tue, 09 Mar 2021 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-survey-2020/</guid>
+ <description>
+
+
+
+
+<h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
+
+<p>World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
+crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
+an amount of active contributors. So listening and understanding our community is of high importance.</p>
+
+<p>It&rsquo;s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!</p>
+
+<h2 id="overview-of-the-user">Overview of the user</h2>
+
+<p><img src="What_best_describes_your_current_occupation.png" alt="" /></p>
+
+<p><strong>What best describes your current occupation? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Data Engineer</td>
+<td>115</td>
+<td>56.65</td>
+</tr>
+
+<tr>
+<td>Developer</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>DevOps</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Solutions Architect</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>Data Scientist</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>Data Analyst</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>Support Engineer</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> with
+visible slight increase in ML use cases.</p>
+
+<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
+
+<p><strong>How many people work in your company? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>200+</td>
+<td>107</td>
+<td>52.71</td>
+</tr>
+
+<tr>
+<td>51-200</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>11-50</td>
+<td>37</td>
+<td>18.23</td>
+</tr>
+
+<tr>
+<td>1-10</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1-5</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>6-20</td>
+<td>75</td>
+<td>36.95</td>
+</tr>
+
+<tr>
+<td>21-50</td>
+<td>23</td>
+<td>11.33</td>
+</tr>
+
+<tr>
+<td>50+</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+</tbody>
+</table>
+
+<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
+
+<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we are happy with Airflow</td>
+<td>174</td>
+<td>85.71</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
+(8.37% both).</p>
+
+<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api" target="_blank">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.</p>
+
+<p>The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.</p>
+
+<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I wish we could</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>No</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+</tbody>
+</table>
+
+<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there&rsquo;s something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.</p>
+
+<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020 %</th>
+<th>2019 %</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Very Likely</td>
+<td>125</td>
+<td>61.58</td>
+<td>45.45%</td>
+</tr>
+
+<tr>
+<td>Likely</td>
+<td>62</td>
+<td>30.54</td>
+<td>40.26%</td>
+</tr>
+
+<tr>
+<td>Neutral</td>
+<td>11</td>
+<td>5.42</td>
+<td>10.71%</td>
+</tr>
+
+<tr>
+<td>Unlikely</td>
+<td>3</td>
+<td>1.48</td>
+<td>2.60%</td>
+</tr>
+
+<tr>
+<td>Very unlikely</td>
+<td>2</td>
+<td>0.99</td>
+<td>0.97%</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
+
+<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Documentation</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Airflow website</td>
+<td>139</td>
+<td>68.47</td>
+</tr>
+
+<tr>
+<td>Slack</td>
+<td>128</td>
+<td>63.05</td>
+</tr>
+
+<tr>
+<td>Github</td>
+<td>127</td>
+<td>62.56</td>
+</tr>
+
+<tr>
+<td>Stack Overflow</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Airflow Summit Videos</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>The dev mailing list</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Awesome Apache Airflow repository</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here we see that Airflow documentation is the crucial source of information. What&rsquo;s interesting is that more
+than 60% of users are getting information from Github and Slack channels.</p>
+
+<p><img src="Where_are_you_based.png" alt="" /></p>
+
+<h2 id="airflow-uses-cases">Airflow uses cases</h2>
+
+<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we use vanilla Airflow</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Yes, we have small patches (no fork)</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Yes, we have separate fork</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No guide on best practises on developing DAGs</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>There is no easy option to launch Airflow</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Small number of tutorials on different aspects of using Airflow</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Documentation is not clear enough</td>
+<td>53</td>
+<td>26.11</td>
+</tr>
+
+<tr>
+<td>There is no easy option to deploy DAGs to an Airflow instance</td>
+<td>52</td>
+<td>25.62</td>
+</tr>
+
+<tr>
+<td>No problems</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Small number of blogs regarding Airflow</td>
+<td>30</td>
+<td>14.78</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Original Airflow Graphical User Interface</td>
+<td>199</td>
+<td>98.03</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>88</td>
+<td>43.35</td>
+</tr>
+
+<tr>
+<td>API</td>
+<td>48</td>
+<td>23.65</td>
+</tr>
+
+<tr>
+<td>Custom (own created) Airflow Graphical User Interface</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, by triggering another DAG</td>
+<td>87</td>
+<td>42.86</td>
+</tr>
+
+<tr>
+<td>No, I don&rsquo;t combine multiple DAGs</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Yes, through SubDAG</td>
+<td>40</td>
+<td>19.7</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using existing dedicated operators / hooks</td>
+<td>147</td>
+<td>72.41</td>
+</tr>
+
+<tr>
+<td>Using Bash / Python operator</td>
+<td>140</td>
+<td>68.97</td>
+</tr>
+
+<tr>
+<td>Using own custom operators / hooks</td>
+<td>138</td>
+<td>67.98</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Amazon Web Services</td>
+<td>121</td>
+<td>59.61</td>
+</tr>
+
+<tr>
+<td>Internal company systems</td>
+<td>113</td>
+<td>55.67</td>
+</tr>
+
+<tr>
+<td>Google Cloud Platform / Google APIs</td>
+<td>97</td>
+<td>47.78</td>
+</tr>
+
+<tr>
+<td>Hadoop / Spark / Flink / Other Apache software</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Microsoft Azure</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>I do not use external services in my Airflow DAGs</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+</tbody>
+</table>
+
+<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt="" /></p>
+
+<p><strong>Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Adding new operators/sensors and hooks</td>
+<td>119</td>
+<td>58.62</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t use Airflow plugins</td>
+<td>69</td>
+<td>33.99</td>
+</tr>
+
+<tr>
+<td>Adding AppBuilder views &amp; menu items</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>Adding new executors</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Adding OperatorExtraLinks</td>
+<td>13</td>
+<td>6.4</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td></td>
+<td></td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you use Airflow&rsquo;s data lineage feature? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, I will use such feature if fully supported in Airflow</td>
+<td>105</td>
+<td>51.72</td>
+</tr>
+
+<tr>
+<td>No, data lineage isn’t a concern for my usage.</td>
+<td>68</td>
+<td>33.5</td>
+</tr>
+
+<tr>
+<td>Yes, I use another data lineage product</td>
+<td>24</td>
+<td>11.82</td>
+</tr>
+
+<tr>
+<td>Yes, I use custom implementation</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+
+<tr>
+<td>Yes, I use Airflow&rsquo;s experimental data lineage feature</td>
+<td>1</td>
+<td>0.49</td>
+</tr>
+</tbody>
+</table>
+
+<p>When asked what lineage product users use, the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.</p>
+
+<h2 id="deployment">Deployment</h2>
+
+<p><strong>How many active DAGs do you have in your largest Airflow instance? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 20</td>
+<td>64</td>
+<td>32</td>
+</tr>
+
+<tr>
+<td>21-40</td>
+<td>33</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>41-60</td>
+<td>13</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>61-100</td>
+<td>32</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>201-300</td>
+<td>8</td>
+<td>4</td>
+</tr>
+
+<tr>
+<td>301-999</td>
+<td>12</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>1000+</td>
+<td>10</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What is the maximum number of tasks that you have used in one DAG? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 10</td>
+<td>42</td>
+<td>21</td>
+</tr>
+
+<tr>
+<td>11-20</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>21-30</td>
+<td>15</td>
+<td>7</td>
+</tr>
+
+<tr>
+<td>31-40</td>
+<td>11</td>
+<td>5</td>
+</tr>
+
+<tr>
+<td>41-50</td>
+<td>22</td>
+<td>11</td>
+</tr>
+
+<tr>
+<td>51-100</td>
+<td>39</td>
+<td>19</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>201-500</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>501+</td>
+<td>11</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which version of Airflow do you use currently? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1.10.14</td>
+<td>55</td>
+<td>27.09</td>
+</tr>
+
+<tr>
+<td>2.0.0+</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+
+<tr>
+<td>1.10.12</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>1.10.10</td>
+<td>26</td>
+<td>12.81</td>
+</tr>
+
+<tr>
+<td>1.10.11</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>1.10.5 or older</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>1.10.9</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>1.10.13</td>
+<td>7</td>
+<td>3.45</td>
+</tr>
+
+<tr>
+<td>1.10.6</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.7</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.8</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>This was probably one of the most important questions in the survey. While it&rsquo;s good to see
+that more than 60% of users use one of three latest Airflow versions, it&rsquo;s worrying that the rest
+are using versions that are old or have known security vulnerabilities.</p>
+
+<p>Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.</p>
+
+<p><strong>What meta-database do you use? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Postgres 12</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Postgres 9.6</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Postgres 11</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>MySQL 5.7</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>MySQL 8.0</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Postgres 10</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>Postgres 13</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p>This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.</p>
+
+<p>It&rsquo;s good to know that users rather avoid using SQLite in production deployments!</p>
+
+<p><strong>What executor type do you use? (single choice)</strong></p>
+
+<p><img src="What_executor_type_do_you_use.png" alt="" /></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020</th>
+<th>2019</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Celery</td>
+<td>100</td>
+<td>49.26%</td>
+<td>44.81%</td>
+</tr>
+
+<tr>
+<td>Kubernetes</td>
+<td>48</td>
+<td>23.65%</td>
+<td>16.88%</td>
+</tr>
+
+<tr>
+<td>Local</td>
+<td>40</td>
+<td>19.7%</td>
+<td>27.60%</td>
+</tr>
+
+<tr>
+<td>Sequential</td>
+<td>10</td>
+<td>4.93%</td>
+<td>7.14%</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>5</td>
+<td>2.46%</td>
+<td>3.57</td>
+</tr>
+</tbody>
+</table>
+
+<p>In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users&rsquo; deployments are growing, and they need more scalable solutions.</p>
+
+<p>Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.</p>
+
+<p><strong>What metrics do you use to monitor Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I do not use monitoring</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>External monitoring service</td>
+<td>60</td>
+<td>29.56</td>
+</tr>
+
+<tr>
+<td>Information from metadatabase</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Statsd</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+</tbody>
+</table>
+
+<p>The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.</p>
+
+<p><strong>How do you deploy Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>On virtual machines (for example using AWS EC2)</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
+<td>35</td>
+<td>17.24</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using custom deployments)</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+
+<tr>
+<td>On premises</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using another helm chart)</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using Apache Airflow&rsquo;s helm chart)</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
+10 percent more than in 2019. There&rsquo;s slightly increase in usage of Airflow via
+managed services (14.61% in 2019).</p>
+
+<p><strong>Do you use containerisation for deployment? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, using helm chart / kubernetes</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>No, I don’t use containerisation</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Yes, single docker image</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Yes, using docker compose</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+</tbody>
+</table>
+
+<p>Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
+42% of those users use docker-compose for deployments.</p>
+
+<p><strong>How do you distribute your DAGs? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Bake them into the docker image</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Shared files system</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+</tbody>
+</table>
+
+<p>The most popular way of distributing DAGs seems to be using a synchronizing process. About
+40% of users use this process together with Kubernetes deployments.</p>
+
+<h2 id="future-of-airflow">Future of Airflow</h2>
+
+<p><strong>In your opinion, what could be improved in Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Web UI</td>
+<td>100</td>
+<td>49.26</td>
+</tr>
+
+<tr>
+<td>Examples, how-to, onboarding documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Logging, monitoring and alerting</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Technical documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Scheduler performance</td>
+<td>83</td>
+<td>40.89</td>
+</tr>
+
+<tr>
+<td>DAG authoring</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Authentication and authorization</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>REST API</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>Reliability</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>External integration e.g. AWS, GCP, Apache products</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Security</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Everything work fine for me</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which features would most interest you? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>DAG versioning</td>
+<td>109</td>
+<td>53.69</td>
+</tr>
+
+<tr>
+<td>Builtin statistics</td>
+<td>71</td>
+<td>34.98</td>
+</tr>
+
+<tr>
+<td>Improved data lineage</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>Scheduling at the start of the interval</td>
+<td>63</td>
+<td>31.03</td>
+</tr>
+
+<tr>
+<td>Stateless workers</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>More option to configure schedules (time units, increments)</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Multi-tenant deployment</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>DAG fetcher (AIP-5)</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>Generic transfer operator</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>I have everything I need</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+
+<tr>
+<td>Nothing</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Will you consider migrating to Airflow 2.0? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, as soon as possible</td>
+<td>81</td>
+<td>39.9</td>
+</tr>
+
+<tr>
+<td>Yes, once it’s mature (for example after 2.1)</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>I am already using Airflow 2.0+</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t know yet</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>No, I do not plan to migrate</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What are the features of Airflow 2.0 you are most excited about? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>General performance improvements</td>
+<td>133</td>
+<td>65.52</td>
+</tr>
+
+<tr>
+<td>Refreshed WebUI</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>Scheduler HA</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>Official docker image</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>@task decorator</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Official helm chart</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Providers packages</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>Configurable XCom backends</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>CeleryKubernetesExecutor</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<h2 id="summary">Summary</h2>
+
+<p>From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
+This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and
+are willing to upgrade to new ones.</p>
+
+<p>Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow
+deployments. However, we hope that with the increase of adoption there will be an increase in people willing
+to share their experience and tools.</p>
+
+ </description>
+ </item>
+
+ <item>
<title>Blog: Airflow Survey 2019</title>
<link>/blog/airflow-survey/</link>
<pubDate>Wed, 11 Dec 2019 00:00:00 +0000</pubDate>
diff --git a/blog/tags/users/index.html b/blog/tags/users/index.html
index 2d45b69..941f164 100644
--- a/blog/tags/users/index.html
+++ b/blog/tags/users/index.html
@@ -37,7 +37,7 @@
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="og:updated_time" content="2019-12-11T00:00:00+00:00" /><meta property="og:site_name" content="Apache Airflow" />
+<meta property="og:updated_time" content="2021-03-09T00:00:00+00:00" /><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="users">
<meta itemprop="description" content="Platform created by the community to programmatically author, schedule and monitor workflows.">
@@ -505,6 +505,45 @@ if (!doNotTrack) {
</div>
+ <span class="bodytext__medium--brownish-grey box-event__blogpost--date">Tue, Mar 9, 2021</span>
+ </div>
+ <p class="box-event__blogpost--header">Airflow Survey 2020</p>
+ <p class="box-event__blogpost--author">Tomek Urbaszek</p>
+ <p class="box-event__blogpost--description">We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-survey-2020/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/community/">Community</a>
+
+
+
+ <a class="tag" href="/blog/tags/survey/">Survey</a>
+
+
+
+ <a class="tag" href="/blog/tags/users/">Users</a>
+
+
+ </div>
<span class="bodytext__medium--brownish-grey box-event__blogpost--date">Wed, Dec 11, 2019</span>
</div>
<p class="box-event__blogpost--header">Airflow Survey 2019</p>
diff --git a/blog/tags/users/index.xml b/blog/tags/users/index.xml
index df21fad..f8b82ab 100644
--- a/blog/tags/users/index.xml
+++ b/blog/tags/users/index.xml
@@ -4,7 +4,7 @@
<link>/blog/tags/users/</link>
<description>Recent content in users on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
- <lastBuildDate>Wed, 11 Dec 2019 00:00:00 +0000</lastBuildDate>
+ <lastBuildDate>Tue, 09 Mar 2021 00:00:00 +0000</lastBuildDate>
<atom:link href="/blog/tags/users/index.xml" rel="self" type="application/rss+xml" />
@@ -15,6 +15,1605 @@
<item>
+ <title>Blog: Airflow Survey 2020</title>
+ <link>/blog/airflow-survey-2020/</link>
+ <pubDate>Tue, 09 Mar 2021 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-survey-2020/</guid>
+ <description>
+
+
+
+
+<h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
+
+<p>World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
+crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
+an amount of active contributors. So listening and understanding our community is of high importance.</p>
+
+<p>It&rsquo;s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!</p>
+
+<h2 id="overview-of-the-user">Overview of the user</h2>
+
+<p><img src="What_best_describes_your_current_occupation.png" alt="" /></p>
+
+<p><strong>What best describes your current occupation? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Data Engineer</td>
+<td>115</td>
+<td>56.65</td>
+</tr>
+
+<tr>
+<td>Developer</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>DevOps</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Solutions Architect</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>Data Scientist</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>Data Analyst</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>Support Engineer</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> with
+visible slight increase in ML use cases.</p>
+
+<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
+
+<p><strong>How many people work in your company? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>200+</td>
+<td>107</td>
+<td>52.71</td>
+</tr>
+
+<tr>
+<td>51-200</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>11-50</td>
+<td>37</td>
+<td>18.23</td>
+</tr>
+
+<tr>
+<td>1-10</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1-5</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>6-20</td>
+<td>75</td>
+<td>36.95</td>
+</tr>
+
+<tr>
+<td>21-50</td>
+<td>23</td>
+<td>11.33</td>
+</tr>
+
+<tr>
+<td>50+</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+</tbody>
+</table>
+
+<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
+
+<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we are happy with Airflow</td>
+<td>174</td>
+<td>85.71</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
+(8.37% both).</p>
+
+<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api" target="_blank">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.</p>
+
+<p>The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.</p>
+
+<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I wish we could</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>No</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+</tbody>
+</table>
+
+<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there&rsquo;s something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.</p>
+
+<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020 %</th>
+<th>2019 %</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Very Likely</td>
+<td>125</td>
+<td>61.58</td>
+<td>45.45%</td>
+</tr>
+
+<tr>
+<td>Likely</td>
+<td>62</td>
+<td>30.54</td>
+<td>40.26%</td>
+</tr>
+
+<tr>
+<td>Neutral</td>
+<td>11</td>
+<td>5.42</td>
+<td>10.71%</td>
+</tr>
+
+<tr>
+<td>Unlikely</td>
+<td>3</td>
+<td>1.48</td>
+<td>2.60%</td>
+</tr>
+
+<tr>
+<td>Very unlikely</td>
+<td>2</td>
+<td>0.99</td>
+<td>0.97%</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
+
+<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Documentation</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Airflow website</td>
+<td>139</td>
+<td>68.47</td>
+</tr>
+
+<tr>
+<td>Slack</td>
+<td>128</td>
+<td>63.05</td>
+</tr>
+
+<tr>
+<td>Github</td>
+<td>127</td>
+<td>62.56</td>
+</tr>
+
+<tr>
+<td>Stack Overflow</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Airflow Summit Videos</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>The dev mailing list</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Awesome Apache Airflow repository</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here we see that Airflow documentation is the crucial source of information. What&rsquo;s interesting is that more
+than 60% of users are getting information from Github and Slack channels.</p>
+
+<p><img src="Where_are_you_based.png" alt="" /></p>
+
+<h2 id="airflow-uses-cases">Airflow uses cases</h2>
+
+<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we use vanilla Airflow</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Yes, we have small patches (no fork)</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Yes, we have separate fork</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No guide on best practises on developing DAGs</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>There is no easy option to launch Airflow</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Small number of tutorials on different aspects of using Airflow</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Documentation is not clear enough</td>
+<td>53</td>
+<td>26.11</td>
+</tr>
+
+<tr>
+<td>There is no easy option to deploy DAGs to an Airflow instance</td>
+<td>52</td>
+<td>25.62</td>
+</tr>
+
+<tr>
+<td>No problems</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Small number of blogs regarding Airflow</td>
+<td>30</td>
+<td>14.78</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Original Airflow Graphical User Interface</td>
+<td>199</td>
+<td>98.03</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>88</td>
+<td>43.35</td>
+</tr>
+
+<tr>
+<td>API</td>
+<td>48</td>
+<td>23.65</td>
+</tr>
+
+<tr>
+<td>Custom (own created) Airflow Graphical User Interface</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, by triggering another DAG</td>
+<td>87</td>
+<td>42.86</td>
+</tr>
+
+<tr>
+<td>No, I don&rsquo;t combine multiple DAGs</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Yes, through SubDAG</td>
+<td>40</td>
+<td>19.7</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using existing dedicated operators / hooks</td>
+<td>147</td>
+<td>72.41</td>
+</tr>
+
+<tr>
+<td>Using Bash / Python operator</td>
+<td>140</td>
+<td>68.97</td>
+</tr>
+
+<tr>
+<td>Using own custom operators / hooks</td>
+<td>138</td>
+<td>67.98</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Amazon Web Services</td>
+<td>121</td>
+<td>59.61</td>
+</tr>
+
+<tr>
+<td>Internal company systems</td>
+<td>113</td>
+<td>55.67</td>
+</tr>
+
+<tr>
+<td>Google Cloud Platform / Google APIs</td>
+<td>97</td>
+<td>47.78</td>
+</tr>
+
+<tr>
+<td>Hadoop / Spark / Flink / Other Apache software</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Microsoft Azure</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>I do not use external services in my Airflow DAGs</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+</tbody>
+</table>
+
+<p><img src="What_external_services_do_you_use_in_your_Airflow_DAGs.png" alt="" /></p>
+
+<p><strong>Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Adding new operators/sensors and hooks</td>
+<td>119</td>
+<td>58.62</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t use Airflow plugins</td>
+<td>69</td>
+<td>33.99</td>
+</tr>
+
+<tr>
+<td>Adding AppBuilder views &amp; menu items</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>Adding new executors</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Adding OperatorExtraLinks</td>
+<td>13</td>
+<td>6.4</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td></td>
+<td></td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you use Airflow&rsquo;s data lineage feature? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, I will use such feature if fully supported in Airflow</td>
+<td>105</td>
+<td>51.72</td>
+</tr>
+
+<tr>
+<td>No, data lineage isn’t a concern for my usage.</td>
+<td>68</td>
+<td>33.5</td>
+</tr>
+
+<tr>
+<td>Yes, I use another data lineage product</td>
+<td>24</td>
+<td>11.82</td>
+</tr>
+
+<tr>
+<td>Yes, I use custom implementation</td>
+<td>5</td>
+<td>2.46</td>
+</tr>
+
+<tr>
+<td>Yes, I use Airflow&rsquo;s experimental data lineage feature</td>
+<td>1</td>
+<td>0.49</td>
+</tr>
+</tbody>
+</table>
+
+<p>When asked what lineage product users use, the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.</p>
+
+<h2 id="deployment">Deployment</h2>
+
+<p><strong>How many active DAGs do you have in your largest Airflow instance? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 20</td>
+<td>64</td>
+<td>32</td>
+</tr>
+
+<tr>
+<td>21-40</td>
+<td>33</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>41-60</td>
+<td>13</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>61-100</td>
+<td>32</td>
+<td>16</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>201-300</td>
+<td>8</td>
+<td>4</td>
+</tr>
+
+<tr>
+<td>301-999</td>
+<td>12</td>
+<td>6</td>
+</tr>
+
+<tr>
+<td>1000+</td>
+<td>10</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What is the maximum number of tasks that you have used in one DAG? (open question)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Number of DAGs</th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>&lt; 10</td>
+<td>42</td>
+<td>21</td>
+</tr>
+
+<tr>
+<td>11-20</td>
+<td>31</td>
+<td>15</td>
+</tr>
+
+<tr>
+<td>21-30</td>
+<td>15</td>
+<td>7</td>
+</tr>
+
+<tr>
+<td>31-40</td>
+<td>11</td>
+<td>5</td>
+</tr>
+
+<tr>
+<td>41-50</td>
+<td>22</td>
+<td>11</td>
+</tr>
+
+<tr>
+<td>51-100</td>
+<td>39</td>
+<td>19</td>
+</tr>
+
+<tr>
+<td>101-200</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>201-500</td>
+<td>16</td>
+<td>8</td>
+</tr>
+
+<tr>
+<td>501+</td>
+<td>11</td>
+<td>5</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which version of Airflow do you use currently? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1.10.14</td>
+<td>55</td>
+<td>27.09</td>
+</tr>
+
+<tr>
+<td>2.0.0+</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+
+<tr>
+<td>1.10.12</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>1.10.10</td>
+<td>26</td>
+<td>12.81</td>
+</tr>
+
+<tr>
+<td>1.10.11</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>1.10.5 or older</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>1.10.9</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>1.10.13</td>
+<td>7</td>
+<td>3.45</td>
+</tr>
+
+<tr>
+<td>1.10.6</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.7</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>1.10.8</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>This was probably one of the most important questions in the survey. While it&rsquo;s good to see
+that more than 60% of users use one of three latest Airflow versions, it&rsquo;s worrying that the rest
+are using versions that are old or have known security vulnerabilities.</p>
+
+<p>Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.</p>
+
+<p><strong>What meta-database do you use? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Postgres 12</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Postgres 9.6</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Postgres 11</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>MySQL 5.7</td>
+<td>27</td>
+<td>13.3</td>
+</tr>
+
+<tr>
+<td>MySQL 8.0</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Postgres 10</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>19</td>
+<td>9.36</td>
+</tr>
+
+<tr>
+<td>Postgres 13</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p>This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.</p>
+
+<p>It&rsquo;s good to know that users rather avoid using SQLite in production deployments!</p>
+
+<p><strong>What executor type do you use? (single choice)</strong></p>
+
+<p><img src="What_executor_type_do_you_use.png" alt="" /></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020</th>
+<th>2019</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Celery</td>
+<td>100</td>
+<td>49.26%</td>
+<td>44.81%</td>
+</tr>
+
+<tr>
+<td>Kubernetes</td>
+<td>48</td>
+<td>23.65%</td>
+<td>16.88%</td>
+</tr>
+
+<tr>
+<td>Local</td>
+<td>40</td>
+<td>19.7%</td>
+<td>27.60%</td>
+</tr>
+
+<tr>
+<td>Sequential</td>
+<td>10</td>
+<td>4.93%</td>
+<td>7.14%</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>5</td>
+<td>2.46%</td>
+<td>3.57</td>
+</tr>
+</tbody>
+</table>
+
+<p>In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users&rsquo; deployments are growing, and they need more scalable solutions.</p>
+
+<p>Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.</p>
+
+<p><strong>What metrics do you use to monitor Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I do not use monitoring</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>External monitoring service</td>
+<td>60</td>
+<td>29.56</td>
+</tr>
+
+<tr>
+<td>Information from metadatabase</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Statsd</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+</tbody>
+</table>
+
+<p>The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.</p>
+
+<p><strong>How do you deploy Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>On virtual machines (for example using AWS EC2)</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Using a managed service like Astronomer, Google Composer or AWS MWAA</td>
+<td>35</td>
+<td>17.24</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using custom deployments)</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+
+<tr>
+<td>On premises</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using another helm chart)</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>On Kubernetes (using Apache Airflow&rsquo;s helm chart)</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
+10 percent more than in 2019. There&rsquo;s slightly increase in usage of Airflow via
+managed services (14.61% in 2019).</p>
+
+<p><strong>Do you use containerisation for deployment? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, using helm chart / kubernetes</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>No, I don’t use containerisation</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Yes, single docker image</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>Yes, using docker compose</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+</tbody>
+</table>
+
+<p>Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
+42% of those users use docker-compose for deployments.</p>
+
+<p><strong>How do you distribute your DAGs? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using a synchronizing process (Git sync, GCS fuse, etc)</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Bake them into the docker image</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Shared files system</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+</tbody>
+</table>
+
+<p>The most popular way of distributing DAGs seems to be using a synchronizing process. About
+40% of users use this process together with Kubernetes deployments.</p>
+
+<h2 id="future-of-airflow">Future of Airflow</h2>
+
+<p><strong>In your opinion, what could be improved in Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Web UI</td>
+<td>100</td>
+<td>49.26</td>
+</tr>
+
+<tr>
+<td>Examples, how-to, onboarding documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Logging, monitoring and alerting</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Technical documentation</td>
+<td>90</td>
+<td>44.33</td>
+</tr>
+
+<tr>
+<td>Scheduler performance</td>
+<td>83</td>
+<td>40.89</td>
+</tr>
+
+<tr>
+<td>DAG authoring</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Authentication and authorization</td>
+<td>58</td>
+<td>28.57</td>
+</tr>
+
+<tr>
+<td>REST API</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>Reliability</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>External integration e.g. AWS, GCP, Apache products</td>
+<td>36</td>
+<td>17.73</td>
+</tr>
+
+<tr>
+<td>Security</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>20</td>
+<td>9.85</td>
+</tr>
+
+<tr>
+<td>Everything work fine for me</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>I don’t know</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which features would most interest you? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>DAG versioning</td>
+<td>109</td>
+<td>53.69</td>
+</tr>
+
+<tr>
+<td>Builtin statistics</td>
+<td>71</td>
+<td>34.98</td>
+</tr>
+
+<tr>
+<td>Improved data lineage</td>
+<td>65</td>
+<td>32.02</td>
+</tr>
+
+<tr>
+<td>Scheduling at the start of the interval</td>
+<td>63</td>
+<td>31.03</td>
+</tr>
+
+<tr>
+<td>Stateless workers</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>More option to configure schedules (time units, increments)</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Multi-tenant deployment</td>
+<td>49</td>
+<td>24.14</td>
+</tr>
+
+<tr>
+<td>DAG fetcher (AIP-5)</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>Generic transfer operator</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>I have everything I need</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+
+<tr>
+<td>Nothing</td>
+<td>11</td>
+<td>5.42</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Will you consider migrating to Airflow 2.0? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, as soon as possible</td>
+<td>81</td>
+<td>39.9</td>
+</tr>
+
+<tr>
+<td>Yes, once it’s mature (for example after 2.1)</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>I am already using Airflow 2.0+</td>
+<td>39</td>
+<td>19.21</td>
+</tr>
+
+<tr>
+<td>I don&rsquo;t know yet</td>
+<td>8</td>
+<td>3.94</td>
+</tr>
+
+<tr>
+<td>No, I do not plan to migrate</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What are the features of Airflow 2.0 you are most excited about? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>General performance improvements</td>
+<td>133</td>
+<td>65.52</td>
+</tr>
+
+<tr>
+<td>Refreshed WebUI</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>Scheduler HA</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>Official docker image</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>@task decorator</td>
+<td>56</td>
+<td>27.59</td>
+</tr>
+
+<tr>
+<td>Official helm chart</td>
+<td>51</td>
+<td>25.12</td>
+</tr>
+
+<tr>
+<td>Providers packages</td>
+<td>41</td>
+<td>20.2</td>
+</tr>
+
+<tr>
+<td>Configurable XCom backends</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>CeleryKubernetesExecutor</td>
+<td>31</td>
+<td>15.27</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<h2 id="summary">Summary</h2>
+
+<p>From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
+This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and
+are willing to upgrade to new ones.</p>
+
+<p>Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow
+deployments. However, we hope that with the increase of adoption there will be an increase in people willing
+to share their experience and tools.</p>
+
+ </description>
+ </item>
+
+ <item>
<title>Blog: Airflow Survey 2019</title>
<link>/blog/airflow-survey/</link>
<pubDate>Wed, 11 Dec 2019 00:00:00 +0000</pubDate>
diff --git a/index.html b/index.html
index d77e6c0..b85e54e 100644
--- a/index.html
+++ b/index.html
@@ -1226,12 +1226,12 @@ if (!doNotTrack) {
<div id="integrations-container" class="list-items">
- <a class="list-item" href="/docs/apache-airflow-providers-google/stable/operators/cloud/datastore.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-docker/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Google Datastore</span>
+ <span class="box-event__integration--name">Docker Swarm</span>
</div>
</div>
@@ -1239,12 +1239,12 @@ if (!doNotTrack) {
- <a class="list-item" href="/docs/apache-airflow-providers-apache-pinot/stable/index.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-amazon/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Apache Pinot</span>
+ <span class="box-event__integration--name">Amazon DynamoDB</span>
</div>
</div>
@@ -1252,12 +1252,12 @@ if (!doNotTrack) {
- <a class="list-item" href="/docs/apache-airflow-providers-google/stable/operators/cloud/mlengine.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-amazon/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Google Machine Learning Engine</span>
+ <span class="box-event__integration--name">Amazon Redshift</span>
</div>
</div>
@@ -1265,12 +1265,12 @@ if (!doNotTrack) {
- <a class="list-item" href="/docs/apache-airflow-providers-amazon/stable/operators/glacier.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-google/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Amazon Glacier</span>
+ <span class="box-event__integration--name">Google Cloud Key Management Service (KMS)</span>
</div>
</div>
@@ -1278,12 +1278,12 @@ if (!doNotTrack) {
- <a class="list-item" href="/docs/apache-airflow-providers-google/stable/operators/marketing_platform/search_ads.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-amazon/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Google Search Ads 360</span>
+ <span class="box-event__integration--name">Amazon CloudWatch Logs</span>
</div>
</div>
@@ -1291,12 +1291,12 @@ if (!doNotTrack) {
- <a class="list-item" href="/docs/apache-airflow-providers-yandex/stable/operators.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-amazon/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Yandex.Cloud Dataproc</span>
+ <span class="box-event__integration--name">AWS Batch</span>
</div>
</div>
@@ -1304,12 +1304,12 @@ if (!doNotTrack) {
- <a class="list-item" href="/docs/apache-airflow-providers-microsoft-winrm/stable/index.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-amazon/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Windows Remote Management (WinRM)</span>
+ <span class="box-event__integration--name">Amazon Athena</span>
</div>
</div>
@@ -1317,12 +1317,12 @@ if (!doNotTrack) {
- <a class="list-item" href="/docs/apache-airflow-providers-google/stable/operators/cloud/gcs.html">
+ <a class="list-item" href="/docs/apache-airflow-providers-cloudant/stable/index.html">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Google Cloud Storage (GCS)</span>
+ <span class="box-event__integration--name">IBM Cloudant</span>
</div>
</div>
diff --git a/index.xml b/index.xml
index d1047e0..c4ed8c5 100644
--- a/index.xml
+++ b/index.xml
@@ -13,6 +13,1605 @@
<item>
+ <title>Blog: Airflow Survey 2020</title>
+ <link>/blog/airflow-survey-2020/</link>
+ <pubDate>Tue, 09 Mar 2021 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-survey-2020/</guid>
+ <description>
+
+
+
+
+<h1 id="apache-airflow-survey-2020">Apache Airflow Survey 2020</h1>
+
+<p>World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
+crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
+an amount of active contributors. So listening and understanding our community is of high importance.</p>
+
+<p>It&rsquo;s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!</p>
+
+<h2 id="overview-of-the-user">Overview of the user</h2>
+
+<p><img src="What_best_describes_your_current_occupation.png" alt="" /></p>
+
+<p><strong>What best describes your current occupation? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Data Engineer</td>
+<td>115</td>
+<td>56.65</td>
+</tr>
+
+<tr>
+<td>Developer</td>
+<td>28</td>
+<td>13.79</td>
+</tr>
+
+<tr>
+<td>DevOps</td>
+<td>17</td>
+<td>8.37</td>
+</tr>
+
+<tr>
+<td>Solutions Architect</td>
+<td>14</td>
+<td>6.9</td>
+</tr>
+
+<tr>
+<td>Data Scientist</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>10</td>
+<td>4.93</td>
+</tr>
+
+<tr>
+<td>Data Analyst</td>
+<td>4</td>
+<td>1.97</td>
+</tr>
+
+<tr>
+<td>Support Engineer</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p>Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to <a href="https://airflow.apache.org/blog/airflow-survey/" target="_blank">those from 2019</a> with
+visible slight increase in ML use cases.</p>
+
+<p>Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.</p>
+
+<p><strong>How many people work in your company? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>200+</td>
+<td>107</td>
+<td>52.71</td>
+</tr>
+
+<tr>
+<td>51-200</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>11-50</td>
+<td>37</td>
+<td>18.23</td>
+</tr>
+
+<tr>
+<td>1-10</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How many people in your company use Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>1-5</td>
+<td>84</td>
+<td>41.38</td>
+</tr>
+
+<tr>
+<td>6-20</td>
+<td>75</td>
+<td>36.95</td>
+</tr>
+
+<tr>
+<td>21-50</td>
+<td>23</td>
+<td>11.33</td>
+</tr>
+
+<tr>
+<td>50+</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+</tbody>
+</table>
+
+<p>Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.</p>
+
+<p><strong>Are you considering moving to other workflow engines? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we are happy with Airflow</td>
+<td>174</td>
+<td>85.71</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>29</td>
+<td>14.29</td>
+</tr>
+</tbody>
+</table>
+
+<p>Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of <strong>easier workflow writing experience</strong> (12.32%), <strong>better UI/UX</strong> and <strong>faster scheduler</strong>
+(8.37% both).</p>
+
+<p>While the first point may be addressed by <a href="http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api" target="_blank">TaskFlow API</a> in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.</p>
+
+<p>The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.</p>
+
+<p><strong>Are you or your team actively participating in Airflow development - contributing? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>I wish we could</td>
+<td>99</td>
+<td>48.77</td>
+</tr>
+
+<tr>
+<td>No</td>
+<td>59</td>
+<td>29.06</td>
+</tr>
+
+<tr>
+<td>Yes</td>
+<td>45</td>
+<td>22.17</td>
+</tr>
+</tbody>
+</table>
+
+<p>This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there&rsquo;s something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.</p>
+
+<p><strong>How likely are you to recommend Apache Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>2020 %</th>
+<th>2019 %</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Very Likely</td>
+<td>125</td>
+<td>61.58</td>
+<td>45.45%</td>
+</tr>
+
+<tr>
+<td>Likely</td>
+<td>62</td>
+<td>30.54</td>
+<td>40.26%</td>
+</tr>
+
+<tr>
+<td>Neutral</td>
+<td>11</td>
+<td>5.42</td>
+<td>10.71%</td>
+</tr>
+
+<tr>
+<td>Unlikely</td>
+<td>3</td>
+<td>1.48</td>
+<td>2.60%</td>
+</tr>
+
+<tr>
+<td>Very unlikely</td>
+<td>2</td>
+<td>0.99</td>
+<td>0.97%</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.</p>
+
+<p><strong>What is your source of information about Airflow? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Documentation</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Airflow website</td>
+<td>139</td>
+<td>68.47</td>
+</tr>
+
+<tr>
+<td>Slack</td>
+<td>128</td>
+<td>63.05</td>
+</tr>
+
+<tr>
+<td>Github</td>
+<td>127</td>
+<td>62.56</td>
+</tr>
+
+<tr>
+<td>Stack Overflow</td>
+<td>72</td>
+<td>35.47</td>
+</tr>
+
+<tr>
+<td>Airflow Summit Videos</td>
+<td>44</td>
+<td>21.67</td>
+</tr>
+
+<tr>
+<td>The dev mailing list</td>
+<td>33</td>
+<td>16.26</td>
+</tr>
+
+<tr>
+<td>Awesome Apache Airflow repository</td>
+<td>21</td>
+<td>10.34</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p>Here we see that Airflow documentation is the crucial source of information. What&rsquo;s interesting is that more
+than 60% of users are getting information from Github and Slack channels.</p>
+
+<p><img src="Where_are_you_based.png" alt="" /></p>
+
+<h2 id="airflow-uses-cases">Airflow uses cases</h2>
+
+<p><strong>Do you have any customisation of Airflow? (single choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No, we use vanilla Airflow</td>
+<td>154</td>
+<td>75.86</td>
+</tr>
+
+<tr>
+<td>Yes, we have small patches (no fork)</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Yes, we have separate fork</td>
+<td>15</td>
+<td>7.39</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>When onboarding new members to airflow, what is the biggest problem? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>No guide on best practises on developing DAGs</td>
+<td>102</td>
+<td>50.25</td>
+</tr>
+
+<tr>
+<td>There is no easy option to launch Airflow</td>
+<td>64</td>
+<td>31.53</td>
+</tr>
+
+<tr>
+<td>Small number of tutorials on different aspects of using Airflow</td>
+<td>57</td>
+<td>28.08</td>
+</tr>
+
+<tr>
+<td>Documentation is not clear enough</td>
+<td>53</td>
+<td>26.11</td>
+</tr>
+
+<tr>
+<td>There is no easy option to deploy DAGs to an Airflow instance</td>
+<td>52</td>
+<td>25.62</td>
+</tr>
+
+<tr>
+<td>No problems</td>
+<td>34</td>
+<td>16.75</td>
+</tr>
+
+<tr>
+<td>Small number of blogs regarding Airflow</td>
+<td>30</td>
+<td>14.78</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Which interface(s) of Airflow do you use as part of your current role? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Original Airflow Graphical User Interface</td>
+<td>199</td>
+<td>98.03</td>
+</tr>
+
+<tr>
+<td>CLI</td>
+<td>88</td>
+<td>43.35</td>
+</tr>
+
+<tr>
+<td>API</td>
+<td>48</td>
+<td>23.65</td>
+</tr>
+
+<tr>
+<td>Custom (own created) Airflow Graphical User Interface</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>3</td>
+<td>1.48</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>Do you combine multiple DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Yes, by triggering another DAG</td>
+<td>87</td>
+<td>42.86</td>
+</tr>
+
+<tr>
+<td>No, I don&rsquo;t combine multiple DAGs</td>
+<td>79</td>
+<td>38.92</td>
+</tr>
+
+<tr>
+<td>Yes, through SubDAG</td>
+<td>40</td>
+<td>19.7</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>18</td>
+<td>8.87</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>How do you integrate with external services? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Using existing dedicated operators / hooks</td>
+<td>147</td>
+<td>72.41</td>
+</tr>
+
+<tr>
+<td>Using Bash / Python operator</td>
+<td>140</td>
+<td>68.97</td>
+</tr>
+
+<tr>
+<td>Using own custom operators / hooks</td>
+<td>138</td>
+<td>67.98</td>
+</tr>
+
+<tr>
+<td>Other</td>
+<td>12</td>
+<td>5.91</td>
+</tr>
+</tbody>
+</table>
+
+<p><strong>What external services do you use in your Airflow DAGs? (multiple choice)</strong></p>
+
+<table>
+<thead>
+<tr>
+<th></th>
+<th>No.</th>
+<th>%</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td>Amazon Web Services</td>
+<td>121</td>
... 1454 lines suppressed ...