You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/20 21:28:42 UTC
[GitHub] [airflow-site] turbaszek opened a new pull request #385: Add Apache Airflow Survey 2020 blog post
turbaszek opened a new pull request #385:
URL: https://github.com/apache/airflow-site/pull/385
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579718562
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many of drawbacks was addressed in 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance? **
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5
+
+**What is the maximum number of tasks that you have used in one DAG? **
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow version it's worrying that the rest
+is using versions that are old or have known security vulnerabilities.
+
+Additionally, more than 20% users is already using 2.0.0+ versions which is reasonably good information.
+
+**What meta-database do you use?**
+
+| | No. | % |
+|--------------|-------|-------|
+| Postgres 12 | 36 | 17.73 |
+| Postgres 9.6 | 33 | 16.26 |
+| Postgres 11 | 31 | 15.27 |
+| MySQL 5.7 | 27 | 13.3 |
+| MySQL 8.0 | 20 | 9.85 |
+| Postgres 10 | 20 | 9.85 |
+| Other | 19 | 9.36 |
+| Postgres 13 | 18 | 8.87 |
+
+This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.
+
+It's good to know that users rather avoid using SQLite in production deployments!
+
+**What executor type do you use?**
+
+
+
+| | No. | 2020% | 2019% |
+|------------|-------|-------|-------|
+| Celery | 100 | 49.26 | 44.81%|
+| Kubernetes | 48 | 23.65 | 16.88%|
+| Local | 40 | 19.7 | 27.60%|
+| Sequential | 10 | 4.93 | 7.14% |
+| Other | 5 | 2.46 | 3.57 |
+
+In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users' deployment are growing, and they need more scalable solutions.
+
+Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.
+
+
+**What metrics do you use to monitor Airflow? **
Review comment:
```suggestion
**What metrics do you use to monitor Airflow?**
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] leahecole commented on pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
leahecole commented on pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#issuecomment-786946483
Sorry I'm late to the reviewing party! Just a couple of nits. Great job @turbaszek !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek commented on pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#issuecomment-782919185
Thanks @kaxil and @XD-DENG for suggestions!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] leahecole commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
leahecole commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r583975483
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
Review comment:
Do we also have the counts for 2019? It would be helpful to see the count in comparison to 45% as well as the 125/61.58% comparisson.
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
Review comment:
```suggestion
While the first point may be addressed by [TaskFlow API](http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api) in Airflow 2.0 the other two are definitely addressed
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
Review comment:
These percentage don't add up to 100 - were users allowed to choose more than one option? Adding more detail about the percentage may help folks to make sense of the numbers.
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product users use, the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance?**
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5 |
+
+**What is the maximum number of tasks that you have used in one DAG?**
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow versions, it's worrying that the rest
+are using versions that are old or have known security vulnerabilities.
+
+Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.
+
+**What meta-database do you use?**
+
+| | No. | % |
+|--------------|-------|-------|
+| Postgres 12 | 36 | 17.73 |
+| Postgres 9.6 | 33 | 16.26 |
+| Postgres 11 | 31 | 15.27 |
+| MySQL 5.7 | 27 | 13.3 |
+| MySQL 8.0 | 20 | 9.85 |
+| Postgres 10 | 20 | 9.85 |
+| Other | 19 | 9.36 |
+| Postgres 13 | 18 | 8.87 |
+
+This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.
+
+It's good to know that users rather avoid using SQLite in production deployments!
+
+**What executor type do you use?**
+
+
+
+| | No. | 2020% | 2019% |
+|------------|-------|-------|-------|
+| Celery | 100 | 49.26 | 44.81%|
+| Kubernetes | 48 | 23.65 | 16.88%|
+| Local | 40 | 19.7 | 27.60%|
+| Sequential | 10 | 4.93 | 7.14% |
+| Other | 5 | 2.46 | 3.57 |
+
+In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users' deployments are growing, and they need more scalable solutions.
+
+Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.
+
+
+**What metrics do you use to monitor Airflow?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| I do not use monitoring | 65 | 32.02 |
+| External monitoring service | 60 | 29.56 |
+| Information from metadatabase | 51 | 25.12 |
+| Statsd | 49 | 24.14 |
+| Other | 31 | 15.27 |
+
+The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.
+
+**How do you deploy Airflow?**
+
+| | No. | % |
+|----------------------------------------------------------------------|-------|-------|
+| On virtual machines (for example using AWS EC2) | 64 | 31.53 |
+| Using a managed service like Astronomer, Google Composer or AWS MWAA | 35 | 17.24 |
+| On Kubernetes (using custom deployments) | 29 | 14.29 |
+| On premises | 28 | 13.79 |
+| On Kubernetes (using another helm chart) | 20 | 9.85 |
+| On Kubernetes (using Apache Airflow's helm chart) | 17 | 8.37 |
+| Other | 12 | 5.91 |
+
+Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
+10 percent more than in 2019. There's slightly increase in usage of Airflow via
+managed services (14.61% in 2019).
+
+**Do you use containerisation for deployment?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, using helm chart / kubernetes | 58 | 28.57 |
+| No, I don’t use containerisation | 57 | 28.08 |
+| Yes, single docker image | 49 | 24.14 |
+| Yes, using docker compose | 39 | 19.21 |
+
+Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
+42% of those users use docker-compose for deployments.
+
+**How do you distribute your DAGs?**
+
+| | No. | % |
+|---------------------------------------------------------|-------|-------|
+| Using a synchronizing process (Git sync, GCS fuse, etc) | 79 | 38.92 |
+| Bake them into the docker image | 56 | 27.59 |
+| Shared files system | 34 | 16.75 |
+| Other | 20 | 9.85 |
+| I don’t know | 14 | 6.9 |
+
+The most popular way of distributing DAGs seems to be using a synchronizing process. About
+40% of users use this process together with Kubernetes deployments.
+
+## Future of Airflow
Review comment:
Same comment as above about percentages for the categories here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r584085221
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
Review comment:
Imho the count will be misleading as there was more responses last year, so using % gives us a common base line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r583968184
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
Review comment:
Yeah this needs updating
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579718255
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many of drawbacks was addressed in 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance? **
Review comment:
```suggestion
**How many active DAGs do you have in your largest Airflow instance?**
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many of drawbacks was addressed in 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance? **
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5
+
+**What is the maximum number of tasks that you have used in one DAG? **
Review comment:
```suggestion
**What is the maximum number of tasks that you have used in one DAG?**
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579718411
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many of drawbacks was addressed in 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance? **
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5
+
+**What is the maximum number of tasks that you have used in one DAG? **
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow version it's worrying that the rest
+is using versions that are old or have known security vulnerabilities.
+
+Additionally, more than 20% users is already using 2.0.0+ versions which is reasonably good information.
Review comment:
```suggestion
This was probably one of the most important questions in the survey. While it's good to see
that more than 60% of users use one of three latest Airflow version it's worrying that the rest
are using versions that are old or have known security vulnerabilities.
Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r584085221
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
Review comment:
Imho the count will be misleading as there was more responses last year, so using % gives us a objective metric
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] XD-DENG commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579785905
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
Review comment:
```suggestion
Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
```
nit. To be consistent with the header above.
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
Review comment:
```suggestion
our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
```
Adding link to 2019 survey so easier for readers to cross-compare.
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance?**
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5 |
+
+**What is the maximum number of tasks that you have used in one DAG?**
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow version it's worrying that the rest
+are using versions that are old or have known security vulnerabilities.
+
+Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.
+
+**What meta-database do you use?**
+
+| | No. | % |
+|--------------|-------|-------|
+| Postgres 12 | 36 | 17.73 |
+| Postgres 9.6 | 33 | 16.26 |
+| Postgres 11 | 31 | 15.27 |
+| MySQL 5.7 | 27 | 13.3 |
+| MySQL 8.0 | 20 | 9.85 |
+| Postgres 10 | 20 | 9.85 |
+| Other | 19 | 9.36 |
+| Postgres 13 | 18 | 8.87 |
+
+This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.
+
+It's good to know that users rather avoid using SQLite in production deployments!
+
+**What executor type do you use?**
+
+
+
+| | No. | 2020% | 2019% |
+|------------|-------|-------|-------|
+| Celery | 100 | 49.26 | 44.81%|
+| Kubernetes | 48 | 23.65 | 16.88%|
+| Local | 40 | 19.7 | 27.60%|
+| Sequential | 10 | 4.93 | 7.14% |
+| Other | 5 | 2.46 | 3.57 |
+
+In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users' deployment are growing, and they need more scalable solutions.
+
+Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.
+
+
+**What metrics do you use to monitor Airflow?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| I do not use monitoring | 65 | 32.02 |
+| External monitoring service | 60 | 29.56 |
+| Information from metadatabase | 51 | 25.12 |
+| Statsd | 49 | 24.14 |
+| Other | 31 | 15.27 |
+
+The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.
+
+**How do you deploy Airflow?**
+
+| | No. | % |
+|----------------------------------------------------------------------|-------|-------|
+| On virtual machines (for example using AWS EC2) | 64 | 31.53 |
+| Using a managed service like Astronomer, Google Composer or AWS MWAA | 35 | 17.24 |
+| On Kubernetes (using custom deployments) | 29 | 14.29 |
+| On premises | 28 | 13.79 |
+| On Kubernetes (using another helm chart) | 20 | 9.85 |
+| On Kubernetes (using Apache Airflow's helm chart) | 17 | 8.37 |
+| Other | 12 | 5.91 |
+
+Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This about
Review comment:
```suggestion
Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
Review comment:
```suggestion
The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance?**
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5 |
+
+**What is the maximum number of tasks that you have used in one DAG?**
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow version it's worrying that the rest
Review comment:
```suggestion
that more than 60% of users use one of three latest Airflow versions, it's worrying that the rest
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
Review comment:
```suggestion
That said - if you know something we can improve please reach out via Slack, dev list or Github
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance?**
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5 |
+
+**What is the maximum number of tasks that you have used in one DAG?**
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow version it's worrying that the rest
+are using versions that are old or have known security vulnerabilities.
+
+Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.
+
+**What meta-database do you use?**
+
+| | No. | % |
+|--------------|-------|-------|
+| Postgres 12 | 36 | 17.73 |
+| Postgres 9.6 | 33 | 16.26 |
+| Postgres 11 | 31 | 15.27 |
+| MySQL 5.7 | 27 | 13.3 |
+| MySQL 8.0 | 20 | 9.85 |
+| Postgres 10 | 20 | 9.85 |
+| Other | 19 | 9.36 |
+| Postgres 13 | 18 | 8.87 |
+
+This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.
+
+It's good to know that users rather avoid using SQLite in production deployments!
+
+**What executor type do you use?**
+
+
+
+| | No. | 2020% | 2019% |
+|------------|-------|-------|-------|
+| Celery | 100 | 49.26 | 44.81%|
+| Kubernetes | 48 | 23.65 | 16.88%|
+| Local | 40 | 19.7 | 27.60%|
+| Sequential | 10 | 4.93 | 7.14% |
+| Other | 5 | 2.46 | 3.57 |
+
+In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users' deployment are growing, and they need more scalable solutions.
Review comment:
```suggestion
suggest that users' deployments are growing, and they need more scalable solutions.
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance?**
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5 |
+
+**What is the maximum number of tasks that you have used in one DAG?**
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow version it's worrying that the rest
+are using versions that are old or have known security vulnerabilities.
+
+Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.
+
+**What meta-database do you use?**
+
+| | No. | % |
+|--------------|-------|-------|
+| Postgres 12 | 36 | 17.73 |
+| Postgres 9.6 | 33 | 16.26 |
+| Postgres 11 | 31 | 15.27 |
+| MySQL 5.7 | 27 | 13.3 |
+| MySQL 8.0 | 20 | 9.85 |
+| Postgres 10 | 20 | 9.85 |
+| Other | 19 | 9.36 |
+| Postgres 13 | 18 | 8.87 |
+
+This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.
+
+It's good to know that users rather avoid using SQLite in production deployments!
+
+**What executor type do you use?**
+
+
+
+| | No. | 2020% | 2019% |
+|------------|-------|-------|-------|
+| Celery | 100 | 49.26 | 44.81%|
+| Kubernetes | 48 | 23.65 | 16.88%|
+| Local | 40 | 19.7 | 27.60%|
+| Sequential | 10 | 4.93 | 7.14% |
+| Other | 5 | 2.46 | 3.57 |
+
+In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users' deployment are growing, and they need more scalable solutions.
+
+Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.
+
+
+**What metrics do you use to monitor Airflow?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| I do not use monitoring | 65 | 32.02 |
+| External monitoring service | 60 | 29.56 |
+| Information from metadatabase | 51 | 25.12 |
+| Statsd | 49 | 24.14 |
+| Other | 31 | 15.27 |
+
+The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.
+
+**How do you deploy Airflow?**
+
+| | No. | % |
+|----------------------------------------------------------------------|-------|-------|
+| On virtual machines (for example using AWS EC2) | 64 | 31.53 |
+| Using a managed service like Astronomer, Google Composer or AWS MWAA | 35 | 17.24 |
+| On Kubernetes (using custom deployments) | 29 | 14.29 |
+| On premises | 28 | 13.79 |
+| On Kubernetes (using another helm chart) | 20 | 9.85 |
+| On Kubernetes (using Apache Airflow's helm chart) | 17 | 8.37 |
+| Other | 12 | 5.91 |
+
+Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This about
+10 percent point more than in 2019. There's slightly increase in usage of Airflow via
Review comment:
```suggestion
10 percent more than in 2019. There's slightly increase in usage of Airflow via
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
Review comment:
```suggestion
Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.
```
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
Review comment:
```suggestion
When asked what lineage product users use, the answers were varying from custom tools
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r584085349
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
Review comment:
Good point, I will add to question line information about single/multi choice 👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r580042316
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
Review comment:
```suggestion
While the first point may be addressed by (TaskFlow API)[http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api] in Airflow 2.0 the other two are definitely addressed
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579711691
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
Review comment:
This got to be fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579717563
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many of drawbacks was addressed in 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
Review comment:
```suggestion
It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
is true, we will learn next year!
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579718181
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many of drawbacks was addressed in 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants mentioned also
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing?**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out to use either via slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow?**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is a good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow?**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow?**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem?**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role?**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs?**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services?**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs?**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for?**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature?**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product user use the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance? **
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5
Review comment:
```suggestion
| 1000+ | 10 | 5 |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek merged pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek merged pull request #385:
URL: https://github.com/apache/airflow-site/pull/385
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] turbaszek commented on pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#issuecomment-793103863
@leahecole @XD-DENG please take a look to unblock merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r579717780
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,426 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "AAAAAAAA"
+tags: ["community", "survey", "users"]
+date: "2020-02-15"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily.
+
+It's important to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many of drawbacks was addressed in 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+
+## Overview of the user
+
+**What best describes your current occupation?**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company?**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow?**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines?**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engine. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by (TaskFlow API)[link] in Airflow 2.0 the other two are definitely addressed
Review comment:
Just a reminder to add link to `TaskFlow API`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow-site] kaxil commented on a change in pull request #385: Add Apache Airflow Survey 2020 blog post
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #385:
URL: https://github.com/apache/airflow-site/pull/385#discussion_r589745477
##########
File path: landing-pages/site/content/en/blog/airflow-survey-2020/index.md
##########
@@ -0,0 +1,434 @@
+---
+title: "Airflow Survey 2020"
+linkTitle: "Airflow Survey 2020"
+author: "Tomek Urbaszek"
+twitter: "turbaszek"
+github: "turbaszek"
+linkedin: "tomaszurbaszek"
+description: "We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance."
+tags: ["community", "survey", "users"]
+date: "2021-03-09"
+---
+# Apache Airflow Survey 2020
+
+World of data processing tools is growing steadily. Apache Airflow seems to be already considered as
+crucial component of this complex ecosystem. We observe steady growth in number of users as well as in
+an amount of active contributors. So listening and understanding our community is of high importance.
+
+It's worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and
+possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this
+is true, we will learn next year!
+
+## Overview of the user
+
+**What best describes your current occupation? (single choice)**
+
+| | No. | % |
+|---------------------|-------|-------|
+| Data Engineer | 115 | 56.65 |
+| Developer | 28 | 13.79 |
+| DevOps | 17 | 8.37 |
+| Solutions Architect | 14 | 6.9 |
+| Data Scientist | 12 | 5.91 |
+| Other | 10 | 4.93 |
+| Data Analyst | 4 | 1.97 |
+| Support Engineer | 3 | 1.48 |
+
+Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of
+our users are data engineers, scientists or analysts. The 2020 results are similar to [those from 2019](https://airflow.apache.org/blog/airflow-survey/) with
+visible slight increase in ML use cases.
+
+Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.
+
+**How many people work in your company? (single choice)**
+
+| | No. | % |
+|--------|-------|-------|
+| 200+ | 107 | 52.71 |
+| 51-200 | 44 | 21.67 |
+| 11-50 | 37 | 18.23 |
+| 1-10 | 15 | 7.39 |
+
+**How many people in your company use Airflow? (single choice)**
+
+| | No. | % |
+|-------|-------|-------|
+| 1-5 | 84 | 41.38 |
+| 6-20 | 75 | 36.95 |
+| 21-50 | 23 | 11.33 |
+| 50+ | 21 | 10.34 |
+
+Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work
+fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.
+
+
+**Are you considering moving to other workflow engines? (single choice)**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| No, we are happy with Airflow | 174 | 85.71 |
+| Yes | 29 | 14.29 |
+
+Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually
+justified by need of **easier workflow writing experience** (12.32%), **better UI/UX** and **faster scheduler**
+(8.37% both).
+
+While the first point may be addressed by [TaskFlow API](http://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskflow-api) in Airflow 2.0 the other two are definitely addressed
+in the new major version. And the early feedback from 2.0 users seems to be confirming it.
+
+The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned
+Luigi, Kubeflow or custom solutions.
+
+
+**Are you or your team actively participating in Airflow development - contributing? (single choice)**
+
+| | No. | % |
+|-----------------|-------|-------|
+| I wish we could | 99 | 48.77 |
+| No | 59 | 29.06 |
+| Yes | 45 | 22.17 |
+
+This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project!
+But it would be good to learn if there's something else than time that is stopping people who wish to contribute
+from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve.
+That said - if you know something we can improve please reach out via Slack, dev list or Github
+discussions.
+
+**How likely are you to recommend Apache Airflow? (single choice)**
+
+| | No. | 2020 % | 2019 % |
+|---------------|-------|--------|--------|
+| Very Likely | 125 | 61.58 | 45.45% |
+| Likely | 62 | 30.54 | 40.26% |
+| Neutral | 11 | 5.42 | 10.71% |
+| Unlikely | 3 | 1.48 | 2.60% |
+| Very unlikely | 2 | 0.99 | 0.97% |
+
+Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.
+
+**What is your source of information about Airflow? (multiple choice)**
+
+| | No. | % |
+|-----------------------------------|-------|-------|
+| Documentation | 154 | 75.86 |
+| Airflow website | 139 | 68.47 |
+| Slack | 128 | 63.05 |
+| Github | 127 | 62.56 |
+| Stack Overflow | 72 | 35.47 |
+| Airflow Summit Videos | 44 | 21.67 |
+| The dev mailing list | 33 | 16.26 |
+| Awesome Apache Airflow repository | 21 | 10.34 |
+| Other | 15 | 7.39 |
+
+Here we see that Airflow documentation is the crucial source of information. What's interesting is that more
+than 60% of users are getting information from Github and Slack channels.
+
+
+
+## Airflow uses cases
+
+**Do you have any customisation of Airflow? (single choice)**
+
+| | No. | % |
+|--------------------------------------|-------|-------|
+| No, we use vanilla Airflow | 154 | 75.86 |
+| Yes, we have small patches (no fork) | 34 | 16.75 |
+| Yes, we have separate fork | 15 | 7.39 |
+
+**When onboarding new members to airflow, what is the biggest problem? (multiple choice)**
+
+| | No. | % |
+|-----------------------------------------------------------------|-------|-------|
+| No guide on best practises on developing DAGs | 102 | 50.25 |
+| There is no easy option to launch Airflow | 64 | 31.53 |
+| Small number of tutorials on different aspects of using Airflow | 57 | 28.08 |
+| Documentation is not clear enough | 53 | 26.11 |
+| There is no easy option to deploy DAGs to an Airflow instance | 52 | 25.62 |
+| No problems | 34 | 16.75 |
+| Small number of blogs regarding Airflow | 30 | 14.78 |
+
+**Which interface(s) of Airflow do you use as part of your current role? (multiple choice)**
+
+| | No. | % |
+|-------------------------------------------------------|-------|-------|
+| Original Airflow Graphical User Interface | 199 | 98.03 |
+| CLI | 88 | 43.35 |
+| API | 48 | 23.65 |
+| Custom (own created) Airflow Graphical User Interface | 12 | 5.91 |
+| Other | 3 | 1.48 |
+
+**Do you combine multiple DAGs? (multiple choice)**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, by triggering another DAG | 87 | 42.86 |
+| No, I don't combine multiple DAGs | 79 | 38.92 |
+| Yes, through SubDAG | 40 | 19.7 |
+| Other | 18 | 8.87 |
+
+
+**How do you integrate with external services? (multiple choice)**
+
+| | No. | % |
+|---------------------------------------------|-------|-------|
+| Using existing dedicated operators / hooks | 147 | 72.41 |
+| Using Bash / Python operator | 140 | 68.97 |
+| Using own custom operators / hooks | 138 | 67.98 |
+| Other | 12 | 5.91 |
+
+**What external services do you use in your Airflow DAGs? (multiple choice)**
+
+| | No. | % |
+|---------------------------------------------------|-------|-------|
+| Amazon Web Services | 121 | 59.61 |
+| Internal company systems | 113 | 55.67 |
+| Google Cloud Platform / Google APIs | 97 | 47.78 |
+| Hadoop / Spark / Flink / Other Apache software | 72 | 35.47 |
+| Microsoft Azure | 21 | 10.34 |
+| Other | 19 | 9.36 |
+| I do not use external services in my Airflow DAGs | 5 | 2.46 |
+
+
+
+**Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)**
+
+| | No. | % |
+|----------------------------------------|-------|-------|
+| Adding new operators/sensors and hooks | 119 | 58.62 |
+| I don't use Airflow plugins | 69 | 33.99 |
+| Adding AppBuilder views & menu items | 27 | 13.3 |
+| Adding new executors | 17 | 8.37 |
+| Adding OperatorExtraLinks | 13 | 6.4 |
+| Other
+
+**Do you use Airflow's data lineage feature? (single choice)**
+
+| | No. | % |
+|------------------------------------------------------------|-------|-------|
+| No, I will use such feature if fully supported in Airflow | 105 | 51.72 |
+| No, data lineage isn’t a concern for my usage. | 68 | 33.5 |
+| Yes, I use another data lineage product | 24 | 11.82 |
+| Yes, I use custom implementation | 5 | 2.46 |
+| Yes, I use Airflow's experimental data lineage feature | 1 | 0.49 |
+
+When asked what lineage product users use, the answers were varying from custom tools
+to known product like Amundsen, Atlas or dbt.
+
+## Deployment
+
+**How many active DAGs do you have in your largest Airflow instance? (open question)**
+
+| Number of DAGs | No. | % |
+|-------------------|-------|-----|
+| < 20 | 64 | 32 |
+| 21-40 | 33 | 16 |
+| 41-60 | 13 | 6 |
+| 61-100 | 32 | 16 |
+| 101-200 | 31 | 15 |
+| 201-300 | 8 | 4 |
+| 301-999 | 12 | 6 |
+| 1000+ | 10 | 5 |
+
+**What is the maximum number of tasks that you have used in one DAG? (open question)**
+
+| Number of DAGs | No. | % |
+|------------------|-------|-----|
+| < 10 | 42 | 21 |
+| 11-20 | 31 | 15 |
+| 21-30 | 15 | 7 |
+| 31-40 | 11 | 5 |
+| 41-50 | 22 | 11 |
+| 51-100 | 39 | 19 |
+| 101-200 | 16 | 8 |
+| 201-500 | 16 | 8 |
+| 501+ | 11 | 5 |
+
+**Which version of Airflow do you use currently? (single choice)**
+
+| | No. | % |
+|-----------------|-------|-------|
+| 1.10.14 | 55 | 27.09 |
+| 2.0.0+ | 45 | 22.17 |
+| 1.10.12 | 27 | 13.3 |
+| 1.10.10 | 26 | 12.81 |
+| 1.10.11 | 14 | 6.9 |
+| 1.10.5 or older | 10 | 4.93 |
+| 1.10.9 | 8 | 3.94 |
+| 1.10.13 | 7 | 3.45 |
+| 1.10.6 | 4 | 1.97 |
+| 1.10.7 | 4 | 1.97 |
+| 1.10.8 | 3 | 1.48 |
+
+This was probably one of the most important questions in the survey. While it's good to see
+that more than 60% of users use one of three latest Airflow versions, it's worrying that the rest
+are using versions that are old or have known security vulnerabilities.
+
+Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.
+
+**What meta-database do you use? (single choice)**
+
+| | No. | % |
+|--------------|-------|-------|
+| Postgres 12 | 36 | 17.73 |
+| Postgres 9.6 | 33 | 16.26 |
+| Postgres 11 | 31 | 15.27 |
+| MySQL 5.7 | 27 | 13.3 |
+| MySQL 8.0 | 20 | 9.85 |
+| Postgres 10 | 20 | 9.85 |
+| Other | 19 | 9.36 |
+| Postgres 13 | 18 | 8.87 |
+
+This means that more about 69% of users decide to use Postgres as their meta-database.
+MySQL is the choice of nearly 24% users. The other responses included some MySQL versions
+like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.
+
+It's good to know that users rather avoid using SQLite in production deployments!
+
+**What executor type do you use? (single choice)**
+
+
+
+| | No. | 2020 | 2019 |
+|------------|-------|--------|--------|
+| Celery | 100 | 49.26% | 44.81% |
+| Kubernetes | 48 | 23.65% | 16.88% |
+| Local | 40 | 19.7% | 27.60% |
+| Sequential | 10 | 4.93% | 7.14% |
+| Other | 5 | 2.46% | 3.57 |
+
+In comparison to previous year it seems that more users use currently Celery and
+Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may
+suggest that users' deployments are growing, and they need more scalable solutions.
+
+Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest
+is using other brokers or is not sure what is used in their deployments.
+
+
+**What metrics do you use to monitor Airflow? (multiple choice)**
+
+| | No. | % |
+|-------------------------------|-------|-------|
+| I do not use monitoring | 65 | 32.02 |
+| External monitoring service | 60 | 29.56 |
+| Information from metadatabase | 51 | 25.12 |
+| Statsd | 49 | 24.14 |
+| Other | 31 | 15.27 |
+
+The other responses included mostly information about tools used by users
+including DataDog and Prometheus exporter.
+
+**How do you deploy Airflow? (single choice)**
+
+| | No. | % |
+|----------------------------------------------------------------------|-------|-------|
+| On virtual machines (for example using AWS EC2) | 64 | 31.53 |
+| Using a managed service like Astronomer, Google Composer or AWS MWAA | 35 | 17.24 |
+| On Kubernetes (using custom deployments) | 29 | 14.29 |
+| On premises | 28 | 13.79 |
+| On Kubernetes (using another helm chart) | 20 | 9.85 |
+| On Kubernetes (using Apache Airflow's helm chart) | 17 | 8.37 |
+| Other | 12 | 5.91 |
+
+Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about
+10 percent more than in 2019. There's slightly increase in usage of Airflow via
+managed services (14.61% in 2019).
+
+**Do you use containerisation for deployment? (single choice)**
+
+| | No. | % |
+|------------------------------------|-------|-------|
+| Yes, using helm chart / kubernetes | 58 | 28.57 |
+| No, I don’t use containerisation | 57 | 28.08 |
+| Yes, single docker image | 49 | 24.14 |
+| Yes, using docker compose | 39 | 19.21 |
+
+Among users who do not use Kubernetes based deployments 58% of them use containerisation. About
+42% of those users use docker-compose for deployments.
+
+**How do you distribute your DAGs? (single choice)**
+
+| | No. | % |
+|---------------------------------------------------------|-------|-------|
+| Using a synchronizing process (Git sync, GCS fuse, etc) | 79 | 38.92 |
+| Bake them into the docker image | 56 | 27.59 |
+| Shared files system | 34 | 16.75 |
+| Other | 20 | 9.85 |
+| I don’t know | 14 | 6.9 |
+
+The most popular way of distributing DAGs seems to be using a synchronizing process. About
+40% of users use this process together with Kubernetes deployments.
+
+## Future of Airflow
+
+**In your opinion, what could be improved in Airflow? (multiple choice)**
+
+| | No. | % |
+|-----------------------------------------------------------|-------|-------|
+| Web UI | 100 | 49.26 |
+| Examples, how-to, onboarding documentation | 90 | 44.33 |
+| Logging, monitoring and alerting | 90 | 44.33 |
+| Technical documentation | 90 | 44.33 |
+| Scheduler performance | 83 | 40.89 |
+| DAG authoring | 64 | 31.53 |
+| Authentication and authorization | 58 | 28.57 |
+| REST API | 51 | 25.12 |
+| Other | 44 | 21.67 |
+| Reliability | 41 | 20.2 |
+| External integration e.g. AWS, GCP, Apache products | 36 | 17.73 |
+| Security | 28 | 13.79 |
+| CLI | 20 | 9.85 |
+| Everything work fine for me | 14 | 6.9 |
+| I don’t know | 4 | 1.97 |
+
+**Which features would most interest you? (multiple choice)**
+
+| | No. | % |
+|-------------------------------------------------------------|-------|-------|
+| DAG versioning | 109 | 53.69 |
+| Builtin statistics | 71 | 34.98 |
+| Improved data lineage | 65 | 32.02 |
+| Scheduling at the start of the interval | 63 | 31.03 |
+| Stateless workers | 59 | 29.06 |
+| More option to configure schedules (time units, increments) | 57 | 28.08 |
+| Multi-tenant deployment | 49 | 24.14 |
+| DAG fetcher (AIP-5) | 39 | 19.21 |
+| Generic transfer operator | 34 | 16.75 |
+| Other | 33 | 16.26 |
+| I have everything I need | 11 | 5.42 |
+| Nothing | 11 | 5.42 |
+
+**Will you consider migrating to Airflow 2.0? (single choice)**
+
+| | No. | % |
+|-----------------------------------------------|-------|-------|
+| Yes, as soon as possible | 81 | 39.9 |
+| Yes, once it’s mature (for example after 2.1) | 72 | 35.47 |
+| I am already using Airflow 2.0+ | 39 | 19.21 |
+| I don't know yet | 8 | 3.94 |
+| No, I do not plan to migrate | 3 | 1.48 |
+
+**What are the features of Airflow 2.0 you are most excited about? (multiple choice)**
+
+| | No. | % |
+|----------------------------------|-------|-------|
+| General performance improvements | 133 | 65.52 |
+| Refreshed WebUI | 102 | 50.25 |
+| Scheduler HA | 99 | 48.77 |
+| Official docker image | 84 | 41.38 |
+| @task decorator | 56 | 27.59 |
+| Official helm chart | 51 | 25.12 |
+| Providers packages | 41 | 20.2 |
+| Configurable XCom backends | 33 | 16.26 |
+| CeleryKubernetesExecutor | 31 | 15.27 |
+| Other | 12 | 5.91 |
+
+## Summary
+
+From open source point of view is good to see that many people would love to contribute to Apache Airflow.
+This means that there are resources that if unleashed may make our community even stronger. From product
+perspective it's important to know that users are usually using the latest versions of our software and
+are willing to upgrade to new ones.
+
+Finally, there are still some things to improve - documentation, on boarding guides and plug-and-play airflow
+deployments. However, we hope that with the increase of adoption there will be an increase in people willing
+to share their experience and tools.
Review comment:
```suggestion
From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow.
This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and
are willing to upgrade to new ones.
Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow
deployments. However, we hope that with the increase of adoption there will be an increase in people willing
to share their experience and tools.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org