You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/12/31 14:07:52 UTC
[PR] Include plugins in the architecture diagrams [airflow]
potiuk opened a new pull request, #36513:
URL: https://github.com/apache/airflow/pull/36513
When it comes to access and management plugins management follows different pattern than DAG files. While DAG files can (and should) be modified by DAG authors, the whole idea of Plugins was to make it only possible to modify plugins folder (and installl plugin-enabled packages) by the Deploymenet Managers, not DAG authors.
The difference is quite important because even in a simplest installation, airflow webserver never needs to access DAG files, while it should be able to access plugins.
This is even more profound in the environment (leading in the future to multi-tenant deployments) plugins are not 'per-tenant" - they must be installed and managed by deployment manager, because those plugins can be used by Airflow Webservers.
In the future we might want to make distinction between these two different types of plugins, because theorethically it would be possible to distingquish "scheduler, worker & triggerer" plugins from the "webserver" plugins - however we do not have such disctinction today and whoever manages plugins folder is impacting both webserver and workers.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1872962105
> I feel all the different arrow styles require a legend. What does blue vs red and solid vs dashed indicate?
Yep. Will add it. There is a good reasoning for that and deliberate style/coloring. I will describe it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438907553
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
Review Comment:
I removed this chapter altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1877679277
@BasPH -> do you think it's cool to go ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438892537
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
-Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything, either with high-level support from one of our providers, or directly as a command using the shell or Python :doc:`operators`.
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
Review Comment:
Yeah. Goopd idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440828944
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
Review Comment:
yep. I also added a separate sentence explaining that there is no tirggerrer so taask deferral is not possible (task deferral is mentioned above and linked to the documentation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438929790
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
-Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything, either with high-level support from one of our providers, or directly as a command using the shell or Python :doc:`operators`.
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
Review Comment:
Added more links to specific docs in the overview.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1873975167
I will take a closer look at other comments but one thing first:
> I see several roles being mentioned, but don't think it's currently clear what those are (Deployment manager and Operations User). Such roles can have different names in different companies, so I don't think it's immediately clear what those mean.
> Can we align/have a doc describing Airflow's vision on different roles in an organisation? That way we can link to it when mentioned in a doc, and the reader can map Airflow's role names to their own organisation.
We already have it. It's called "User types" but we shoud change it it in user role - and its rather comprehensively explained in our security model (which my change links to from the architecture) :
https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#airflow-security-model-user-types
Also Deployment Manager's responsibilities are very well described in the installation document: https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html where Deployment Manager is not mentioned "per se" but in this page "you" when referred in the docs means "Deployment Manager" - which we should clarify.
The question is - should it be a separate page with **just** user roles, and link to it from all three ?
* Security Model
* Installation
* Architecure Overview
Sounds like a new "User Roles" page in the "Administration and Deployment" ? WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440820704
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
Review Comment:
I split that into three separate paragraphs. - one about scalability one about security and following chapter about roles. I decided for now not to split out separate "roles" document. I just briefly mentioned the roles and described it here, and linked to the "security model" for more details.
I also moved the following chapter
```
Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
either with high-level support from one of our providers, or directly as a command using the shell
or Python :doc:`operators`.
```
to the top-level description, as it feels that it should be there rather than in the "components" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "BasPH (via GitHub)" <gi...@apache.org>.
BasPH commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1439342399
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
+
+Such installation typically does not separate user roles - deployment, configuration, operation, authoring
+and maintenance are all done by the same person and there are no security perimeters between the components.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+If you want to run Airflow on a single machine in a simple single-machine setup, you can skip the
+more complex diagrams below and go straight to the :ref:`overview:workloads` section.
+
+Distributed Airflow architecture
+................................
+
+This is the architecture of Airflow where components of Airflow are distributed among multiple machines
+and where various roles of users are introduced - **Deployment Manager**, **DAG author**,
+**Operations User**. You can read more about those various roles in the :doc:`/security/security_model`.
+
+In case of distributed deployment, it is much more important to consider security aspects of the components.
+The *webserver* does not have access to the *DAG files* directly (the code you see in the Code tab of the
+UI is synchronized via the *metadata database*). This way the *webserver* cannot execute any
+code submitted by **DAG author**. It can only execute code that is installed as *installed packages* or
+*plugins* by the **Deployment Manager**. Also the **Operations User** has only access to the UI and can
+only trigger DAGs and tasks, but cannot author DAGs.
Review Comment:
```suggestion
The *webserver* does not have access to the *DAG files* directly. The code in the Code tab of the
UI is read from the *metadata database*. The *webserver* cannot execute any code submitted by the **DAG author**. It can only execute code that is installed as an *installed package* or *plugin* by the **Deployment Manager**. The **Operations User** only has access to the UI and can only trigger DAGs and tasks, but cannot author DAGs.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440802365
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
Review Comment:
Nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438910073
##########
docs/apache-airflow/img/diagram_basic_airflow_architecture.png:
##########
Review Comment:
Free for any usage, no attribution needed. Should be fine (matticons)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438900041
##########
docs/apache-airflow/img/diagram_basic_airflow_architecture.png:
##########
Review Comment:
Just checking it seems this framework only contains "ugly" icons, then I opt for diagrams.programming.flowchart.MultipleDocuments :-D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1875878444
I think with all those changes, iterations it starts to look reallly good and rather professional:
This:
<img width="1314" alt="Screenshot 2024-01-03 at 20 43 29" src="https://github.com/apache/airflow/assets/595491/dcfeac38-1f1d-4b6b-b126-1a7282ba0c97">
And this:
<img width="1315" alt="Screenshot 2024-01-03 at 20 34 39" src="https://github.com/apache/airflow/assets/595491/a12375fa-98e6-49c9-8cba-ce8c76783431">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "BasPH (via GitHub)" <gi...@apache.org>.
BasPH commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1439322639
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
Review Comment:
```suggestion
------------------
Airflow's architecture consists of multiple components. The following sections describe each component's function and whether they're required for a bare-minimum Airflow installation, or an optional component to achieve better Airflow extensibility, performance, and scalability.
```
It's a best practice not to stack headings. Let's write at least a short descriptive text explaining what's coming.
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
Review Comment:
```suggestion
A DAG specifies the dependencies between tasks, which defines the order in which to execute the tasks. Tasks describe what to do, be it fetching data, running analysis, triggering other systems, or more.
```
Small rewording for readability. Retries can also be defined on task-level + I don't think retries are critical to the Airflow architecture so I'd leave it out here.
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
Review Comment:
```suggestion
Some Airflow components are optional and can enable better extensibility, scalability, and performance in your Airflow:
```
Would explain _why_ they're optional here.
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
Review Comment:
IMO it's confusing this is mentioned in both the required and optional components. Would leave it only in the required components, because without DAG folder you have no functioning Airflow.
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
Review Comment:
By "deployed Python applications" you're referring to Airflow components, right?
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
+
+Such installation typically does not separate user roles - deployment, configuration, operation, authoring
+and maintenance are all done by the same person and there are no security perimeters between the components.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+If you want to run Airflow on a single machine in a simple single-machine setup, you can skip the
+more complex diagrams below and go straight to the :ref:`overview:workloads` section.
+
+Distributed Airflow architecture
+................................
+
+This is the architecture of Airflow where components of Airflow are distributed among multiple machines
+and where various roles of users are introduced - **Deployment Manager**, **DAG author**,
+**Operations User**. You can read more about those various roles in the :doc:`/security/security_model`.
+
+In case of distributed deployment, it is much more important to consider security aspects of the components.
Review Comment:
```suggestion
In the case of a distributed deployment, it is important to consider the security aspects of the components.
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
Review Comment:
Let's make this header a different style because it's now on the same level as "Required components" and "Optional components".
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
Review Comment:
```suggestion
* Optional *dag processor*, which parses DAG files and serializes them into the
*metadata database*. By default, the *dag processor* process is part of the scheduler, but it can be run as a separate component for scalability reasons.
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
Review Comment:
```suggestion
* A folder of *DAG files* is read by the *scheduler* to figure out what tasks to run and when and to
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
Review Comment:
```suggestion
where deferred tasks are not used, a triggerer is not necessary. More about deferring tasks can be
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
Review Comment:
I'm confused about the key message of this paragraph. What is the key message here? "Airflow is scalable"? "Airflow supports roles"?
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
+
+Such installation typically does not separate user roles - deployment, configuration, operation, authoring
+and maintenance are all done by the same person and there are no security perimeters between the components.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+If you want to run Airflow on a single machine in a simple single-machine setup, you can skip the
+more complex diagrams below and go straight to the :ref:`overview:workloads` section.
+
+Distributed Airflow architecture
+................................
+
+This is the architecture of Airflow where components of Airflow are distributed among multiple machines
+and where various roles of users are introduced - **Deployment Manager**, **DAG author**,
+**Operations User**. You can read more about those various roles in the :doc:`/security/security_model`.
+
+In case of distributed deployment, it is much more important to consider security aspects of the components.
+The *webserver* does not have access to the *DAG files* directly (the code you see in the Code tab of the
+UI is synchronized via the *metadata database*). This way the *webserver* cannot execute any
+code submitted by **DAG author**. It can only execute code that is installed as *installed packages* or
+*plugins* by the **Deployment Manager**. Also the **Operations User** has only access to the UI and can
+only trigger DAGs and tasks, but cannot author DAGs.
+
+The *DAG files* need to be synchronized between all the components that use them - *scheduler*,
+*triggerer* and *workers*. The *DAG files* can be synchronized by various mechanisms - for example by
+using a shared filesystem, or by using a version control system like Git.
Review Comment:
> or by using a version control system like Git
I think that doesn't cover the full message - the user needs to set up Git sync for that to work.
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
Review Comment:
```suggestion
* A *metadata database*, used by the *scheduler* and *webserver*, to store state of workflows and tasks.
```
Shouldn't we just say "all Airflow components"?
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
Review Comment:
```suggestion
* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality (similar to installed packages).
Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
```
Would describe the key message here (what is a plugin), and leave out deployment details.
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
Review Comment:
Could you add a link to the local executor here?
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
Review Comment:
```suggestion
A minimal Airflow installation consists of the following components:
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
Review Comment:
```suggestion
* Optional *triggerer*, which executes deferred tasks in an asyncio event loop. In a basic installation
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
+
+Such installation typically does not separate user roles - deployment, configuration, operation, authoring
+and maintenance are all done by the same person and there are no security perimeters between the components.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+If you want to run Airflow on a single machine in a simple single-machine setup, you can skip the
+more complex diagrams below and go straight to the :ref:`overview:workloads` section.
+
+Distributed Airflow architecture
+................................
+
+This is the architecture of Airflow where components of Airflow are distributed among multiple machines
+and where various roles of users are introduced - **Deployment Manager**, **DAG author**,
+**Operations User**. You can read more about those various roles in the :doc:`/security/security_model`.
+
+In case of distributed deployment, it is much more important to consider security aspects of the components.
+The *webserver* does not have access to the *DAG files* directly (the code you see in the Code tab of the
+UI is synchronized via the *metadata database*). This way the *webserver* cannot execute any
+code submitted by **DAG author**. It can only execute code that is installed as *installed packages* or
+*plugins* by the **Deployment Manager**. Also the **Operations User** has only access to the UI and can
+only trigger DAGs and tasks, but cannot author DAGs.
Review Comment:
```suggestion
The *webserver* does not have access to the *DAG files* directly. The code in the Code tab of the
UI is read from the *metadata database*. The *webserver* cannot execute any code submitted by **DAG author**. It can only execute code that is installed as *installed packages* or *plugins* by the **Deployment Manager**. The **Operations User** only has access to the UI and can only trigger DAGs and tasks, but cannot author DAGs.
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
Review Comment:
Let's break this sentence up into multiple sentences. How about:
```suggestion
machine. Such a deployment usually uses the LocalExecutor, where the *scheduler* and the *workers* are in
the same Python process and the *DAG files* are read directly from the local filesystem by the *scheduler*.
The *webserver* runs on the same machine as the *scheduler*.
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
+
+Such installation typically does not separate user roles - deployment, configuration, operation, authoring
Review Comment:
```suggestion
Such an installation typically does not separate user roles - deployment, configuration, operation, authoring
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1875880879
And:
<img width="1316" alt="Screenshot 2024-01-03 at 20 45 41" src="https://github.com/apache/airflow/assets/595491/1a4a6d98-0819-4d8e-acec-44795013666c">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440846010
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
Review Comment:
Wow. This is where "require resolving conversations" turned out to be really helpful. I totally missed these two last comments when I reviewed it few moments ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438896755
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
Review Comment:
Thought a bit and I think I will simply add much simpler diagram very much like the original one - showing "self-contained" installation - without workers and triggerers and with single user and make it the first diagram - with explanation that this is the simplest configuration and explaination that it only serves very basic installation. This way the diagrams will get gradually more complex and showing "true" nature of some of the decisions made, while not confusing the users who **just** want to start their journey with Airflow.
I believe that will satisfy both - your desire @BasPH to have things simple and my desire to use the architecture diagrams of ours to explain some design decisions we have and to explain those who are looking for self-deployment of Airflow and want to understand our intentions.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438907293
##########
docs/apache-airflow/img/diagram_basic_airflow_architecture.png:
##########
Review Comment:
How about "just using" https://openclipart.org/detail/211760/matticonsfolderyellow ? I am not a licensing expert, is creative commons allowed to be used here when results are distributed with Apache License?
Also this would be OK: https://freesvg.org/brown-folder-icon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1873012436
Should be quite a bit nicer (I will still need to figure out how to do anchoring a bit better but the icons should be nicer now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1878779756
Ok. Will merge it now - we can always fix things later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440807358
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
Review Comment:
I merged this with **dag processor** paragraph before:
```
* Optional *dag processor*, which parses DAG files and serializes them into the
*metadata database*. By default, the *dag processor* process is part of the scheduler, but it can
be run as a separate component for scalability and security reasons. If *dag processor* is present
*scheduler** does not need to read the *DAG files* directly. More about
processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440794816
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
Review Comment:
Nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1873023020
Right I think I found a nice-enough layout.
Looks like "ortho" layout for edges is rather difficult for edge anchoring (even if Iike it most) - but with some graphviz tweaking seems that "spline" , "polyline", "line" layouts for "basic", "distributed", "dag processor" diagrams respectively combined wiht minlen for edges for better spacing produced much nicer and cleaner diagrams.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1872959349
This is part of preparation to the last stage of multi-tentant last AIP and documentation around it.
It was also partially result of the issue https://github.com/apache/airflow/issues/36510 which shows that Airflow's architecture - especially when it comes to "which component accesses what" and also "who has access to what" is oftem misunderstood (for some reason - for example - Trieggerer in Composer is configured as a component that does not have access to DAG files. Which was not really how we intended it to work. Since we have not put Triggers in Plugins our intention was that Triggerer CAN use Tiriggers defined in the DAG files and we allow DAG Authors to develop new Triggers (unlike Timetables, Listeners, Extra Linnks - all of which can only come from installed packages and plugins.
I think adding visual representation of it and showing the difference between DAG files, Plugins and packages is important to avoid confusion on what rolese all these "entities" have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438904061
##########
docs/apache-airflow/img/diagram_basic_airflow_architecture.png:
##########
Review Comment:
You can provide custom icons - no problem - this is how I added "scalable python" (i.e. multiple Python instances) ones - I draw it myself :D. We can iterate quite a bit and get better icons here - no problem.
Maybe you would like to suggest some - with the right usage rights of course @jscheffl ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440805208
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
Review Comment:
Yes. but also security reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440847798
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440808429
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
Review Comment:
Agree/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440835976
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
+the same component and the *DAG files* are read directly from the local filesystem by the *scheduler*
+and there is no separate *triggerer* to run deferred tasks and *webserver* runs on the same machine
+as the *scheduler*.
+
+Such installation typically does not separate user roles - deployment, configuration, operation, authoring
+and maintenance are all done by the same person and there are no security perimeters between the components.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+If you want to run Airflow on a single machine in a simple single-machine setup, you can skip the
+more complex diagrams below and go straight to the :ref:`overview:workloads` section.
+
+Distributed Airflow architecture
+................................
+
+This is the architecture of Airflow where components of Airflow are distributed among multiple machines
+and where various roles of users are introduced - **Deployment Manager**, **DAG author**,
+**Operations User**. You can read more about those various roles in the :doc:`/security/security_model`.
+
+In case of distributed deployment, it is much more important to consider security aspects of the components.
+The *webserver* does not have access to the *DAG files* directly (the code you see in the Code tab of the
+UI is synchronized via the *metadata database*). This way the *webserver* cannot execute any
+code submitted by **DAG author**. It can only execute code that is installed as *installed packages* or
+*plugins* by the **Deployment Manager**. Also the **Operations User** has only access to the UI and can
+only trigger DAGs and tasks, but cannot author DAGs.
+
+The *DAG files* need to be synchronized between all the components that use them - *scheduler*,
+*triggerer* and *workers*. The *DAG files* can be synchronized by various mechanisms - for example by
+using a shared filesystem, or by using a version control system like Git.
Review Comment:
I removed specific mentioning of the mechanisms - instead I linked to our helm-chart's `managing-dag-files` document that explains all the various options:
```
The *DAG files* need to be synchronized between all the components that use them - *scheduler*,
*triggerer* and *workers*. The *DAG files* can be synchronized by various mechanisms - typical
ways how DAGs can be synchronized are described in :doc:`helm-chart:manage-dag-files` ot our
Helm Chart documentation. Helm chart is one of the ways how to deploy Airflow in K8S cluster.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1872970410
> All the crossed dashed arrows make it difficult to follow the arrow. Would stick with solid lines.
Yep. I will work a bit on those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438898796
##########
docs/apache-airflow/img/diagram_basic_airflow_architecture.png:
##########
Review Comment:
I dis-like the icon for the plugin-folder (in terms of visual style) and the dotted arrows are ending on the outside borders of the icon.
Can you use a different icon for the folder?
e.g. something like (not exactly but in general):
https://www.vecteezy.com/vector-art/420464-vector-document-in-folder-icon
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
Review Comment:
I agree with @potiuk - but maybe this is currently also in the light of many security reviews. If no agreement, then the reasoning can be shortened a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #36513:
URL: https://github.com/apache/airflow/pull/36513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "BasPH (via GitHub)" <gi...@apache.org>.
BasPH commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438889862
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
Review Comment:
Would leave this out.
##########
docs/apache-airflow/img/diagram_dag_processor_airflow_architecture.py:
##########
@@ -64,10 +67,21 @@ def generate_dag_processor_airflow_diagram():
dag_processors = Custom("DAG\nProcessor(s)", PYTHON_MULTIPROCESS_LOGO.as_posix())
dag_files = MultipleDocuments("DAG files")
+ with Cluster("Deploymet environment"):
Review Comment:
```suggestion
with Cluster("Deployment environment"):
```
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
Review Comment:
What is the key message here?
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
-Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything, either with high-level support from one of our providers, or directly as a command using the shell or Python :doc:`operators`.
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
Review Comment:
Would add links here to the providers packages and specific operators mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438892404
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
Review Comment:
I actual;ly think of removing executor completely from that image. For all practical. purposes executor as component does not exist, it's merely a configuration parameter of scheduler and nowhere in "deployment architecture" executor is needed (it was there however in the original diagram). I will try to remove it (including the whole paragraph here and let's see if that is going to be better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438892198
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
Review Comment:
This is a source of major confusion. Many our users still think that webserver accesses the DAG files and needs it even 3 years after releasing Airflow 2. I've been answering at least 10 times to our users that this is not the case as they still continued to deploy Airflow webserver WITH DAG folder even if this was useless.
I believe we failed to communicate that - that's why I added it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438892537
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
-Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything, either with high-level support from one of our providers, or directly as a command using the shell or Python :doc:`operators`.
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
Review Comment:
Yeah. Good idea.
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
.. image:: ../img/diagram_basic_airflow_architecture.png
-Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component in Airflow overall, handling the actual task execution.
+Note that executor is only briefly mentioned as a link between Scheduler and workers.
+The executors will generally also introduce other components to let them talk to their workers -
+like a task queue - but you can still think of the executor and its workers as a single logical
+component in Airflow overall, handling the actual task execution - where executor is controlled by the
+scheduler.
Review Comment:
I actually think of removing executor completely from that image. For all practical. purposes executor as component does not exist, it's merely a configuration parameter of scheduler and nowhere in "deployment architecture" executor is needed (it was there however in the original diagram). I will try to remove it (including the whole paragraph here and let's see if that is going to be better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440800421
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
Review Comment:
Nit - to make it future-proof, I just wrote `components`. Not all airflow components are going to use the database in the future (that's the whole point of introducing the internal API). And the images are showing which one are accessing the DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1875866207
Applied the review comments @BasPH. I decided not to separate out the "roles" into separate document - but to briefly mention the roies and explicitly referring to security-model document for more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438907524
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
Review Comment:
OK. Let me know @BasPH and @jscheffl what you think now. We should - of course - tweak it still a bit (better icons as discussed below and possibly play a bit with graphviz/diagram layout to add some margins and get it a bit more readable. Plus I can add a few links to other parts of the documentation when I am mentioning other pieces - to get it all nicely hyperlinked.
But I **think** this one might achieve good balance between the simplicity and being able to explain concepts and architecture in "gradually more complex" ways - without overloading the users who just want to install airflow and use it locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438907595
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
Review Comment:
I **think** this might be resolved - please reopen if you think it's not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1872996653
> While we're changing the diagram, how about displaying a generic "database" icon instead of the PostgreSQL logo? PostgreSQL is the most used database but we do support others too.
--> diagrams.programming.flowchart.Database ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "BasPH (via GitHub)" <gi...@apache.org>.
BasPH commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1438914237
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -43,13 +43,26 @@ An Airflow installation generally consists of the following components:
Basic airflow architecture
--------------------------
-This is the basic architecture of Airflow that you'll see in simple installations:
+This is the basic architecture of Airflow that you'll see in simple installations.
+
+Note that even in the simple installation, Airflow Webserver does not access the DAG files directly
+(the code you see in the Code tab of the UI is synchronized via the metadata database).
+
+This is to allow for a more secure deployment, where the webserver has never the need or even
+possibility to executed code submitted by DAG authors, the only custom code that webserver can
+execute are the plugins and those should be only installed and deployed by the Airflow Deployment Manager.
Review Comment:
@potiuk Sounds good. It's not my goal to keep things simple per se. However, I do think we should introduce complex topics more gradually, one concept at a time. A piece of text, or an image, becomes difficult to comprehend if multiple previously unmentioned concepts are introduced at the same time to the reader.
So starting with a bare minimum diagram, and then saying "here's how a more complex setup might look like" is indeed a good approach to introduce the reader to a complex topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36513:
URL: https://github.com/apache/airflow/pull/36513#issuecomment-1873026702
And even a but nicer now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440793892
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
Review Comment:
Agree. Nice description!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Include plugins in the architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36513:
URL: https://github.com/apache/airflow/pull/36513#discussion_r1440823172
##########
docs/apache-airflow/core-concepts/overview.rst:
##########
@@ -18,49 +18,151 @@
Architecture Overview
=====================
-Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a :doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called :doc:`tasks`, arranged with dependencies and data flows taken into account.
+Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
+:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/edge_label_example.png
:alt: An example Airflow DAG, rendered in Graph
-A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries; the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.
+A DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries;
+the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems,
+or more.
-An Airflow installation generally consists of the following components:
+Airflow components
+------------------
-* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled workflows, and submitting :doc:`tasks` to the executor to run.
+Required components
+...................
-* An :doc:`executor <executor/index>`, which handles running tasks. In the default Airflow installation, this runs everything *inside* the scheduler, but most production-suitable executors actually push task execution out to *workers*.
+Minimal Airflow installation consists of the following components:
-* A *triggerer*, which executes deferred tasks - executed in an async-io event loop.
+* A :doc:`scheduler <../administration-and-deployment/scheduler>`, which handles both triggering scheduled
+ workflows, and submitting :doc:`tasks` to the executor to run. The :doc:`executor <executor/index>`, is
+ a configuration property of the *scheduler*, not a separate component and runs within the scheduler
+ process. There are several executors available out of the box, and you can also write your own.
-* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.
+* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
+ DAGs and tasks.
-* A folder of *DAG files*, read by the scheduler and executor (and any workers the executor has)
+* A folder of *DAG files*, is read by the *scheduler* to figure out what tasks to run and when and to
+ run them.
-* A *metadata database*, used by the scheduler, executor and webserver to store state.
+* A *metadata database*, used by the *scheduler*, and *webserver* to store state of workflows and tasks.
+ Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
+ Airflow to work.
+Optional components
+...................
-Basic airflow architecture
---------------------------
+There are also some optional components that are not present in the basic installation
-This is the basic architecture of Airflow that you'll see in simple installations:
+* Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
+ worker might be part of the scheduler not a separate component. It can be run as a long running process
+ in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
+ :doc:`KubernetesExecutor <executor/kubernetes>`.
+
+* Optional *triggerer*, which executes deferred tasks in an async-io event loop. In basic installation
+ where deferred tasks are not used, triggerer might not be present. More about deferring tasks can be
+ found in :doc:`/authoring-and-scheduling/deferring`.
+
+* Optional *dag processor*, which parses DAG files and synchronizes them into the
+ *metadata database* in basic installation *dag processor* might be part of the scheduler not
+ a separate component.
+
+* A folder of *DAG files*, is read by *dag processor*, *workers* and *triggerer* when they are running.
+ If *dag processor* is present *scheduler** does not need to read the *DAG files* directly. More about
+ processing DAG files can be found in :doc:`/authoring-and-scheduling/dagfile-processing`
+
+* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality and by placing Python
+ files in the plugins folder you can extend Airflow functionality (similarly as via installed packages).
+ Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ plugins can be found in :doc:`/authoring-and-scheduling/plugins`.
+
+Deploying Airflow components
+............................
+
+All the components are Python applications that can be deployed using various deployment mechanisms.
+
+They deployed Python applications can have extra *installed packages* installed in their Python environment.
+This is useful for example to install custom operators or sensors or extend Airflow functionality with custom
+plugins.
+
+While Airflow can be run in a single machine and with simple installation where only *scheduler* and
+*webserver* are deployed, Airflow is designed to be scalable and secure, and is able ot run in a distributed
+environment - where various components can run on different machines, with different security perimeters
+and can be scaled by running multiple instances of the components above. Also while single person can run and
+manage Airflow installation, Airflow Deployment in more complex setup can involve various roles of users
+as described in the :doc:`/security/security_model`.
+
+Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
+either with high-level support from one of our providers, or directly as a command using the shell
+or Python :doc:`operators`.
+
+Architecture Diagrams
+---------------------
+
+The diagrams below show different ways to deploy Airflow - gradually from the simple "one machine" and
+single person deployment, to a more complex deployment with separate components, separate user roles and
+finally with more isolated security perimeters.
+
+The meaning of the different connection types in the diagrams below is as follows:
+
+* **brown solid lines** represent *DAG files* submission and synchronization
+* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
+* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
+* **black solid lines** represent accessing the UI to manage execution of the workflows
+* **red dashed lines** represent accessing the *metadata database* by all components
+
+Basic Airflow deployment
+........................
+
+This is the simplest deployment of Airflow, usually operated and managed on a single
+machine. Such deployment usually uses Local Executor, where the *scheduler* and the *workers* are in
Review Comment:
Absolutely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org