You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/12/03 17:51:01 UTC

[PR] Generate airflow architecture diagram [airflow]

potiuk opened a new pull request, #36035:
URL: https://github.com/apache/airflow/pull/36035

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36035:
URL: https://github.com/apache/airflow/pull/36035#issuecomment-1839761024

   @BasPH - are you ok with keeping two users ? I do feel it's much `clearer` this way. And it sets the stage for 2.9 changes and "security-focus" we have now (and sets the stage for future, more "isolation" cases...
   
   I also quite like this one:
   
   ![image](https://github.com/apache/airflow/assets/595491/10a6887e-65f3-4645-9c5d-242924015f5f)
   
   
   Where the DAGs are "flowing" from left to righ -> from the author to someone who sees the result of execution :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "BasPH (via GitHub)" <gi...@apache.org>.
BasPH commented on code in PR #36035:
URL: https://github.com/apache/airflow/pull/36035#discussion_r1413194319


##########
dev/diagrams/pyproject.toml:
##########
@@ -0,0 +1,25 @@
+[build-system]
+requires = [
+  "setuptools",
+  "wheel",
+]
+
+[project]
+name = "generate-airflow-architecture"
+version = "0.1.0"
+description = "Generate airflow architecture"
+authors = [{ name = "Apache Airflow PMC", email = "dev@airflow.apache.org" }]
+requires-python = ">=3.8"
+classifiers = [
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+    "diagrams",
+    "rich"
+]
+scripts.generate_airflow_architecture = "generate_airflow_architectire:cli"

Review Comment:
   ```suggestion
   scripts.generate_airflow_architecture = "generate_airflow_architecture:cli"
   ```



##########
dev/diagrams/generate_airflow_architecture_diagram.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from diagrams.onprem.database import PostgreSQL
+from diagrams.programming.flowchart import MultipleDocuments
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+IMAGE_NAME = "basic_airflow_architecture_diagram.png"
+
+LOCAL_DIR = Path(__file__).parent
+AIRFLOW_SOURCES_ROOT = Path(__file__).parent.parent.parent
+DOCS_IMAGES_DIR = AIRFLOW_SOURCES_ROOT / "docs" / "apache-airflow" / "img"
+
+PYTHON_MULTI_PROCESS_LOGO = AIRFLOW_SOURCES_ROOT / "dev" / "images" / "python_multiprocess_logo.png"
+
+
+def generate_basic_airflow_diagram():
+    with Diagram(name="Basic Airflow Architecture Diagram", show=False, direction="TB", curvestyle="curved"):
+        operations_user = User("Operations User")
+        with Cluster("UI"):
+            with Cluster("Webserver"):
+                webservers = Custom("Webservers", PYTHON_MULTI_PROCESS_LOGO.as_posix())

Review Comment:
   ```suggestion
                   webservers = Custom("Webserver(s)", PYTHON_MULTI_PROCESS_LOGO.as_posix())
   ```
   
   Would write `(s)` to indicate it can be one or more



##########
dev/diagrams/README.md:
##########
@@ -0,0 +1,26 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+In order to regenerate the diagrams you need to:
+
+* Have [Graphviz](https://graphviz.org/) installed on your system
+* Create and activate a virtual environment of yours

Review Comment:
   We have docs for setting up a virtualenv right? Let's link to those



##########
dev/diagrams/generate_airflow_architecture_diagram.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from diagrams.onprem.database import PostgreSQL
+from diagrams.programming.flowchart import MultipleDocuments
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+IMAGE_NAME = "basic_airflow_architecture_diagram.png"
+
+LOCAL_DIR = Path(__file__).parent
+AIRFLOW_SOURCES_ROOT = Path(__file__).parent.parent.parent
+DOCS_IMAGES_DIR = AIRFLOW_SOURCES_ROOT / "docs" / "apache-airflow" / "img"
+
+PYTHON_MULTI_PROCESS_LOGO = AIRFLOW_SOURCES_ROOT / "dev" / "images" / "python_multiprocess_logo.png"
+
+
+def generate_basic_airflow_diagram():
+    with Diagram(name="Basic Airflow Architecture Diagram", show=False, direction="TB", curvestyle="curved"):
+        operations_user = User("Operations User")
+        with Cluster("UI"):
+            with Cluster("Webserver"):
+                webservers = Custom("Webservers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        operations_user >> Edge(color="brown", style="dotted", reverse=True) >> webservers
+
+        with Cluster("Scheduling"):
+            with Cluster("Scheduler"):
+                scheduler = Custom("Scheduler", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        metadata_db = PostgreSQL("Metadata DB")
+
+        dag_author = User("DAG Author")
+        dag_files = MultipleDocuments("DAG files")
+
+        dag_author >> Edge(color="black", style="solid", reverse=False) >> dag_files
+
+        with Cluster("Execution"):
+            with Cluster("Workers"):
+                workers = Custom("Workers", PYTHON_MULTI_PROCESS_LOGO.as_posix())

Review Comment:
   ```suggestion
                   workers = Custom("Worker(s)", PYTHON_MULTI_PROCESS_LOGO.as_posix())
   ```



##########
dev/diagrams/generate_airflow_architecture_diagram.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from diagrams.onprem.database import PostgreSQL
+from diagrams.programming.flowchart import MultipleDocuments
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+IMAGE_NAME = "basic_airflow_architecture_diagram.png"
+
+LOCAL_DIR = Path(__file__).parent
+AIRFLOW_SOURCES_ROOT = Path(__file__).parent.parent.parent
+DOCS_IMAGES_DIR = AIRFLOW_SOURCES_ROOT / "docs" / "apache-airflow" / "img"
+
+PYTHON_MULTI_PROCESS_LOGO = AIRFLOW_SOURCES_ROOT / "dev" / "images" / "python_multiprocess_logo.png"
+
+
+def generate_basic_airflow_diagram():
+    with Diagram(name="Basic Airflow Architecture Diagram", show=False, direction="TB", curvestyle="curved"):
+        operations_user = User("Operations User")
+        with Cluster("UI"):
+            with Cluster("Webserver"):
+                webservers = Custom("Webservers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        operations_user >> Edge(color="brown", style="dotted", reverse=True) >> webservers
+
+        with Cluster("Scheduling"):
+            with Cluster("Scheduler"):
+                scheduler = Custom("Scheduler", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        metadata_db = PostgreSQL("Metadata DB")
+
+        dag_author = User("DAG Author")
+        dag_files = MultipleDocuments("DAG files")
+
+        dag_author >> Edge(color="black", style="solid", reverse=False) >> dag_files
+
+        with Cluster("Execution"):
+            with Cluster("Workers"):
+                workers = Custom("Workers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+            with Cluster("Triggerers"):
+                triggerer = Custom("Triggerers", PYTHON_MULTI_PROCESS_LOGO.as_posix())

Review Comment:
   ```suggestion
                   triggerer = Custom("Triggerer(s)", PYTHON_MULTI_PROCESS_LOGO.as_posix())
   ```



##########
dev/diagrams/generate_airflow_architecture_diagram.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from diagrams.onprem.database import PostgreSQL
+from diagrams.programming.flowchart import MultipleDocuments
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+IMAGE_NAME = "basic_airflow_architecture_diagram.png"
+
+LOCAL_DIR = Path(__file__).parent
+AIRFLOW_SOURCES_ROOT = Path(__file__).parent.parent.parent
+DOCS_IMAGES_DIR = AIRFLOW_SOURCES_ROOT / "docs" / "apache-airflow" / "img"
+
+PYTHON_MULTI_PROCESS_LOGO = AIRFLOW_SOURCES_ROOT / "dev" / "images" / "python_multiprocess_logo.png"
+
+
+def generate_basic_airflow_diagram():
+    with Diagram(name="Basic Airflow Architecture Diagram", show=False, direction="TB", curvestyle="curved"):
+        operations_user = User("Operations User")
+        with Cluster("UI"):
+            with Cluster("Webserver"):
+                webservers = Custom("Webservers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        operations_user >> Edge(color="brown", style="dotted", reverse=True) >> webservers
+
+        with Cluster("Scheduling"):
+            with Cluster("Scheduler"):
+                scheduler = Custom("Scheduler", PYTHON_MULTI_PROCESS_LOGO.as_posix())

Review Comment:
   ```suggestion
                   scheduler = Custom("Scheduler(s)", PYTHON_MULTI_PROCESS_LOGO.as_posix())
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36035:
URL: https://github.com/apache/airflow/pull/36035#issuecomment-1837585007

   > Few minor comments + for a basic diagram I'd show only one user in the diagram, with an arrow to both the DAG files & webserver.
   
   I thought about it, and I think It's good to mention DAG Authors and Ops Users separately. They are part of our security model and I think it would be great to keep them separated here - becuse essentially they are different users.  I think there are a lot of misconceptions on the "users" accessing the DAGs and UI to be the same - but in fact all teh security mechanism and even often actually UI users are often separate ones. .. 
   
   I will have some more iterations on that - and I think we should rewrite more of our graphs (and I am planning to use the tool to add diagrams for the security model of ours (and later for multi-tenancy), so maybe it's indeed a good idea to keep one user and link to more "Complex" variants of the architecture separately ? WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36035:
URL: https://github.com/apache/airflow/pull/36035#issuecomment-1837604071

   I introduced two diagrams now - one basic, and one with standalond DAG file processor. While it is not yet fullly - mutlitenant, this already gives some good properties (like scheduler not having accesss DAG files at all) and having this picture described now is a good idea - and reflecting the current architecture.
   
   I also added a "dashed" line showing "executors" -> link between scheduler and workers and it looks better in Right to Lefte form - it also shows nicely the progression of things that happen with the tasks - scheduler being on the left of workers and triggerers.
   
   I converted the script to be entirely in pre-commit and added hash check so that it will not be running unnecessarily even in CI with `--all-files`. This way we will not recreate the images when not needed - only when the diagram sources change.
   
   I left two types of users for now. - I still think it is a good idea even for "basic" diagram.
   
   I also hope we will regenerate more diagrams using the same approach - celery , kubernetes, logging etc. - they will be so much 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36035:
URL: https://github.com/apache/airflow/pull/36035#discussion_r1413729110


##########
docs/apache-airflow/security/security_model.rst:
##########
@@ -32,8 +29,23 @@ security reports are handled by the security team of Airflow, head to
 Airflow security model - user types
 -----------------------------------
 
-The Airflow security model involves different types of users with
-varying access and capabilities:
+The Airflow security model involves different types of users with varying access and capabilities:
+
+While - in smaller installations - all the actions related to Airflow can be performed by a single user,
+in larger installations it is apparent that there different responsibilities, roles and
+capabilities that need to be separated.
+
+This is why Airflow supports the following user types:

Review Comment:
   ```suggestion
   This is why Airflow has the following user types:
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36035:
URL: https://github.com/apache/airflow/pull/36035#issuecomment-1838444557

   cc: @mhenc  @vincbeck  -> I think this is very closely reflecting the "trusted" / "untrusted" split we were always using when it comes to AIP-44. 
   
   The nice thing about it is when we get to Internal-API prime time and introduction, the 2nd diagram will become way simpler - because it will get just an "Internal API" shielding the left side of the graph from the Database (outside of the "DAG Execution" zone.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36035:
URL: https://github.com/apache/airflow/pull/36035#issuecomment-1838438374

   I came up with much nicer layout . I think it's very close to what I had in mind.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36035:
URL: https://github.com/apache/airflow/pull/36035#issuecomment-1838243276

   Ok. I added a few more touches and the "DAG file processor" case is now much nicer and cleaner shows what I wanted to show - separation between the part where DAG files are actually parsed and executed and when they are not.
   
   In this case also separation betwen the Users is much more apparent - showing that the UI user has no influence on Arbitrary DAG code execution while the DAG author does not.
   
   For me this is really first stepping stone/diagram that we will explain in the future for multi-tenancy architecture (which will mostly be showing how you can build  Airfflow from it's building block in order to achieve multi-tenant architecture if you really want. 
   
   So I think it's worth to gradually introduce this architecture (and link to the architecture from our Security Model which describes the details about those different types of users and their capabilities. I'v also added links between the architecture and security model involved, as I think this is a great way to educate the users on security implications of the architecture
   they chose. 
   
   WDYT @BasPH ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36035:
URL: https://github.com/apache/airflow/pull/36035#discussion_r1413208446


##########
dev/diagrams/generate_airflow_architecture_diagram.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from diagrams.onprem.database import PostgreSQL
+from diagrams.programming.flowchart import MultipleDocuments
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+IMAGE_NAME = "basic_airflow_architecture_diagram.png"
+
+LOCAL_DIR = Path(__file__).parent
+AIRFLOW_SOURCES_ROOT = Path(__file__).parent.parent.parent
+DOCS_IMAGES_DIR = AIRFLOW_SOURCES_ROOT / "docs" / "apache-airflow" / "img"
+
+PYTHON_MULTI_PROCESS_LOGO = AIRFLOW_SOURCES_ROOT / "dev" / "images" / "python_multiprocess_logo.png"
+
+
+def generate_basic_airflow_diagram():
+    with Diagram(name="Basic Airflow Architecture Diagram", show=False, direction="TB", curvestyle="curved"):
+        operations_user = User("Operations User")
+        with Cluster("UI"):
+            with Cluster("Webserver"):
+                webservers = Custom("Webservers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        operations_user >> Edge(color="brown", style="dotted", reverse=True) >> webservers
+
+        with Cluster("Scheduling"):
+            with Cluster("Scheduler"):
+                scheduler = Custom("Scheduler", PYTHON_MULTI_PROCESS_LOGO.as_posix())

Review Comment:
   Done



##########
dev/diagrams/generate_airflow_architecture_diagram.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from diagrams.onprem.database import PostgreSQL
+from diagrams.programming.flowchart import MultipleDocuments
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+IMAGE_NAME = "basic_airflow_architecture_diagram.png"
+
+LOCAL_DIR = Path(__file__).parent
+AIRFLOW_SOURCES_ROOT = Path(__file__).parent.parent.parent
+DOCS_IMAGES_DIR = AIRFLOW_SOURCES_ROOT / "docs" / "apache-airflow" / "img"
+
+PYTHON_MULTI_PROCESS_LOGO = AIRFLOW_SOURCES_ROOT / "dev" / "images" / "python_multiprocess_logo.png"
+
+
+def generate_basic_airflow_diagram():
+    with Diagram(name="Basic Airflow Architecture Diagram", show=False, direction="TB", curvestyle="curved"):
+        operations_user = User("Operations User")
+        with Cluster("UI"):
+            with Cluster("Webserver"):
+                webservers = Custom("Webservers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        operations_user >> Edge(color="brown", style="dotted", reverse=True) >> webservers
+
+        with Cluster("Scheduling"):
+            with Cluster("Scheduler"):
+                scheduler = Custom("Scheduler", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        metadata_db = PostgreSQL("Metadata DB")
+
+        dag_author = User("DAG Author")
+        dag_files = MultipleDocuments("DAG files")
+
+        dag_author >> Edge(color="black", style="solid", reverse=False) >> dag_files
+
+        with Cluster("Execution"):
+            with Cluster("Workers"):
+                workers = Custom("Workers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+            with Cluster("Triggerers"):
+                triggerer = Custom("Triggerers", PYTHON_MULTI_PROCESS_LOGO.as_posix())

Review Comment:
   Done



##########
dev/diagrams/generate_airflow_architecture_diagram.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+
+from diagrams import Cluster, Diagram, Edge
+from diagrams.custom import Custom
+from diagrams.onprem.client import User
+from diagrams.onprem.database import PostgreSQL
+from diagrams.programming.flowchart import MultipleDocuments
+from rich.console import Console
+
+console = Console(width=400, color_system="standard")
+
+IMAGE_NAME = "basic_airflow_architecture_diagram.png"
+
+LOCAL_DIR = Path(__file__).parent
+AIRFLOW_SOURCES_ROOT = Path(__file__).parent.parent.parent
+DOCS_IMAGES_DIR = AIRFLOW_SOURCES_ROOT / "docs" / "apache-airflow" / "img"
+
+PYTHON_MULTI_PROCESS_LOGO = AIRFLOW_SOURCES_ROOT / "dev" / "images" / "python_multiprocess_logo.png"
+
+
+def generate_basic_airflow_diagram():
+    with Diagram(name="Basic Airflow Architecture Diagram", show=False, direction="TB", curvestyle="curved"):
+        operations_user = User("Operations User")
+        with Cluster("UI"):
+            with Cluster("Webserver"):
+                webservers = Custom("Webservers", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        operations_user >> Edge(color="brown", style="dotted", reverse=True) >> webservers
+
+        with Cluster("Scheduling"):
+            with Cluster("Scheduler"):
+                scheduler = Custom("Scheduler", PYTHON_MULTI_PROCESS_LOGO.as_posix())
+
+        metadata_db = PostgreSQL("Metadata DB")
+
+        dag_author = User("DAG Author")
+        dag_files = MultipleDocuments("DAG files")
+
+        dag_author >> Edge(color="black", style="solid", reverse=False) >> dag_files
+
+        with Cluster("Execution"):
+            with Cluster("Workers"):
+                workers = Custom("Workers", PYTHON_MULTI_PROCESS_LOGO.as_posix())

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36035:
URL: https://github.com/apache/airflow/pull/36035#discussion_r1413197799


##########
dev/diagrams/README.md:
##########
@@ -0,0 +1,26 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+In order to regenerate the diagrams you need to:
+
+* Have [Graphviz](https://graphviz.org/) installed on your system
+* Create and activate a virtual environment of yours

Review Comment:
   Actually  I thought a bit on that and I will move it to pre-commit, i need to add a bit more checks there (not sure if the generation is reproducible so I will likely need to add some hash calculation - but it will be better because it will not require to setup the venve at all (will be handled by pre-commit automatically)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36035:
URL: https://github.com/apache/airflow/pull/36035#discussion_r1413197864


##########
dev/diagrams/pyproject.toml:
##########
@@ -0,0 +1,25 @@
+[build-system]
+requires = [
+  "setuptools",
+  "wheel",
+]
+
+[project]
+name = "generate-airflow-architecture"
+version = "0.1.0"
+description = "Generate airflow architecture"
+authors = [{ name = "Apache Airflow PMC", email = "dev@airflow.apache.org" }]
+requires-python = ">=3.8"
+classifiers = [
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+    "diagrams",
+    "rich"
+]
+scripts.generate_airflow_architecture = "generate_airflow_architectire:cli"

Review Comment:
   Yeah. I will remove it eventually when I move it to pre-commit :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36035:
URL: https://github.com/apache/airflow/pull/36035#issuecomment-1838344084

   cc: @feluelle  -> this is also result of what we talked about many months ago - the inspiration came from https://github.com/feluelle/airflow-diagrams :) .. I hope we can convert all the diagrams we have in Airflow to use it (and need a bit more familiarity with manipulating attributes of the nodes and edges to help graphviz to come out with a bit better layouts so I hope we can tap into your experience there :D 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Replace architecture diagram of Airflow with diagrams-generated one [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #36035:
URL: https://github.com/apache/airflow/pull/36035


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org