You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/20 16:51:46 UTC
[GitHub] [airflow] vikramkoka opened a new pull request #10428: Add architecture diagram for basic airflow deployment
vikramkoka opened a new pull request #10428:
URL: https://github.com/apache/airflow/pull/10428
This is primarily intended for showing the architecture for a basic Airflow development setup.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michael-o commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
michael-o commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677927497
Why was I mailed? I am not a committer on this project!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feng-tao commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
feng-tao commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677925030
draw.io?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677911583
> Need some way to be able to generate the image from sources. Example here: #10380
I understand and appreciate the intent, but I believe that Mermaid is very limited by comparison to the productivity with tools such as LucidChart, which is what I used to draw this and several other Airflow architecture diagrams up.
I would be happy to share somewhere the Visio export of the architecture diagrams as well, since that be used to edit the drawing in other tools.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #10428:
URL: https://github.com/apache/airflow/pull/10428
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10428: Add architecture diagram for basic airflow deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474136295
##########
File path: docs/arch-basic.rst
##########
@@ -0,0 +1,42 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Basic airflow architecture
+==========================
+
+Primarily intended for development use, the basic airflow architecture with the Local and Sequential executors is an excellent starting point for understanding the architecture of Apache Airflow.
+
+.. image:: img/arch-diag-basic.png
Review comment:
Question. How do we edit this diagram if anything changes? I believe we should have some way to reproduce the work by any member of the community, so that we can make modification to the image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10428: Add architecture diagram for basic airflow deployment
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677780646
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itβs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better π.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474244103
##########
File path: docs/start.rst
##########
@@ -78,6 +78,38 @@ run the commands below.
--start-date 2015-01-01 \
--end-date 2015-01-02
+Basic Airflow architecture
Review comment:
But it can stay here as well. I don't have a strong opinion. It won't be a problem to move it in the future when we clean up some docs. I am happy that you wanted to describe it to us in the documentation. It's very helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #10428: Add architecture diagram for basic airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677781347
π
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677923265
>
> I would be happy to share somewhere the Visio export of the architecture diagrams as well, since that be used to edit the drawing in other tools.
Sure, but I am afraid neither Visio, nor Lucidchart can't be accepted here. Those are paid tools that have not an easy way to be replicate by other community members who cannot afford to pay for those. You might think my it's a "limting" approach, but I think it's quite the opposite - limiting people to be able to edit any of our code to those who can afford a paid version of a software is much more limiting. I think it's quite a value, that we can edit any of our resources by a free OSS tools is a value on its own.
I'd love to hear other's thoughts about it @apache/airflow-committers -committers. Is it just me?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474215185
##########
File path: docs/start.rst
##########
@@ -78,6 +78,38 @@ run the commands below.
--start-date 2015-01-01 \
--end-date 2015-01-02
+Basic Airflow architecture
Review comment:
What do you think about moving this to a separate page? This is useful not only for people who want to start using, but for anyone who wants to understand how it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677927415
> > I would be happy to share somewhere the Visio export of the architecture diagrams as well, since that be used to edit the drawing in other tools.
>
> Sure, but I am afraid neither Visio, nor Lucidchart can't be accepted here. Those are paid tools that have not an easy way to be replicate by other community members who cannot afford to pay for those. You might think my it's a "limting" approach, but I think it's quite the opposite - limiting people to be able to edit any of our code to those who can afford a paid version of a software is much more limiting. I think it's quite a value, that we can edit any of our resources by a free OSS tools is a value on its own.
>
> I'd love to hear other's thoughts about it @apache/airflow-committers -committers. Is it just me?
Wait, this is blown way out of proportion! I don't understand the issue when the source is available too. And regarding the image, I would actually prefer people using the tools they want PowerPoint, Visio or whatever they feel to create such diagrams and contribute. The point is to make the barrier as small as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474296467
##########
File path: docs/start.rst
##########
@@ -78,6 +78,38 @@ run the commands below.
--start-date 2015-01-01 \
--end-date 2015-01-02
+Basic Airflow architecture
Review comment:
@potiuk Looks like you mentioned entire Apache committers again, can we stop mentioning any list at all and just mention people by their ids, thus avoiding this issues please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474218045
##########
File path: docs/start.rst
##########
@@ -78,6 +78,38 @@ run the commands below.
--start-date 2015-01-01 \
--end-date 2015-01-02
+Basic Airflow architecture
Review comment:
Any suggestions where do you think would we should keep it?
A main reason to have it over was the new users know the basic architecture but we are open for suggestions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dennisbyrne commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
dennisbyrne commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677927316
Please remove me from the email. I'm pretty sure you have the wrong Dennis.
Dennis
On Thu, Aug 20, 2020 at 2:52 PM Tao Feng <no...@github.com> wrote:
> draw.io?
>
> β
> You are receiving this because you are on a team that was mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/pull/10428#issuecomment-677925030>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAA4BAXFI4NXAZQK3FECYJTSBWLJDANCNFSM4QGLSYOQ>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677902818
I really think we should have a way to generate the diagrams for the change. I love where the PR goes, but I think as a community we need to have a way to regenerate the image from sourcs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677923265
>
> I would be happy to share somewhere the Visio export of the architecture diagrams as well, since that be used to edit the drawing in other tools.
Sure, but I am afraid neither Visio, nor Lucidchart can't be accepted here. Those are paid tools that have not an easy way to be replicate by other community members who cannot afford to pay for those. You might think my it's a "limting" approach, but I think it's quite the opposite - limiting people to be able to edit any of our code to those who can afford a paid version of a software is much more limiting. I think it's quite a value, that we can edit any of our resources by a free OSS tools is a value on its own.
I'd love to hear other's thoughts about it @apache/apache-committers. Is it just me?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474242967
##########
File path: docs/start.rst
##########
@@ -78,6 +78,38 @@ run the commands below.
--start-date 2015-01-01 \
--end-date 2015-01-02
+Basic Airflow architecture
Review comment:
I think we can put it between UI and Concepts
<img width="256" alt="Screenshot 2020-08-20 at 22 05 32" src="https://user-images.githubusercontent.com/12058428/90820227-5d832580-e331-11ea-9126-fd481ebb8a6c.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10428: Add architecture diagram for basic airflow deployment
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474136806
##########
File path: docs/arch-basic.rst
##########
@@ -0,0 +1,42 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Basic airflow architecture
+==========================
+
+Primarily intended for development use, the basic airflow architecture with the Local and Sequential executors is an excellent starting point for understanding the architecture of Apache Airflow.
+
+.. image:: img/arch-diag-basic.png
Review comment:
So we basically need:
a) Source
b) a way how to generate it
:).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474290691
##########
File path: docs/start.rst
##########
@@ -78,6 +78,38 @@ run the commands below.
--start-date 2015-01-01 \
--end-date 2015-01-02
+Basic Airflow architecture
Review comment:
Cool, yeah let's keep it as it is for now. As we add more we can move everything to a dedicated section :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#discussion_r474290503
##########
File path: docs/arch-basic.rst
##########
@@ -0,0 +1,42 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+Basic airflow architecture
+==========================
+
+Primarily intended for development use, the basic airflow architecture with the Local and Sequential executors is an excellent starting point for understanding the architecture of Apache Airflow.
+
+.. image:: img/arch-diag-basic.png
Review comment:
Ya I was meaning to get in touch with you regarding this but went out for dinner in a restaurant :D -- anyways what I was going to say is we will upload all the architecture diagrams to https://cwiki.apache.org/confluence/display/AIRFLOW/File+lists so anyone would be able to download and recreate it easily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #10428: Add architecture diagram for basic Airflow deployment
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #10428:
URL: https://github.com/apache/airflow/pull/10428#issuecomment-677923646
> Need some way to be able to generate the image from sources. Example here: #10380
Ya I was meaning to get in touch with you regarding this but went out for dinner in a restaurant :D -- anyways what I was going to say is we will upload all the architecture diagrams to https://cwiki.apache.org/confluence/display/AIRFLOW/File+lists so anyone would be able to download and recreate it easily. I will also check the UML edit option for Lucidchart and see if it helps but I think we are good for uploading the sources to Confluence.
I have created a dedicated Lucidchart diagrams space for all the future files too https://cwiki.apache.org/confluence/display/AIRFLOW/Lucidchart+Diagrams?flashId=-672732258
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org