You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/03/06 08:23:01 UTC
[PR] Add Multi-Tenant architecture diagram [airflow]
potiuk opened a new pull request, #37931:
URL: https://github.com/apache/airflow/pull/37931
This image is not yet connected anywhere, I just want to keep it in Airflow while we are discussing "AIP-67 - Multi-tenant Deployment of Airflow components" and while we are iterating it - it will be easy to collaborate on the diagrams.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Multi-Tenant architecture diagram [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #37931:
URL: https://github.com/apache/airflow/pull/37931#issuecomment-1981252608
Also:
- Should we put the internal API component in each tenant box?
- I am a bit confused by the links between the DAG files and triggerers and workers. Do these link exist? I thought the only component using DAG files is DAG processor and then the other components just use parsed DAGs from DB. But I might be wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Multi-Tenant architecture diagram [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #37931:
URL: https://github.com/apache/airflow/pull/37931#issuecomment-1986367720
Awesome!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Multi-Tenant architecture diagram [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #37931:
URL: https://github.com/apache/airflow/pull/37931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Multi-Tenant architecture diagram [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37931:
URL: https://github.com/apache/airflow/pull/37931#issuecomment-1981389611
> Also:
>
> * Should we put the internal API component in each tenant box?
They are really "on the edge" so it's difficult to make them both in and out the tenant box - but I will try to figure out something :)
> * I am a bit confused by the links between the DAG files and triggerers and workers. Do these link exist? I thought the only component using DAG files is DAG processor and then the other components just use parsed DAGs from DB. But I might be wrong
The picture is correct.
1) Workers
Workers cannot execute the DAGs from DB because they have to execute the code (With the imports and all the other things that are in the DAG folder) - the DB contains two things there:
a) Serialized for of DAG - json (that UI uses to display the DAG and Scheduler to schedule tasks from)
b) Code of the DAG file where the DAG comes from - but this one does not contain any imported libraries, utils etc. - it's literally source code of the DAG file the DAG came from.
So not nearly enough to get workers to run tasks. You **really** need to have access to DAG folder and be able to execute any code there from the worker.
2) The Triggerers
Triggerers have to execute Triggers. And you can define Triggers not only in your operators but also you can define custom Triggers in your DAG code and Trigger shoud be able to execute them in the event loop. And for that Triggerer similarly to worker should have access to DAG folder and be able to parse the complete code (including imports of any shared code(,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Multi-Tenant architecture diagram [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #37931:
URL: https://github.com/apache/airflow/pull/37931#issuecomment-1981248192
Statics are failing du to recently merged new rule I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Multi-Tenant architecture diagram [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #37931:
URL: https://github.com/apache/airflow/pull/37931#issuecomment-1981455386
> > Also:
> >
> > * Should we put the internal API component in each tenant box?
>
> They are really "on the edge" so it's difficult to make them both in and out the tenant box - but I will try to figure out something :)
>
> > * I am a bit confused by the links between the DAG files and triggerers and workers. Do these link exist? I thought the only component using DAG files is DAG processor and then the other components just use parsed DAGs from DB. But I might be wrong
>
> The picture is correct.
>
> 1. Workers
>
> Workers cannot execute the DAGs from DB because they have to execute the code (With the imports and all the other things that are in the DAG folder) - the DB contains two things there:
>
> a) Serialized for of DAG - json (that UI uses to display the DAG and Scheduler to schedule tasks from) b) Code of the DAG file where the DAG comes from - but this one does not contain any imported libraries, utils etc. - it's literally source code of the DAG file the DAG came from.
>
> So not nearly enough to get workers to run tasks. You **really** need to have access to DAG folder and be able to execute any code there from the worker.
>
> 2. The Triggerers
>
> Triggerers have to execute Triggers. And you can define Triggers not only in your operators but also you can define custom Triggers in your DAG code and Trigger shoud be able to execute them in the event loop. And for that Triggerer similarly to worker should have access to DAG folder and be able to parse the complete code (including imports of any shared code(,
I see. Thanks for the explanations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Multi-Tenant architecture diagram [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37931:
URL: https://github.com/apache/airflow/pull/37931#issuecomment-1986227873
Should be much nicer now @vincbeck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org