Feed Back on Remote Dag Fetching Proposal

[Ian Davison <Ia...@1010data.com>]

Hey Everyone,

I just created a first draft of an Airflow Improvement Proposal for Remote Dag Fetching https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher
The initial PR is located here: https://github.com/apache/incubator-airflow/pull/3138

I tried to go through and encapsulate all the various ideas discussed in the PR, but I’m sure I missed some. As well, there were a number of design decisions I felt should be further discussed by the community. I would really appreciate people’s feedback so we can start working towards a fully fleshed out design. I think the ability to remotely fetch dags, and to stop crawling a file system is the next big improvement for Airflow.

Thanks!
Ian Davison
The content of this e-mail message and any attached files transmitted with it are to be treated as confidential information and are intended solely for the use of the individual or entity to whom they are addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any reproduction, distribution, or disclosure of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.***1010data, Inc. and its affiliates ("1010data") are not responsible for any advice concerning the use of its software or services or the data manipulated by its software, except to the extent 1010data has specifically undertaken such responsibility in a validly binding contract. 1010data shall have no responsibility for any decision concerning the appropriate method of use or application of its software or services or the data therein in connection with any transaction. Any decision concerning how, where and when to use this facility remains the sole responsibility of the user. NO AGREEMENTS MAY BE ENTERED INTO WITH ANY 1010DATA COMPANY VIA EMAIL EXCHANGE REGARDLESS OF THE TITLE OF THE AUTHOR OF THE EMAILS. ALL AGREEMENTS MUST BE IN THE FORM OF A FORMAL WRITTEN AGREEMENT SIGNED BY THE CHIEF EXECUTIVE OFFICER, PRESIDENT, CHIEF OPERATING OFFICER, CHIEF FINANCIAL OFFICER OR CHIEF PEOPLE OFFICER OF 1010DATA AND AN AUTHORIZED REPRESENTATIVE OF THE OTHER PARTY.
Hey Everyone,

I just created a first draft of an Airflow Improvement Proposal for Remote Dag Fetching https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher
The initial PR is located here: https://github.com/apache/incubator-airflow/pull/3138

I tried to go through and encapsulate all the various ideas discussed in the PR, but I’m sure I missed some. As well, there were a number of design decisions I felt should be further discussed by the community. I would really appreciate people’s feedback so we can start working towards a fully fleshed out design. I think the ability to remotely fetch dags, and to stop crawling a file system is the next big improvement for Airflow.

Thanks!
Ian Davison
The content of this e-mail message and any attached files transmitted with it are to be treated as confidential information and are intended solely for the use of the individual or entity to whom they are addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any reproduction, distribution, or disclosure of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.***1010data, Inc. and its affiliates ("1010data") are not responsible for any advice concerning the use of its software or services or the data manipulated by its software, except to the extent 1010data has specifically undertaken such responsibility in a validly binding contract. 1010data shall have no responsibility for any decision concerning the appropriate method of use or application of its software or services or the data therein in connection with any transaction. Any decision concerning how, where and when to use this facility remains the sole responsibility of the user. NO AGREEMENTS MAY BE ENTERED INTO WITH ANY 1010DATA COMPANY VIA EMAIL EXCHANGE REGARDLESS OF THE TITLE OF THE AUTHOR OF THE EMAILS. ALL AGREEMENTS MUST BE IN THE FORM OF A FORMAL WRITTEN AGREEMENT SIGNED BY THE CHIEF EXECUTIVE OFFICER, PRESIDENT, CHIEF OPERATING OFFICER, CHIEF FINANCIAL OFFICER OR CHIEF PEOPLE OFFICER OF 1010DATA AND AN AUTHORIZED REPRESENTATIVE OF THE OTHER PARTY.