You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Chao-Han Tsai <mi...@gmail.com> on 2019/05/01 04:23:18 UTC
Re: [DISCUSS] AIP-5: Remote DAG Fetcher
Hi all,
I would like to bump this thread up again to seek for more feedbacks from
you for the AIP-5:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
In short, this AIP aims to enable Airflow to fetch DAG files from remote
storage systems (git, s3,...) outside the file system local to the service.
This grant a much greater flexibility, eases implementation, and
standardizes ways to sync remote sources of DAGs with Airflow. It also
opens up new opportunities for Airflow such as exposing all versions of
DAGs to Airflow to support versioned DagRuns.
Cheers,
Chao-Han
On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai <mi...@gmail.com> wrote:
> Hi Airflowers,
>
> We had an email thread
> <https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E>
> and PR <https://github.com/apache/airflow/pull/3138> about supporting
> remote DAG fetcher in the past and there was an AIP
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Deprecated-+AIP-5+DagFetcher> about
> it. But the author seems to be not working on it anymore, so I picked it up
> and came up with a new AIP:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
>
> I would like to hear your feedback and suggestions for improvement!
>
> Chao-Han Tsai
>
>
--
Chao-Han Tsai
Re: [DISCUSS] AIP-5: Remote DAG Fetcher
Posted by James Meickle <jm...@quantopian.com.INVALID>.
I left some comments on this, and on AIP-20. I'm glad to see this moving
forward as I think it's a foundational building block, even if it won't be
so useful immediately :)
On Thu, May 2, 2019 at 5:31 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> I've left some comments on the wiki page. In short: nice idea, but I'd
> like to see a few more of the ideas expanded upon first before we vote on
> this.
>
> -ash
>
> > On 1 May 2019, at 05:23, Chao-Han Tsai <mi...@gmail.com> wrote:
> >
> > Hi all,
> >
> > I would like to bump this thread up again to seek for more feedbacks from
> > you for the AIP-5:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
> >
> > In short, this AIP aims to enable Airflow to fetch DAG files from remote
> > storage systems (git, s3,...) outside the file system local to the
> service.
> > This grant a much greater flexibility, eases implementation, and
> > standardizes ways to sync remote sources of DAGs with Airflow. It also
> > opens up new opportunities for Airflow such as exposing all versions of
> > DAGs to Airflow to support versioned DagRuns.
> >
> > Cheers,
> >
> > Chao-Han
> >
> >
> >
> > On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai <mi...@gmail.com>
> wrote:
> >
> >> Hi Airflowers,
> >>
> >> We had an email thread
> >> <
> https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E
> >
> >> and PR <https://github.com/apache/airflow/pull/3138> about supporting
> >> remote DAG fetcher in the past and there was an AIP
> >> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Deprecated-+AIP-5+DagFetcher>
> about
> >> it. But the author seems to be not working on it anymore, so I picked
> it up
> >> and came up with a new AIP:
> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
> >>
> >> I would like to hear your feedback and suggestions for improvement!
> >>
> >> Chao-Han Tsai
> >>
> >>
> >
> > --
> >
> > Chao-Han Tsai
>
>
Re: [DISCUSS] AIP-5: Remote DAG Fetcher
Posted by Ash Berlin-Taylor <as...@apache.org>.
I've left some comments on the wiki page. In short: nice idea, but I'd like to see a few more of the ideas expanded upon first before we vote on this.
-ash
> On 1 May 2019, at 05:23, Chao-Han Tsai <mi...@gmail.com> wrote:
>
> Hi all,
>
> I would like to bump this thread up again to seek for more feedbacks from
> you for the AIP-5:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
>
> In short, this AIP aims to enable Airflow to fetch DAG files from remote
> storage systems (git, s3,...) outside the file system local to the service.
> This grant a much greater flexibility, eases implementation, and
> standardizes ways to sync remote sources of DAGs with Airflow. It also
> opens up new opportunities for Airflow such as exposing all versions of
> DAGs to Airflow to support versioned DagRuns.
>
> Cheers,
>
> Chao-Han
>
>
>
> On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai <mi...@gmail.com> wrote:
>
>> Hi Airflowers,
>>
>> We had an email thread
>> <https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E>
>> and PR <https://github.com/apache/airflow/pull/3138> about supporting
>> remote DAG fetcher in the past and there was an AIP
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Deprecated-+AIP-5+DagFetcher> about
>> it. But the author seems to be not working on it anymore, so I picked it up
>> and came up with a new AIP:
>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
>>
>> I would like to hear your feedback and suggestions for improvement!
>>
>> Chao-Han Tsai
>>
>>
>
> --
>
> Chao-Han Tsai
Re: [DISCUSS] AIP-5: Remote DAG Fetcher
Posted by Tao Feng <fe...@gmail.com>.
A vote thread should be initiated which AIP get accepted if it gets 3
binding vote. There are three AIPs which been accepted so far.
On Wed, May 1, 2019 at 7:49 PM Chao-Han Tsai <mi...@gmail.com> wrote:
> Dear Airflow committers,
>
> What is the current process to get an AIP accepted/approved? I looked
> through the
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
> and
> cwiki but couldn't find it documented anywhere.
>
> Chao-Han
>
> On Tue, Apr 30, 2019 at 9:23 PM Chao-Han Tsai <mi...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I would like to bump this thread up again to seek for more feedbacks from
> > you for the AIP-5:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
> >
> > In short, this AIP aims to enable Airflow to fetch DAG files from remote
> > storage systems (git, s3,...) outside the file system local to the
> service.
> > This grant a much greater flexibility, eases implementation, and
> > standardizes ways to sync remote sources of DAGs with Airflow. It also
> > opens up new opportunities for Airflow such as exposing all versions of
> > DAGs to Airflow to support versioned DagRuns.
> >
> > Cheers,
> >
> > Chao-Han
> >
> >
> >
> > On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai <mi...@gmail.com>
> > wrote:
> >
> >> Hi Airflowers,
> >>
> >> We had an email thread
> >> <
> https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E
> >
> >> and PR <https://github.com/apache/airflow/pull/3138> about supporting
> >> remote DAG fetcher in the past and there was an AIP
> >> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Deprecated-+AIP-5+DagFetcher>
> about
> >> it. But the author seems to be not working on it anymore, so I picked
> it up
> >> and came up with a new AIP:
> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
> >>
> >> I would like to hear your feedback and suggestions for improvement!
> >>
> >> Chao-Han Tsai
> >>
> >>
> >
> > --
> >
> > Chao-Han Tsai
> >
> >
>
> --
>
> Chao-Han Tsai
>
Re: [DISCUSS] AIP-5: Remote DAG Fetcher
Posted by Chao-Han Tsai <mi...@gmail.com>.
Dear Airflow committers,
What is the current process to get an AIP accepted/approved? I looked
through the
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
and
cwiki but couldn't find it documented anywhere.
Chao-Han
On Tue, Apr 30, 2019 at 9:23 PM Chao-Han Tsai <mi...@gmail.com> wrote:
> Hi all,
>
> I would like to bump this thread up again to seek for more feedbacks from
> you for the AIP-5:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
>
> In short, this AIP aims to enable Airflow to fetch DAG files from remote
> storage systems (git, s3,...) outside the file system local to the service.
> This grant a much greater flexibility, eases implementation, and
> standardizes ways to sync remote sources of DAGs with Airflow. It also
> opens up new opportunities for Airflow such as exposing all versions of
> DAGs to Airflow to support versioned DagRuns.
>
> Cheers,
>
> Chao-Han
>
>
>
> On Sat, Mar 23, 2019 at 9:56 PM Chao-Han Tsai <mi...@gmail.com>
> wrote:
>
>> Hi Airflowers,
>>
>> We had an email thread
>> <https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E>
>> and PR <https://github.com/apache/airflow/pull/3138> about supporting
>> remote DAG fetcher in the past and there was an AIP
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Deprecated-+AIP-5+DagFetcher> about
>> it. But the author seems to be not working on it anymore, so I picked it up
>> and came up with a new AIP:
>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher
>>
>> I would like to hear your feedback and suggestions for improvement!
>>
>> Chao-Han Tsai
>>
>>
>
> --
>
> Chao-Han Tsai
>
>
--
Chao-Han Tsai