You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Igor Kholopov <ik...@google.com.INVALID> on 2024/07/30 14:08:53 UTC
[DISCUSS] Extendable parsing controls (a.k.a. Pluggable Parsers)
Hi all,
I've prepared a proposal of the AIP aiming to increase the flexibility and
structure up the DAG processing/ingestion in Airflow 3.0+. This
proposal largely builds on the accepted AIP-66 & AIP-72. It strives to
unblock paths for faster development in remote environments, precise
control of how changes to DAGs are deployed and for DAG authoring beyond
Python files.
For now the draft is available here
https://docs.google.com/document/d/1jgbxg2Ryoui4AhZujEMYkFRFErH3N1MZeCTsqSdmHVI/edit?usp=sharing,
I will move it to cwiki once I have permissions (would appreciate if
someone gave me access to create a page).
Please let me know what you think, looking forward to your feedback.
Kind regards,
Igor
Re: [DISCUSS] Extendable parsing controls (a.k.a. Pluggable Parsers)
Posted by Igor Kholopov <ik...@google.com.INVALID>.
Hello everyone,
As Airflow 3.0 is getting to a final shape I've made a major revision of
AIP-85
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-85+Extendable+DAG+parsing+controls>
based on Airflow 3.0 implementation and PoC that I've prepared at
https://github.com/apache/airflow/pull/45684. PTAL, I hope to address
feedback and push it out for vote in a week or so.
Regards,
Igor
On Fri, Aug 2, 2024 at 1:08 AM Igor Kholopov <ik...@google.com> wrote:
> Small update: I've created the page on wiki:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-85+Extendable+DAG+parsing+controls.
> I suggest moving inline-comments there instead of the originally linked doc.
>
> On Wed, Jul 31, 2024 at 1:57 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Hi Igor,
>>
>> I have added permissions to your cWiki username: ikholopov
>>
>> I am interested in this topic but my feedback will be a bit delayed since
>> I
>> am focussing on reviewing 3.0 items first.
>>
>> Regards,
>> Kaxil
>>
>> On Tue, 30 Jul 2024 at 15:09, Igor Kholopov <ikholopov@google.com.invalid
>> >
>> wrote:
>>
>> > Hi all,
>> >
>> > I've prepared a proposal of the AIP aiming to increase the flexibility
>> and
>> > structure up the DAG processing/ingestion in Airflow 3.0+. This
>> > proposal largely builds on the accepted AIP-66 & AIP-72. It strives to
>> > unblock paths for faster development in remote environments, precise
>> > control of how changes to DAGs are deployed and for DAG authoring beyond
>> > Python files.
>> >
>> > For now the draft is available here
>> >
>> >
>> https://docs.google.com/document/d/1jgbxg2Ryoui4AhZujEMYkFRFErH3N1MZeCTsqSdmHVI/edit?usp=sharing
>> > ,
>> > I will move it to cwiki once I have permissions (would appreciate if
>> > someone gave me access to create a page).
>> >
>> > Please let me know what you think, looking forward to your feedback.
>> >
>> > Kind regards,
>> > Igor
>> >
>>
>
Re: [DISCUSS] Extendable parsing controls (a.k.a. Pluggable Parsers)
Posted by Igor Kholopov <ik...@google.com.INVALID>.
Small update: I've created the page on wiki:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-85+Extendable+DAG+parsing+controls.
I suggest moving inline-comments there instead of the originally linked doc.
On Wed, Jul 31, 2024 at 1:57 AM Kaxil Naik <ka...@gmail.com> wrote:
> Hi Igor,
>
> I have added permissions to your cWiki username: ikholopov
>
> I am interested in this topic but my feedback will be a bit delayed since I
> am focussing on reviewing 3.0 items first.
>
> Regards,
> Kaxil
>
> On Tue, 30 Jul 2024 at 15:09, Igor Kholopov <ik...@google.com.invalid>
> wrote:
>
> > Hi all,
> >
> > I've prepared a proposal of the AIP aiming to increase the flexibility
> and
> > structure up the DAG processing/ingestion in Airflow 3.0+. This
> > proposal largely builds on the accepted AIP-66 & AIP-72. It strives to
> > unblock paths for faster development in remote environments, precise
> > control of how changes to DAGs are deployed and for DAG authoring beyond
> > Python files.
> >
> > For now the draft is available here
> >
> >
> https://docs.google.com/document/d/1jgbxg2Ryoui4AhZujEMYkFRFErH3N1MZeCTsqSdmHVI/edit?usp=sharing
> > ,
> > I will move it to cwiki once I have permissions (would appreciate if
> > someone gave me access to create a page).
> >
> > Please let me know what you think, looking forward to your feedback.
> >
> > Kind regards,
> > Igor
> >
>
Re: [DISCUSS] Extendable parsing controls (a.k.a. Pluggable Parsers)
Posted by Kaxil Naik <ka...@gmail.com>.
Hi Igor,
I have added permissions to your cWiki username: ikholopov
I am interested in this topic but my feedback will be a bit delayed since I
am focussing on reviewing 3.0 items first.
Regards,
Kaxil
On Tue, 30 Jul 2024 at 15:09, Igor Kholopov <ik...@google.com.invalid>
wrote:
> Hi all,
>
> I've prepared a proposal of the AIP aiming to increase the flexibility and
> structure up the DAG processing/ingestion in Airflow 3.0+. This
> proposal largely builds on the accepted AIP-66 & AIP-72. It strives to
> unblock paths for faster development in remote environments, precise
> control of how changes to DAGs are deployed and for DAG authoring beyond
> Python files.
>
> For now the draft is available here
>
> https://docs.google.com/document/d/1jgbxg2Ryoui4AhZujEMYkFRFErH3N1MZeCTsqSdmHVI/edit?usp=sharing
> ,
> I will move it to cwiki once I have permissions (would appreciate if
> someone gave me access to create a page).
>
> Please let me know what you think, looking forward to your feedback.
>
> Kind regards,
> Igor
>