You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Oliveira, Niko" <on...@amazon.com.INVALID> on 2022/11/08 00:41:44 UTC
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Hey folks!
I went ahead and wrote an AIP for this proposal. It can be found here:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
Please leave any feedback here or in the Confluence comments.
Thanks for your time!
________________________________
From: Oliveira, Niko <on...@amazon.com.INVALID>
Sent: Monday, October 24, 2022 2:28 PM
To: dev@airflow.apache.org
Subject: [EXTERNAL] Proposal to Remove Executor Coupling in Core Airlfow Code Base
Hey all!
Recently I have spent some time investigating the occurrences of hardcoded Executor logic within core Airflow code and put together a mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and automatic code snippets).
I'm particularly interested to hear if folks think an AIP would be reasonable for this set of changes or if the community is fine with using Discussions alone and beginning development without an AIP.
https://github.com/apache/airflow/discussions/27241
Thanks for you time!
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by Jeambrun Pierre <pi...@gmail.com>.
Hi all,
Thank you Niko for this proposal.
I can only think of advantages for implementing it, on top of being much
cleaner.
Regards,
Pierre
On Tue 8 Nov 2022 at 19:48, Oliveira, Niko <on...@amazon.com.invalid>
wrote:
> Thanks for the review Jarek!
>
> > My only ask will be to split it into smaller, independent PRs when it
> > gets implemented, so that we can asses better the consequences of some
> > changes
>
> Yupp, totally agree. I plan to create a tracking Issue that breaks down
> the AIP into chunks (linked child Issues). Probably one issue/PR per
> coupling type mentioned in the doc.
>
> Cheers,
> Niko
> ------------------------------
> *From:* Jarek Potiuk <ja...@potiuk.com>
> *Sent:* Tuesday, November 8, 2022 3:52:15 AM
> *To:* dev@airflow.apache.org
> *Subject:* RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core
> Airlfow Code Base
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
> I really like it, and for me it's no brainer - we've been casually
> discussing that we need to do that one sooner rather than later and I
> like the attention to detail and the analysis done by Niko.
>
> My only ask will be to split it into smaller, independent PRs when it
> gets implemented, so that we can asses better the consequences of some
> changes (and whether we have to potentially account for backwards
> compatibility for users who implemented their own executors (not a
> very popular one also because of those problems this AIP is aiming to
> solve, but still we have to account for that).
>
> I personally think if there are no objections, this one is ready to
> start voting on.
>
> J.
>
> On Tue, Nov 8, 2022 at 1:42 AM Oliveira, Niko
> <on...@amazon.com.invalid> wrote:
> >
> > Hey folks!
> >
> > I went ahead and wrote an AIP for this proposal. It can be found here:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
> >
> >
> > Please leave any feedback here or in the Confluence comments.
> >
> > Thanks for your time!
> >
> > ________________________________
> > From: Oliveira, Niko <on...@amazon.com.INVALID>
> > Sent: Monday, October 24, 2022 2:28 PM
> > To: dev@airflow.apache.org
> > Subject: [EXTERNAL] Proposal to Remove Executor Coupling in Core Airlfow
> Code Base
> >
> >
> > Hey all!
> >
> > Recently I have spent some time investigating the occurrences of
> hardcoded Executor logic within core Airflow code and put together a
> mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and
> automatic code snippets).
> >
> > I'm particularly interested to hear if folks think an AIP would be
> reasonable for this set of changes or if the community is fine with using
> Discussions alone and beginning development without an AIP.
> >
> > https://github.com/apache/airflow/discussions/27241
> >
> > Thanks for you time!
> >
>
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
Thanks for the review Jarek!
> My only ask will be to split it into smaller, independent PRs when it
> gets implemented, so that we can asses better the consequences of some
> changes
Yupp, totally agree. I plan to create a tracking Issue that breaks down the AIP into chunks (linked child Issues). Probably one issue/PR per coupling type mentioned in the doc.
Cheers,
Niko
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Tuesday, November 8, 2022 3:52:15 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core Airlfow Code Base
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
I really like it, and for me it's no brainer - we've been casually
discussing that we need to do that one sooner rather than later and I
like the attention to detail and the analysis done by Niko.
My only ask will be to split it into smaller, independent PRs when it
gets implemented, so that we can asses better the consequences of some
changes (and whether we have to potentially account for backwards
compatibility for users who implemented their own executors (not a
very popular one also because of those problems this AIP is aiming to
solve, but still we have to account for that).
I personally think if there are no objections, this one is ready to
start voting on.
J.
On Tue, Nov 8, 2022 at 1:42 AM Oliveira, Niko
<on...@amazon.com.invalid> wrote:
>
> Hey folks!
>
> I went ahead and wrote an AIP for this proposal. It can be found here:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
>
>
> Please leave any feedback here or in the Confluence comments.
>
> Thanks for your time!
>
> ________________________________
> From: Oliveira, Niko <on...@amazon.com.INVALID>
> Sent: Monday, October 24, 2022 2:28 PM
> To: dev@airflow.apache.org
> Subject: [EXTERNAL] Proposal to Remove Executor Coupling in Core Airlfow Code Base
>
>
> Hey all!
>
> Recently I have spent some time investigating the occurrences of hardcoded Executor logic within core Airflow code and put together a mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and automatic code snippets).
>
> I'm particularly interested to hear if folks think an AIP would be reasonable for this set of changes or if the community is fine with using Discussions alone and beginning development without an AIP.
>
> https://github.com/apache/airflow/discussions/27241
>
> Thanks for you time!
>
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by Jarek Potiuk <ja...@potiuk.com>.
I really like it, and for me it's no brainer - we've been casually
discussing that we need to do that one sooner rather than later and I
like the attention to detail and the analysis done by Niko.
My only ask will be to split it into smaller, independent PRs when it
gets implemented, so that we can asses better the consequences of some
changes (and whether we have to potentially account for backwards
compatibility for users who implemented their own executors (not a
very popular one also because of those problems this AIP is aiming to
solve, but still we have to account for that).
I personally think if there are no objections, this one is ready to
start voting on.
J.
On Tue, Nov 8, 2022 at 1:42 AM Oliveira, Niko
<on...@amazon.com.invalid> wrote:
>
> Hey folks!
>
> I went ahead and wrote an AIP for this proposal. It can be found here:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
>
>
> Please leave any feedback here or in the Confluence comments.
>
> Thanks for your time!
>
> ________________________________
> From: Oliveira, Niko <on...@amazon.com.INVALID>
> Sent: Monday, October 24, 2022 2:28 PM
> To: dev@airflow.apache.org
> Subject: [EXTERNAL] Proposal to Remove Executor Coupling in Core Airlfow Code Base
>
>
> Hey all!
>
> Recently I have spent some time investigating the occurrences of hardcoded Executor logic within core Airflow code and put together a mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and automatic code snippets).
>
> I'm particularly interested to hear if folks think an AIP would be reasonable for this set of changes or if the community is fine with using Discussions alone and beginning development without an AIP.
>
> https://github.com/apache/airflow/discussions/27241
>
> Thanks for you time!
>
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
Hey Robert and Utkarsh!
Thanks for your interest!
The tasks which have a smaller dev estimate (the ones marked with S, see the README for more explanation) would be better starter tasks since they're smaller in scope and should just need one maybe two PRs.
Any task that's in the Backlog column and does not have an assignee yet is up for grabs. There are two Small (S) tasks left, you each can grab one of them, just leave a comment on the task you're interested in and I'll assign it to you 😊
I'm on Slack as well, please follow-up there if you need anything I'm happy to help and coach through the changes!
Cheers,
Niko
________________________________
From: Robert Karish <ro...@gmail.com>
Sent: Monday, December 5, 2022 7:56 PM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core Airlfow Code Base
Hey Niko,
I am also a new contributor to Airflow, I’ve only made a few small contributions so far, and I am more of a user (for work) than a contributor. I would like to get involved in this AIP as well because I think this is a great improvement to the project. I’ve looked over the Github Projects page that you’ve made, but I’m not sure yet which available task would be best for me. I will look into these more closely over the next few days. I am active on our Slack channel so if you have any suggestions on a good starter task for this AIP that I could tackle in my spare time let me know on there.
Best,
Robert
On Mon, Dec 5, 2022 at 10:21 PM Utkarsh Sharma <ut...@astronomer.io.invalid> wrote:
Hi Niko,
I'm new to airflow's codebase, but would very much like to work with you on this AIP. Please let me know how can I help you.
Thanks,
Utkarsh Sharma
On Tue, Dec 6, 2022 at 1:58 AM Jarek Potiuk <ja...@potiuk.com>> wrote:
Cool!
On Mon, Dec 5, 2022 at 9:16 PM Oliveira, Niko <on...@amazon.com.invalid> wrote:
Hey folks!
As a follow-up, if you're interested in following along with this project or even taking some tasks, I've created a dashboard using Github's new Projects tool. You can see the backlog of tasks, who's assigned, the estimated size and priority of the task and it's current state: https://github.com/orgs/apache/projects/162/views/7?layout=board
NOTE: See the README (button in the top right corner) for a more detailed explanation of the priority and estimate fields (it's frustrating how hidden the README is for this Projects tool...)
This will be a live board, new tasks will be added as they come up, but the general skeleton is there.
Cheers,
Niko
________________________________
From: Oliveira, Niko <on...@amazon.com.INVALID>
Sent: Monday, November 7, 2022 4:41:44 PM
To: dev@airflow.apache.org<ma...@airflow.apache.org>
Subject: RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core Airlfow Code Base
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hey folks!
I went ahead and wrote an AIP for this proposal. It can be found here:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
Please leave any feedback here or in the Confluence comments.
Thanks for your time!
________________________________
From: Oliveira, Niko <on...@amazon.com.INVALID>
Sent: Monday, October 24, 2022 2:28 PM
To: dev@airflow.apache.org<ma...@airflow.apache.org>
Subject: [EXTERNAL] Proposal to Remove Executor Coupling in Core Airlfow Code Base
Hey all!
Recently I have spent some time investigating the occurrences of hardcoded Executor logic within core Airflow code and put together a mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and automatic code snippets).
I'm particularly interested to hear if folks think an AIP would be reasonable for this set of changes or if the community is fine with using Discussions alone and beginning development without an AIP.
https://github.com/apache/airflow/discussions/27241
Thanks for you time!
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by Robert Karish <ro...@gmail.com>.
Hey Niko,
I am also a new contributor to Airflow, I’ve only made a few small
contributions so far, and I am more of a user (for work) than a
contributor. I would like to get involved in this AIP as well because I
think this is a great improvement to the project. I’ve looked over the
Github Projects page that you’ve made, but I’m not sure yet which available
task would be best for me. I will look into these more closely over the
next few days. I am active on our Slack channel so if you have any
suggestions on a good starter task for this AIP that I could tackle in my
spare time let me know on there.
Best,
Robert
On Mon, Dec 5, 2022 at 10:21 PM Utkarsh Sharma
<ut...@astronomer.io.invalid> wrote:
> Hi Niko,
>
> I'm new to airflow's codebase, but would very much like to work with you
> on this AIP. Please let me know how can I help you.
>
> Thanks,
> Utkarsh Sharma
>
>
> On Tue, Dec 6, 2022 at 1:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Cool!
>>
>> On Mon, Dec 5, 2022 at 9:16 PM Oliveira, Niko <on...@amazon.com.invalid>
>> wrote:
>>
>>> Hey folks!
>>>
>>> As a follow-up, if you're interested in following along with this
>>> project or even taking some tasks, I've created a dashboard using Github's
>>> new Projects tool. You can see the backlog of tasks, who's assigned, the
>>> estimated size and priority of the task and it's current state:
>>> https://github.com/orgs/apache/projects/162/views/7?layout=board
>>>
>>> NOTE: See the README (button in the top right corner) for a more
>>> detailed explanation of the priority and estimate fields (it's frustrating
>>> how hidden the README is for this Projects tool...)
>>>
>>>
>>> This will be a live board, new tasks will be added as they come up, but
>>> the general skeleton is there.
>>>
>>>
>>> Cheers,
>>> Niko
>>>
>>> ------------------------------
>>> *From:* Oliveira, Niko <on...@amazon.com.INVALID>
>>> *Sent:* Monday, November 7, 2022 4:41:44 PM
>>> *To:* dev@airflow.apache.org
>>> *Subject:* RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core
>>> Airlfow Code Base
>>>
>>>
>>> *CAUTION*: This email originated from outside of the organization. Do
>>> not click links or open attachments unless you can confirm the sender and
>>> know the content is safe.
>>>
>>> Hey folks!
>>>
>>> I went ahead and wrote an AIP for this proposal. It can be found here:
>>>
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
>>>
>>> Please leave any feedback here or in the Confluence comments.
>>>
>>> Thanks for your time!
>>>
>>> ------------------------------
>>> *From:* Oliveira, Niko <on...@amazon.com.INVALID>
>>> *Sent:* Monday, October 24, 2022 2:28 PM
>>> *To:* dev@airflow.apache.org
>>> *Subject:* [EXTERNAL] Proposal to Remove Executor Coupling in Core
>>> Airlfow Code Base
>>>
>>>
>>> Hey all!
>>>
>>> Recently I have spent some time investigating the occurrences of
>>> hardcoded Executor logic within core Airflow code and put together a
>>> mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and
>>> automatic code snippets).
>>>
>>> I'm particularly interested to hear if folks think an AIP would be
>>> reasonable for this set of changes or if the community is fine with using
>>> Discussions alone and beginning development without an AIP.
>>>
>>> https://github.com/apache/airflow/discussions/27241
>>>
>>> Thanks for you time!
>>>
>>>
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by Utkarsh Sharma <ut...@astronomer.io.INVALID>.
Hi Niko,
I'm new to airflow's codebase, but would very much like to work with you on
this AIP. Please let me know how can I help you.
Thanks,
Utkarsh Sharma
On Tue, Dec 6, 2022 at 1:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Cool!
>
> On Mon, Dec 5, 2022 at 9:16 PM Oliveira, Niko <on...@amazon.com.invalid>
> wrote:
>
>> Hey folks!
>>
>> As a follow-up, if you're interested in following along with this project
>> or even taking some tasks, I've created a dashboard using Github's new
>> Projects tool. You can see the backlog of tasks, who's assigned, the
>> estimated size and priority of the task and it's current state:
>> https://github.com/orgs/apache/projects/162/views/7?layout=board
>>
>> NOTE: See the README (button in the top right corner) for a more detailed
>> explanation of the priority and estimate fields (it's frustrating how
>> hidden the README is for this Projects tool...)
>>
>>
>> This will be a live board, new tasks will be added as they come up, but
>> the general skeleton is there.
>>
>>
>> Cheers,
>> Niko
>>
>> ------------------------------
>> *From:* Oliveira, Niko <on...@amazon.com.INVALID>
>> *Sent:* Monday, November 7, 2022 4:41:44 PM
>> *To:* dev@airflow.apache.org
>> *Subject:* RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core
>> Airlfow Code Base
>>
>>
>> *CAUTION*: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>> Hey folks!
>>
>> I went ahead and wrote an AIP for this proposal. It can be found here:
>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
>>
>> Please leave any feedback here or in the Confluence comments.
>>
>> Thanks for your time!
>>
>> ------------------------------
>> *From:* Oliveira, Niko <on...@amazon.com.INVALID>
>> *Sent:* Monday, October 24, 2022 2:28 PM
>> *To:* dev@airflow.apache.org
>> *Subject:* [EXTERNAL] Proposal to Remove Executor Coupling in Core
>> Airlfow Code Base
>>
>>
>> Hey all!
>>
>> Recently I have spent some time investigating the occurrences of
>> hardcoded Executor logic within core Airflow code and put together a
>> mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and
>> automatic code snippets).
>>
>> I'm particularly interested to hear if folks think an AIP would be
>> reasonable for this set of changes or if the community is fine with using
>> Discussions alone and beginning development without an AIP.
>>
>> https://github.com/apache/airflow/discussions/27241
>>
>> Thanks for you time!
>>
>>
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by Jarek Potiuk <ja...@potiuk.com>.
Cool!
On Mon, Dec 5, 2022 at 9:16 PM Oliveira, Niko <on...@amazon.com.invalid>
wrote:
> Hey folks!
>
> As a follow-up, if you're interested in following along with this project
> or even taking some tasks, I've created a dashboard using Github's new
> Projects tool. You can see the backlog of tasks, who's assigned, the
> estimated size and priority of the task and it's current state:
> https://github.com/orgs/apache/projects/162/views/7?layout=board
>
> NOTE: See the README (button in the top right corner) for a more detailed
> explanation of the priority and estimate fields (it's frustrating how
> hidden the README is for this Projects tool...)
>
>
> This will be a live board, new tasks will be added as they come up, but
> the general skeleton is there.
>
>
> Cheers,
> Niko
>
> ------------------------------
> *From:* Oliveira, Niko <on...@amazon.com.INVALID>
> *Sent:* Monday, November 7, 2022 4:41:44 PM
> *To:* dev@airflow.apache.org
> *Subject:* RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core
> Airlfow Code Base
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
> Hey folks!
>
> I went ahead and wrote an AIP for this proposal. It can be found here:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
>
> Please leave any feedback here or in the Confluence comments.
>
> Thanks for your time!
>
> ------------------------------
> *From:* Oliveira, Niko <on...@amazon.com.INVALID>
> *Sent:* Monday, October 24, 2022 2:28 PM
> *To:* dev@airflow.apache.org
> *Subject:* [EXTERNAL] Proposal to Remove Executor Coupling in Core
> Airlfow Code Base
>
>
> Hey all!
>
> Recently I have spent some time investigating the occurrences of hardcoded
> Executor logic within core Airflow code and put together a mini-AIP of
> sorts on Github Discussions (it was nice to use GH markdown and automatic
> code snippets).
>
> I'm particularly interested to hear if folks think an AIP would be
> reasonable for this set of changes or if the community is fine with using
> Discussions alone and beginning development without an AIP.
>
> https://github.com/apache/airflow/discussions/27241
>
> Thanks for you time!
>
>
Re: Proposal to Remove Executor Coupling in Core Airlfow Code Base
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
Hey folks!
As a follow-up, if you're interested in following along with this project or even taking some tasks, I've created a dashboard using Github's new Projects tool. You can see the backlog of tasks, who's assigned, the estimated size and priority of the task and it's current state: https://github.com/orgs/apache/projects/162/views/7?layout=board
NOTE: See the README (button in the top right corner) for a more detailed explanation of the priority and estimate fields (it's frustrating how hidden the README is for this Projects tool...)
This will be a live board, new tasks will be added as they come up, but the general skeleton is there.
Cheers,
Niko
________________________________
From: Oliveira, Niko <on...@amazon.com.INVALID>
Sent: Monday, November 7, 2022 4:41:44 PM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL]Proposal to Remove Executor Coupling in Core Airlfow Code Base
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hey folks!
I went ahead and wrote an AIP for this proposal. It can be found here:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airlfow
Please leave any feedback here or in the Confluence comments.
Thanks for your time!
________________________________
From: Oliveira, Niko <on...@amazon.com.INVALID>
Sent: Monday, October 24, 2022 2:28 PM
To: dev@airflow.apache.org
Subject: [EXTERNAL] Proposal to Remove Executor Coupling in Core Airlfow Code Base
Hey all!
Recently I have spent some time investigating the occurrences of hardcoded Executor logic within core Airflow code and put together a mini-AIP of sorts on Github Discussions (it was nice to use GH markdown and automatic code snippets).
I'm particularly interested to hear if folks think an AIP would be reasonable for this set of changes or if the community is fine with using Discussions alone and beginning development without an AIP.
https://github.com/apache/airflow/discussions/27241
Thanks for you time!