You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@gmail.com> on 2024/05/28 19:11:14 UTC
Airflow 3 Dev Call #1 with Proposed Agenda
Hi all,
As discussed in the previous email thread, the first dev call has been
pushed to next Tuesday (4th June 2024).
If you would like to participate in the development of Airflow 3, please
join the dev calls starting next week. The calls will be open to anyone in
the community.
*Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 PM
EST | 9 AM PST)
*One-time registration Link*:
https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
The meeting notes from the call will also be posted on the dev mailing list
and Confluence for archival purposes
at
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes.
*Proposed Agenda*:
1) Agreeing on the Principles to drive Airflow 3 development
2) Agreeing on the Guidelines that help decide if a feature should be in
Airflow 3 or not
For (1), I propose the following principles:
- Considering Airflow 3.0 for early adopters and breaking (and
removing) things. Things can be re-added as needed in upcoming minor
releases.
- Optimize to get foundational pieces in and not "let perfect be
the enemy of good"
- Smoother migration path between AF 2 & 3 especially for DAG Authors
with the existing official Airflow providers.
- Working on features that solidify Airflow as the modern Orchestrator
that also has state-of-the-art support for Data, AI & ML workloads.
- This includes improving scalability & performance of all the
Airflow components.
- Making Airflow aware of what's happening in the task to provide
better auditability, lineage & observability
- Set up the codebase for the next 3-5 years.
- Reducing matrix of supported combinations for reducing complexity
in testing & development. E.g Remove MySQL support to reduce the test matrix
- Simplifying codebase & standardize architecture (e.g
consolidating serialization methods)
- Remove deprecations
- Simplify the Learning Curve for new Airflow users
- Shift focus on Airflow 2 to stability: bug fixes + security fixes
after AF 2.10. This should continue for a longer period of time after AF 3
release
- Target a shorter cycle to release Airflow 3
- so that Airflow 2 branches for features don't diverge
- have enough time between Airflow 3 release and Airflow Summit
2025, so we can have talks about Successful migrations
For (2), I propose the following guidelines:
- Alignment with Core Principles
- Community Demand and Feedback
- Impact on Scalability and Performance
- Implementation Complexity and Maintenance
- Backward Compatibility and Migration Effort
- Workstream Ownership (can be more than one). If no one is available
to lead the workstream, the feature will be parked until a dedicated owner
is found
- For big epics, AIPs & a successful vote on the dev mailing list
Please reply if anyone has anything to add to the agenda or comment on
anything if you disagree.
Looking forward to the call.
Regards,
Kaxil
Re: Airflow 3 Dev Call #1 with Proposed Agenda
Posted by Jarek Potiuk <ja...@potiuk.com>.
Added my comments too :). Looks great
On Wed, May 29, 2024 at 12:14 PM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks Shubham and Jens, I will take a look later today.
>
> On Wed, 29 May 2024 at 03:07, Mehta, Shubham <sh...@amazon.com.invalid>
> wrote:
>
> > Kaxil - thank you for creating the wiki, setting up call invites, and
> > starting this thread. The first draft of the principles looks great. I
> have
> > commented on the wiki with some feedback and personal thoughts. I'm not
> > adding the comments on this thread to keep the wiki as the single place
> for
> > discussions and feedback.
> >
> > Thanks
> > Shubham
> >
> > On 2024-05-28, 12:25 PM, "Kaxil Naik" <kaxilnaik@apache.org <mailto:
> > kaxilnaik@apache.org>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > If the formatting of bullets was lost, check below or at
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
> > >
> >
> >
> > Proposed Agenda:
> > 1) Agreeing on the Principles to drive Airflow 3 development
> > 2) Agreeing on the Guidelines that help decide if a feature should be in
> > Airflow 3 or not
> >
> >
> >
> >
> > For (1), I propose the following principles:
> >
> >
> > - Considering Airflow 3.0 for early adopters and breaking (and removing)
> > things. Things can be re-added as needed in upcoming minor releases.
> > - Optimize to get foundational pieces in and not "let perfect be the
> enemy
> > of good"
> > - Smoother migration path between AF 2 & 3 especially for DAG Authors
> with
> > the existing official Airflow providers.
> > - Working on features that solidify Airflow as the modern Orchestrator
> > that also has state-of-the-art support for Data, AI & ML workloads.
> > - This includes improving scalability & performance of all the Airflow
> > components.
> > - Making Airflow aware of what's happening in the task to provide better
> > auditability, lineage & observability
> > - Set up the codebase for the next 3-5 years.
> > - Reducing matrix of supported combinations for reducing complexity in
> > testing & development. E.g Remove MySQL support to reduce the test matrix
> > - Simplifying codebase & standardize architecture (e.g consolidating
> > serialization methods)
> > - Remove deprecations
> > - Simplify the Learning Curve for new Airflow users
> > - Shift focus on Airflow 2 to stability: bug fixes + security fixes after
> > AF 2.10. This should continue for a longer period of time after AF 3
> release
> > - Target a shorter cycle to release Airflow 3
> > - so that Airflow 2 branches for features don't diverge
> > - have enough time between Airflow 3 release and Airflow Summit 2025, so
> > we can have talks about Successful migrations
> >
> >
> > For (2), I propose the following guidelines:
> >
> >
> > - Alignment with Core Principles
> > - Community Demand and Feedback
> > - Impact on Scalability and Performance
> > - Implementation Complexity and Maintenance
> > - Backward Compatibility and Migration Effort
> > - Workstream Ownership (can be more than one). If no one is available to
> > lead the workstream, the feature will be parked until a dedicated owner
> is
> > found
> > - For big epics, AIPs & a successful vote on the dev mailing list
> >
> >
> > Please reply if anyone has anything to add to the agenda or comment on
> > anything if you disagree.
> >
> >
> > Looking forward to the call.
> >
> >
> > Regards,
> > Kaxil
> >
> >
> > On 2024/05/28 19:11:14 Kaxil Naik wrote:
> > > Hi all,
> > >
> > > As discussed in the previous email thread, the first dev call has been
> > > pushed to next Tuesday (4th June 2024).
> > >
> > > If you would like to participate in the development of Airflow 3,
> please
> > > join the dev calls starting next week. The calls will be open to anyone
> > in
> > > the community.
> > >
> > > *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12
> PM
> > > EST | 9 AM PST)
> > > *One-time registration Link*:
> > >
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > <
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > >
> > >
> > > The meeting notes from the call will also be posted on the dev mailing
> > list
> > > and Confluence for archival purposes
> > > at
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes
> > >.
> > >
> > >
> > >
> > > *Proposed Agenda*:
> > > 1) Agreeing on the Principles to drive Airflow 3 development
> > > 2) Agreeing on the Guidelines that help decide if a feature should be
> in
> > > Airflow 3 or not
> > >
> > >
> > > For (1), I propose the following principles:
> > >
> > > - Considering Airflow 3.0 for early adopters and breaking (and
> > > removing) things. Things can be re-added as needed in upcoming minor
> > > releases.
> > > - Optimize to get foundational pieces in and not "let perfect be
> > > the enemy of good"
> > > - Smoother migration path between AF 2 & 3 especially for DAG Authors
> > > with the existing official Airflow providers.
> > > - Working on features that solidify Airflow as the modern Orchestrator
> > > that also has state-of-the-art support for Data, AI & ML workloads.
> > > - This includes improving scalability & performance of all the
> > > Airflow components.
> > > - Making Airflow aware of what's happening in the task to provide
> > > better auditability, lineage & observability
> > > - Set up the codebase for the next 3-5 years.
> > > - Reducing matrix of supported combinations for reducing complexity
> > > in testing & development. E.g Remove MySQL support to reduce the test
> > matrix
> > > - Simplifying codebase & standardize architecture (e.g
> > > consolidating serialization methods)
> > > - Remove deprecations
> > > - Simplify the Learning Curve for new Airflow users
> > > - Shift focus on Airflow 2 to stability: bug fixes + security fixes
> > > after AF 2.10. This should continue for a longer period of time after
> AF
> > 3
> > > release
> > > - Target a shorter cycle to release Airflow 3
> > > - so that Airflow 2 branches for features don't diverge
> > > - have enough time between Airflow 3 release and Airflow Summit
> > > 2025, so we can have talks about Successful migrations
> > >
> > > For (2), I propose the following guidelines:
> > >
> > > - Alignment with Core Principles
> > > - Community Demand and Feedback
> > > - Impact on Scalability and Performance
> > > - Implementation Complexity and Maintenance
> > > - Backward Compatibility and Migration Effort
> > > - Workstream Ownership (can be more than one). If no one is available
> > > to lead the workstream, the feature will be parked until a dedicated
> > owner
> > > is found
> > > - For big epics, AIPs & a successful vote on the dev mailing list
> > >
> > > Please reply if anyone has anything to add to the agenda or comment on
> > > anything if you disagree.
> > >
> > > Looking forward to the call.
> > >
> > > Regards,
> > > Kaxil
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>
> >
> >
> >
> >
> >
> >
>
Re: Airflow 3 Dev Call #1 with Proposed Agenda
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks Shubham and Jens, I will take a look later today.
On Wed, 29 May 2024 at 03:07, Mehta, Shubham <sh...@amazon.com.invalid>
wrote:
> Kaxil - thank you for creating the wiki, setting up call invites, and
> starting this thread. The first draft of the principles looks great. I have
> commented on the wiki with some feedback and personal thoughts. I'm not
> adding the comments on this thread to keep the wiki as the single place for
> discussions and feedback.
>
> Thanks
> Shubham
>
> On 2024-05-28, 12:25 PM, "Kaxil Naik" <kaxilnaik@apache.org <mailto:
> kaxilnaik@apache.org>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> If the formatting of bullets was lost, check below or at
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
> >
>
>
> Proposed Agenda:
> 1) Agreeing on the Principles to drive Airflow 3 development
> 2) Agreeing on the Guidelines that help decide if a feature should be in
> Airflow 3 or not
>
>
>
>
> For (1), I propose the following principles:
>
>
> - Considering Airflow 3.0 for early adopters and breaking (and removing)
> things. Things can be re-added as needed in upcoming minor releases.
> - Optimize to get foundational pieces in and not "let perfect be the enemy
> of good"
> - Smoother migration path between AF 2 & 3 especially for DAG Authors with
> the existing official Airflow providers.
> - Working on features that solidify Airflow as the modern Orchestrator
> that also has state-of-the-art support for Data, AI & ML workloads.
> - This includes improving scalability & performance of all the Airflow
> components.
> - Making Airflow aware of what's happening in the task to provide better
> auditability, lineage & observability
> - Set up the codebase for the next 3-5 years.
> - Reducing matrix of supported combinations for reducing complexity in
> testing & development. E.g Remove MySQL support to reduce the test matrix
> - Simplifying codebase & standardize architecture (e.g consolidating
> serialization methods)
> - Remove deprecations
> - Simplify the Learning Curve for new Airflow users
> - Shift focus on Airflow 2 to stability: bug fixes + security fixes after
> AF 2.10. This should continue for a longer period of time after AF 3 release
> - Target a shorter cycle to release Airflow 3
> - so that Airflow 2 branches for features don't diverge
> - have enough time between Airflow 3 release and Airflow Summit 2025, so
> we can have talks about Successful migrations
>
>
> For (2), I propose the following guidelines:
>
>
> - Alignment with Core Principles
> - Community Demand and Feedback
> - Impact on Scalability and Performance
> - Implementation Complexity and Maintenance
> - Backward Compatibility and Migration Effort
> - Workstream Ownership (can be more than one). If no one is available to
> lead the workstream, the feature will be parked until a dedicated owner is
> found
> - For big epics, AIPs & a successful vote on the dev mailing list
>
>
> Please reply if anyone has anything to add to the agenda or comment on
> anything if you disagree.
>
>
> Looking forward to the call.
>
>
> Regards,
> Kaxil
>
>
> On 2024/05/28 19:11:14 Kaxil Naik wrote:
> > Hi all,
> >
> > As discussed in the previous email thread, the first dev call has been
> > pushed to next Tuesday (4th June 2024).
> >
> > If you would like to participate in the development of Airflow 3, please
> > join the dev calls starting next week. The calls will be open to anyone
> in
> > the community.
> >
> > *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 PM
> > EST | 9 AM PST)
> > *One-time registration Link*:
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> <
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> >
> >
> > The meeting notes from the call will also be posted on the dev mailing
> list
> > and Confluence for archival purposes
> > at
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes
> >.
> >
> >
> >
> > *Proposed Agenda*:
> > 1) Agreeing on the Principles to drive Airflow 3 development
> > 2) Agreeing on the Guidelines that help decide if a feature should be in
> > Airflow 3 or not
> >
> >
> > For (1), I propose the following principles:
> >
> > - Considering Airflow 3.0 for early adopters and breaking (and
> > removing) things. Things can be re-added as needed in upcoming minor
> > releases.
> > - Optimize to get foundational pieces in and not "let perfect be
> > the enemy of good"
> > - Smoother migration path between AF 2 & 3 especially for DAG Authors
> > with the existing official Airflow providers.
> > - Working on features that solidify Airflow as the modern Orchestrator
> > that also has state-of-the-art support for Data, AI & ML workloads.
> > - This includes improving scalability & performance of all the
> > Airflow components.
> > - Making Airflow aware of what's happening in the task to provide
> > better auditability, lineage & observability
> > - Set up the codebase for the next 3-5 years.
> > - Reducing matrix of supported combinations for reducing complexity
> > in testing & development. E.g Remove MySQL support to reduce the test
> matrix
> > - Simplifying codebase & standardize architecture (e.g
> > consolidating serialization methods)
> > - Remove deprecations
> > - Simplify the Learning Curve for new Airflow users
> > - Shift focus on Airflow 2 to stability: bug fixes + security fixes
> > after AF 2.10. This should continue for a longer period of time after AF
> 3
> > release
> > - Target a shorter cycle to release Airflow 3
> > - so that Airflow 2 branches for features don't diverge
> > - have enough time between Airflow 3 release and Airflow Summit
> > 2025, so we can have talks about Successful migrations
> >
> > For (2), I propose the following guidelines:
> >
> > - Alignment with Core Principles
> > - Community Demand and Feedback
> > - Impact on Scalability and Performance
> > - Implementation Complexity and Maintenance
> > - Backward Compatibility and Migration Effort
> > - Workstream Ownership (can be more than one). If no one is available
> > to lead the workstream, the feature will be parked until a dedicated
> owner
> > is found
> > - For big epics, AIPs & a successful vote on the dev mailing list
> >
> > Please reply if anyone has anything to add to the agenda or comment on
> > anything if you disagree.
> >
> > Looking forward to the call.
> >
> > Regards,
> > Kaxil
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>
> For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>
>
>
>
>
>
>
Re: Airflow 3 Dev Call #1 with Proposed Agenda
Posted by "Mehta, Shubham" <sh...@amazon.com.INVALID>.
Kaxil - thank you for creating the wiki, setting up call invites, and starting this thread. The first draft of the principles looks great. I have commented on the wiki with some feedback and personal thoughts. I'm not adding the comments on this thread to keep the wiki as the single place for discussions and feedback.
Thanks
Shubham
On 2024-05-28, 12:25 PM, "Kaxil Naik" <kaxilnaik@apache.org <ma...@apache.org>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
If the formatting of bullets was lost, check below or at https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024>
Proposed Agenda:
1) Agreeing on the Principles to drive Airflow 3 development
2) Agreeing on the Guidelines that help decide if a feature should be in Airflow 3 or not
For (1), I propose the following principles:
- Considering Airflow 3.0 for early adopters and breaking (and removing) things. Things can be re-added as needed in upcoming minor releases.
- Optimize to get foundational pieces in and not "let perfect be the enemy of good"
- Smoother migration path between AF 2 & 3 especially for DAG Authors with the existing official Airflow providers.
- Working on features that solidify Airflow as the modern Orchestrator that also has state-of-the-art support for Data, AI & ML workloads.
- This includes improving scalability & performance of all the Airflow components.
- Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability
- Set up the codebase for the next 3-5 years.
- Reducing matrix of supported combinations for reducing complexity in testing & development. E.g Remove MySQL support to reduce the test matrix
- Simplifying codebase & standardize architecture (e.g consolidating serialization methods)
- Remove deprecations
- Simplify the Learning Curve for new Airflow users
- Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10. This should continue for a longer period of time after AF 3 release
- Target a shorter cycle to release Airflow 3
- so that Airflow 2 branches for features don't diverge
- have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations
For (2), I propose the following guidelines:
- Alignment with Core Principles
- Community Demand and Feedback
- Impact on Scalability and Performance
- Implementation Complexity and Maintenance
- Backward Compatibility and Migration Effort
- Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found
- For big epics, AIPs & a successful vote on the dev mailing list
Please reply if anyone has anything to add to the agenda or comment on anything if you disagree.
Looking forward to the call.
Regards,
Kaxil
On 2024/05/28 19:11:14 Kaxil Naik wrote:
> Hi all,
>
> As discussed in the previous email thread, the first dev call has been
> pushed to next Tuesday (4th June 2024).
>
> If you would like to participate in the development of Airflow 3, please
> join the dev calls starting next week. The calls will be open to anyone in
> the community.
>
> *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 PM
> EST | 9 AM PST)
> *One-time registration Link*:
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG <https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG>
>
> The meeting notes from the call will also be posted on the dev mailing list
> and Confluence for archival purposes
> at
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes>.
>
>
>
> *Proposed Agenda*:
> 1) Agreeing on the Principles to drive Airflow 3 development
> 2) Agreeing on the Guidelines that help decide if a feature should be in
> Airflow 3 or not
>
>
> For (1), I propose the following principles:
>
> - Considering Airflow 3.0 for early adopters and breaking (and
> removing) things. Things can be re-added as needed in upcoming minor
> releases.
> - Optimize to get foundational pieces in and not "let perfect be
> the enemy of good"
> - Smoother migration path between AF 2 & 3 especially for DAG Authors
> with the existing official Airflow providers.
> - Working on features that solidify Airflow as the modern Orchestrator
> that also has state-of-the-art support for Data, AI & ML workloads.
> - This includes improving scalability & performance of all the
> Airflow components.
> - Making Airflow aware of what's happening in the task to provide
> better auditability, lineage & observability
> - Set up the codebase for the next 3-5 years.
> - Reducing matrix of supported combinations for reducing complexity
> in testing & development. E.g Remove MySQL support to reduce the test matrix
> - Simplifying codebase & standardize architecture (e.g
> consolidating serialization methods)
> - Remove deprecations
> - Simplify the Learning Curve for new Airflow users
> - Shift focus on Airflow 2 to stability: bug fixes + security fixes
> after AF 2.10. This should continue for a longer period of time after AF 3
> release
> - Target a shorter cycle to release Airflow 3
> - so that Airflow 2 branches for features don't diverge
> - have enough time between Airflow 3 release and Airflow Summit
> 2025, so we can have talks about Successful migrations
>
> For (2), I propose the following guidelines:
>
> - Alignment with Core Principles
> - Community Demand and Feedback
> - Impact on Scalability and Performance
> - Implementation Complexity and Maintenance
> - Backward Compatibility and Migration Effort
> - Workstream Ownership (can be more than one). If no one is available
> to lead the workstream, the feature will be parked until a dedicated owner
> is found
> - For big epics, AIPs & a successful vote on the dev mailing list
>
> Please reply if anyone has anything to add to the agenda or comment on
> anything if you disagree.
>
> Looking forward to the call.
>
> Regards,
> Kaxil
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
Re: Airflow 3 Dev Call #1 with Proposed Agenda
Posted by Kaxil Naik <ka...@apache.org>.
If the formatting of bullets was lost, check below or at https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
Proposed Agenda:
1) Agreeing on the Principles to drive Airflow 3 development
2) Agreeing on the Guidelines that help decide if a feature should be in Airflow 3 or not
For (1), I propose the following principles:
- Considering Airflow 3.0 for early adopters and breaking (and removing) things. Things can be re-added as needed in upcoming minor releases.
- Optimize to get foundational pieces in and not "let perfect be the enemy of good"
- Smoother migration path between AF 2 & 3 especially for DAG Authors with the existing official Airflow providers.
- Working on features that solidify Airflow as the modern Orchestrator that also has state-of-the-art support for Data, AI & ML workloads.
- This includes improving scalability & performance of all the Airflow components.
- Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability
- Set up the codebase for the next 3-5 years.
- Reducing matrix of supported combinations for reducing complexity in testing & development. E.g Remove MySQL support to reduce the test matrix
- Simplifying codebase & standardize architecture (e.g consolidating serialization methods)
- Remove deprecations
- Simplify the Learning Curve for new Airflow users
- Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10. This should continue for a longer period of time after AF 3 release
- Target a shorter cycle to release Airflow 3
- so that Airflow 2 branches for features don't diverge
- have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations
For (2), I propose the following guidelines:
- Alignment with Core Principles
- Community Demand and Feedback
- Impact on Scalability and Performance
- Implementation Complexity and Maintenance
- Backward Compatibility and Migration Effort
- Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found
- For big epics, AIPs & a successful vote on the dev mailing list
Please reply if anyone has anything to add to the agenda or comment on anything if you disagree.
Looking forward to the call.
Regards,
Kaxil
On 2024/05/28 19:11:14 Kaxil Naik wrote:
> Hi all,
>
> As discussed in the previous email thread, the first dev call has been
> pushed to next Tuesday (4th June 2024).
>
> If you would like to participate in the development of Airflow 3, please
> join the dev calls starting next week. The calls will be open to anyone in
> the community.
>
> *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 PM
> EST | 9 AM PST)
> *One-time registration Link*:
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
>
> The meeting notes from the call will also be posted on the dev mailing list
> and Confluence for archival purposes
> at
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes.
>
>
>
> *Proposed Agenda*:
> 1) Agreeing on the Principles to drive Airflow 3 development
> 2) Agreeing on the Guidelines that help decide if a feature should be in
> Airflow 3 or not
>
>
> For (1), I propose the following principles:
>
> - Considering Airflow 3.0 for early adopters and breaking (and
> removing) things. Things can be re-added as needed in upcoming minor
> releases.
> - Optimize to get foundational pieces in and not "let perfect be
> the enemy of good"
> - Smoother migration path between AF 2 & 3 especially for DAG Authors
> with the existing official Airflow providers.
> - Working on features that solidify Airflow as the modern Orchestrator
> that also has state-of-the-art support for Data, AI & ML workloads.
> - This includes improving scalability & performance of all the
> Airflow components.
> - Making Airflow aware of what's happening in the task to provide
> better auditability, lineage & observability
> - Set up the codebase for the next 3-5 years.
> - Reducing matrix of supported combinations for reducing complexity
> in testing & development. E.g Remove MySQL support to reduce the test matrix
> - Simplifying codebase & standardize architecture (e.g
> consolidating serialization methods)
> - Remove deprecations
> - Simplify the Learning Curve for new Airflow users
> - Shift focus on Airflow 2 to stability: bug fixes + security fixes
> after AF 2.10. This should continue for a longer period of time after AF 3
> release
> - Target a shorter cycle to release Airflow 3
> - so that Airflow 2 branches for features don't diverge
> - have enough time between Airflow 3 release and Airflow Summit
> 2025, so we can have talks about Successful migrations
>
> For (2), I propose the following guidelines:
>
> - Alignment with Core Principles
> - Community Demand and Feedback
> - Impact on Scalability and Performance
> - Implementation Complexity and Maintenance
> - Backward Compatibility and Migration Effort
> - Workstream Ownership (can be more than one). If no one is available
> to lead the workstream, the feature will be parked until a dedicated owner
> is found
> - For big epics, AIPs & a successful vote on the dev mailing list
>
> Please reply if anyone has anything to add to the agenda or comment on
> anything if you disagree.
>
> Looking forward to the call.
>
> Regards,
> Kaxil
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org